We Keep Coding

Include vs select projects in EfCore

I have this query
var answers = await context.QuestionnaireUserAnswers
.Include(a => a.QuestionAnsweres)
.Include(a => a.QuestionnaireSentOut).ThenInclude(q => q.Questionnaire)
.Include(a => a.User).ThenInclude(w => w.Organization)
.Where(a => a.QuestionnaireSentOut.Questionnaire.Status != QuestionnaireStatus.ArchivedDraft &&
a.QuestionnaireSentOut.Questionnaire.Status != QuestionnaireStatus.ArchivedPublished)
.Where(a => a.QuestionnaireSentOut.QuestionnaireType.Equals(QuestionnaireSentType.Normal))
.Where(a => a.Status.Equals(QuestionAnsweredStatus.Draft))
.Where(a => a.NextReminderDate < DateTime.UtcNow)
.ToListAsync();
which I changed to this
var answers = await context.QuestionnaireUserAnswers
.Where(a => a.QuestionnaireSentOut.Questionnaire.Status != QuestionnaireStatus.ArchivedDraft &&
a.QuestionnaireSentOut.Questionnaire.Status != QuestionnaireStatus.ArchivedPublished)
.Where(a => a.QuestionnaireSentOut.QuestionnaireType.Equals(QuestionnaireSentType.Normal))
.Where(a => a.Status.Equals(QuestionAnsweredStatus.Draft))
.Where(a => a.NextReminderDate < DateTime.UtcNow)
.Select(ans => new QuestionnaireApiServiceReminderModel
{
AnswerId = ans.Id,
Created = ans.Created,
NextReminderDate = ans.NextReminderDate,
QuestionnaireSentOutQuestionnaireTitle = ans.QuestionnaireSentOut.Questionnaire.Title,
QuestionnaireSentOutTitle = ans.QuestionnaireSentOut.Questionnaire.Title,
User = new SimpleUserWithOrganizationId
{
OrganizationId = ans.User.OrganizationId,
UserId = ans.UserId,
Email = ans.User.Email,
Name = ans.User.Name,
}
})
.ToListAsync();
so that I only get those properties that I need.
Down below my code does this
foreach (var answer in answers) {
.. // some calls
answer.NextReminderDate = DateTime.UtcNow.AddDays(7);
}
await context.SaveChangesAsync();
Now if I go with my updated version then I need to fetch the corresponding QuestionnaireUserAnswers before updating it. Which will be multiple roundtrips to database. Does this mean that in this case it is better to use the first query with includes?
One way to handle this could be write something like this
await context.Database.ExecuteSqlInterpolatedAsync($"UPDATE QuestionnaireUserAnswers SET NextReminderDate = {newNextReminderDate} WHERE Id IN ({string.Join(',', answers.Select(x => x.AnswerId))})");
but is this an approved efcore way?
So is this a better solution in this scanerio?
var answers = await context.QuestionnaireUserAnswers
.Where(a => a.QuestionnaireSentOut.Questionnaire.Status != QuestionnaireStatus.ArchivedDraft &&
a.QuestionnaireSentOut.Questionnaire.Status != QuestionnaireStatus.ArchivedPublished)
.Where(a => a.QuestionnaireSentOut.QuestionnaireType.Equals(QuestionnaireSentType.Normal))
.Where(a => a.Status.Equals(QuestionAnsweredStatus.Draft))
.Where(a => a.NextReminderDate < DateTime.UtcNow)
.Select(ans => new QuestionnaireApiServiceReminderModel
{
Answer = ans,
AnswerId = ans.Id,
Created = ans.Created,
NextReminderDate = ans.NextReminderDate,
QuestionnaireSentOutQuestionnaireTitle = ans.QuestionnaireSentOut.Questionnaire.Title,
QuestionnaireSentOutTitle = ans.QuestionnaireSentOut.Questionnaire.Title,
User = new SimpleUserWithOrganizationId
{
OrganizationId = ans.User.OrganizationId,
UserId = ans.UserId,
Email = ans.User.Email,
Name = ans.User.Name,
}
})
.ToListAsync();
//Sending reminders
foreach (var answer in answers)
{
try
{
answer.Answer.NextReminderDate = DateTime.UtcNow.AddDays(7);
QuestionnaireReminder(
answer.User.Name,
answer.User.Email,
answer.QuestionnaireSentOutQuestionnaireTitle,
answer.QuestionnaireSentOutTitle,
answer.Created.ToString(),
answer.AnswerId,
appServer,
answer.User.OrganizationId, _configuration);
}
catch (Exception)
{
await ErrorHandleService.HandleError("Cound not send questionnaire reminder",
"Cound not send questionnaire reminder email to: " + answer.User.Email,
null, null, null, _configuration, sendDeveloperMails, currentVersion, whoAmIName);
}
}
await context.SaveChangesAsync();

The short answer is Yes, and No.
When reading data, project to DTO/ViewModels like your updated code. This reduces the size of the resulting data to pull back just the info from the related entities that is needed.
When updating data, load the tracked entity/entities.
The "No" part of the answer is that in your first query, remove the Include statements, they are not needed, and should be removed to avoid extra data being sent across the wire, potentially forming a Cartesian Product across 1-to-many tables via JOINs.
var answers = await context.QuestionnaireUserAnswers
.Where(a => a.QuestionnaireSentOut.Questionnaire.Status != QuestionnaireStatus.ArchivedDraft &&
a.QuestionnaireSentOut.Questionnaire.Status != QuestionnaireStatus.ArchivedPublished)
.Where(a => a.QuestionnaireSentOut.QuestionnaireType.Equals(QuestionnaireSentType.Normal))
.Where(a => a.Status.Equals(QuestionAnsweredStatus.Draft))
.Where(a => a.NextReminderDate < DateTime.UtcNow)
.ToListAsync();
Include is not needed to perform Where conditions. EF will work those out into the query just fine. It is only needed in cases where you want to access those related entities within working with the resultset, otherwise they would be lazy loaded with potentially several DB round-trips getting queued up.
This is provided the "Some Calls" don't attempt to access related entities on the answer. If your logic does need some data from QuestionaireSentOut or Questionaire etc. then you can use a mix of projection with the entity reference:
.Select(ans => new
{
QuestionnaireSentOutQuestionnaireTitle = ans.QuestionnaireSentOut.Questionnaire.Title,
QuestionnaireSentOutTitle = ans.QuestionnaireSentOut.Questionnaire.Title,
Answer = ans,
User = ans.User.Select(u => new
{
OrganizationId = u.OrganizationId,
UserId = u.UserId,
Email = u.Email,
Name = u.Name,
}
})
Similar to your projection, you can return the answer entity in the resulting query for processing purposes. Inside the loop if the logic needs details from the user or questionaire, it gets it from the resulting projection, then when you go to update the answer itself, update it via the .Answer reference which will be the tracked EF entity. This approach should not be used for Read-type operations where populating a ViewModel to be sent back to a View etc. It's advisable to not mix view models and entities to avoid errors and performance issues with serialization and such. Where I need details like this I will use anonymous types to discourage passing objects with entity references around.

You should be able do this with the free library Entity Framework Plus using the Batch Update feature. The feature was created to work for multiple records, but it will work just as well for just one record.
It constructs an UPDATE ... SET ... WHERE ... query similar to what you wrote yourself, without loading any data from the database into the DbContext.
Sample code:
using Z.EntityFramework.Plus;
var id = .... ;
var newDate = DateTime.UtcNow.AddDays(7);
context.QuestionnaireUserAnswers
.Where(a => a.Id == id)
.Update(a => new QuestionnaireUserAnswer() { NextReminderDate = newDate });
Or for a collection of Id values:
using Z.EntityFramework.Plus;
var ids = new[] { ... , ... , ... };
var newDate = DateTime.UtcNow.AddDays(7);
context.QuestionnaireUserAnswers
.Where(a => ids.Contains(a.Id))
.Update(a => new QuestionnaireUserAnswer() { NextReminderDate = newDate });
A note: I believe this will not update any QuestionnaireUserAnswer objects that are already loaded in your DbContext. This may not be much of a problem in ASP .NET Core (or API) where contexts are short-lived, but it could be an issue if you use the context for a longer period.

Linq one to many with filter

I have an Entity Framework database that I'm querying, so I'm using linq-to-entities.
Here's my query:
// 'Find' is just a wrapper method that returns IQueryable
var q = r.Find(topic =>
topic.PageId != null &&
!topic.Page.IsDeleted &&
topic.Page.IsActive)
// These are standard EF extension methods, which are used to include
linked tables. Note: Page_Topic has a one-to-many relationship with topic.
.Include(topic => topic.Page.Route)
.Include(topic => topic.Page_Topic.Select(pt => pt.Page.Route))
// HERE'S THE QUESTION: This select statement needs to flatten Page_Topic (which it does). But it seems to do it in the wrong place. To explain, if I were to include another column that depended on Page_Topic (for example: 'PillarRoutName2', I'd have to apply the same flattening logic to that column too. Surely the filtering of Page_Topic should be done higher up the query in a DRY way.
.Select(x => new
{
TopicName = x.Name,
HubRouteName = x.Page.Route.Name,
PillarRouteName = x.Page_Topic.FirstOrDefault(y => y.IsPrimary).Page.Route.Name
}).ToList();

Surely the filtering of Page_Topic should be done higher up the query in a DRY way.
Correct! And it's easy to do this:
.Select(x => new
{
TopicName = x.Name,
HubRouteName = x.Page.Route.Name,
FirstTopic = x.Page_Topic.FirstOrDefault(y => y.IsPrimary)
})
.Select(x => new
{
TopicName = x.TopicName,
HubRouteName = x.HubRouteName,
PillarRouteName = x.FirstTopic.Page.Route.Name,
PillarRoutName2 = x.FirstTopic. ...
}).ToList();
Depending on where you start to get properties from FirstTopic you can also use x.Page_Topic.FirstOrDefault(y => y.IsPrimary).Page or .Page.Route in the first part.
Note that you don't need the Includes. They will be ignored because the query is a projection (Select(x => new ...).

Linq - Order by in Include

I have a situation where OrderBy need to be done for Include object. This is how I have tried so far
Customers query = null;
try
{
query = _context.Customers
.Include(x => x.CustomerStatus)
.ThenInclude(x => x.StatusNavigation)
.Select(x => new Customers()
{
Id = x.Id,
Address = x.Address,
Contact = x.Contact,
Name = x.Name,
CustomerStatus = new List<CustomerStatus>
{
x.CustomerStatus.OrderByDescending(y => y.Date).FirstOrDefault()
}
})
.FirstOrDefault(x => x.Id == 3);
}
catch (Exception ex)
{
throw;
}
The above code successfully ordering the include element but it is not including it's child table.
Eg: Customer include CustomerStatus but CustomerStatus not including StatusNavigation tables.
I even tried with this but neither it can help me
_context.Customers
.Include(x => x.CustomerStatus.OrderByDescending(y => y.Date).FirstOrDefault())
.ThenInclude(x => x.StatusNavigation).FirstOrDefault(x => x.Id == 3);
What am I doing wrong please guide me someone
Even I tried this way
var query = _context.CustomerStatus
.GroupBy(x => x.CustomerId)
.Select(x => x.OrderByDescending(y => y.Date).FirstOrDefault())
.Include(x => x.StatusNavigation)
.Join(_context.Customers, first => first.CustomerId, second => second.Id, (first, second) => new Customers
{
Id = second.Id,
Name = second.Name,
Address = second.Address,
Contact = second.Contact,
CustomerStatus = new List<CustomerStatus> {
new CustomerStatus
{
Id = first.Id,
CustomerId = first.CustomerId,
Date = first.Date,
StatusNavigation = first.StatusNavigation
}
},
}).FirstOrDefault(x => x.Id == 3);
but this is hitting a databases a 3 times and filtering the result in memory.
First select all data from customer status and then from status and then from customer then it filter all the data in memory. Is there any other efficient way to do this??
This is how I have prepared by entity class

As #Chris Pratt mentioned once you are doing new Customer inside the select you are creating a new model. You are discarding the models build by the EntityFramework. My suggestion would be have the query just:
query = _context.Customers
.Include(x => x.CustomerStatus)
.ThenInclude(x => x.StatusNavigation);
Like this you would have an IQueryable object which it would not be executed unless you do a select from it:
var customer3 = query.FirstOrDefault(x=>x.Id==3)
Which returns the customer and the interlinked tables (CustomerStatus and StatusNavigation). Then you can create the object that you want:
var customer = new Customers()
{
Id = customer3.Id,
Address = customer3.Address,
Contact = customer3.Contact,
Name = x.Name,
CustomerStatus = new List<CustomerStatus>
{
customer3.CustomerStatus.OrderByDescending(y => y.Date).FirstOrDefault()
}
})
In this way you can reuse the query for creating different response objects and have a single querying to database, but downside is that more memory is used then the original query (even though it shouldn't be too much of an issue).
If the model that is originally return from database doesn't meet the requirements (i.e. you always need to do: CustomerStatus = new List {...} ) it might indicate that the database schema is not well defined to the needs of the application, so a refactoring might be needed.

What I think is happening is that you are actually overriding the Include and ThenInclude. Include is explicitly to eager-load a navigation property. However, you're doing a couple of things that are likely hindering this.
First, you're selecting into a new Customer. That alone may be enough to break the logic of Include. Second, you're overriding what gets put in the CustomerStatus collection. That should ideally be just loaded in automatically via Include, but by altering it to just have the first entity, you're essentially throwing away the effect of Include. (Selecting a relationship is enough to cause a join to be issued, without explicitly calling Include). Third, the ThenInclude is predicated on the Include, so overriding that is probably throwing out the ThenIncude as well.
All this is conjecture. I haven't done anything exactly like what you're doing here before, but nothing else makes sense.
Try selecting into a new CustomerStatus as well:
CustomerStatus = x.CustomerStatus.OrderByDescending(o => o.Date).Select(s => new CustomerStatus
{
x.Id,
x.Status,
x.Date,
x.CustomerId,
x.Customer,
x.StatusNavigation
})
You can remove the Include/ThenInclude at that point, because the act of selecting these relationships will cause the join.

After Reading from Couple of sources (Source 1) and (Source 2). I think what is happening is that If you use select after Include. It disregards Include even if you are using Include query data in select. So to solve this use .AsEnumerable() before calling select.
query = _context.Customers
.Include(x => x.CustomerStatus)
.ThenInclude(x => x.StatusNavigation)
.AsEnumerable()
.Select(x => new Customers()
{
Id = x.Id,
Address = x.Address,
Contact = x.Contact,
Name = x.Name,
CustomerStatus = new List<CustomerStatus>
{
x.CustomerStatus.OrderByDescending(y => y.Date).FirstOrDefault()
}
})
.FirstOrDefault(x => x.Id == 3);

Retain default order for Linq Contains

I want to retain the default order that comes from sql, after processing by Linq also.I know this question has been asked before. Here is a link Linq Where Contains ... Keep default order.
But still i couldn't apply it to my linq query correctly. could anyone pls help me with this? Thanks!
Here is the query
var x = db.ItemTemplates.Where(a => a.MainGroupId == mnId)
.Where(a => a.SubGruopId == sbId)
.FirstOrDefault();
var ids = new List<int> { x.Atribute1, x.Atribute2, x.Atribute3, x.Atribute4 };
var y = db.Atributes.Where(a => ids.Contains(a.AtributeId))
.Select(g => new
{
Name = g.AtributeName,
AtType = g.AtributeType,
Options = g.atributeDetails
.Where(w=>w.AtributeDetailId!=null)
.Select(z => new
{
Value=z.AtributeDetailId,
Text=z.AtDetailVal
})
});

Your assumption is wrong. SQL server is the one that is sending the results back in the order you are getting them. However, you can fix that:
var x = db.ItemTemplates.Where(a => a.MainGroupId == mnId)
.Where(a => a.SubGruopId == sbId)
.FirstOrDefault();
var ids = new List<int> { x.Atribute1, x.Atribute2, x.Atribute3, x.Atribute4 };
var y = db.Atributes.Where(a => ids.Contains(a.AtributeId))
.Select(g => new
{
Id = g.AtributeId,
Name = g.AtributeName,
AtType = g.AtributeType,
Options = g.atributeDetails
.Where(w=>w.AtributeDetailId!=null)
.Select(z => new
{
Value=z.AtributeDetailId,
Text=z.AtDetailVal
})
})
.ToList()
.OrderBy(z=>ids.IndexOf(z.Id));
Feel free to do another select after the orderby to create a new anonymous object without the Id if you absolutely need it to not contain the id.
PS. You might want to correct the spelling of Attribute, and you should be consistent in if you are going to prefix your property names, and how you do so. Your table prefixes everything with Atribute(sp?), and then when you go and cast into your anonymous object, you remove the prefix on all the properties except AtributeType, which you prefix with At. Pick one and stick with it, choose AtName, AtType, AtOptions or Name, Type, Options.

ASP MVC MsSql to MySQL migration

For a low budget project I have to run IIS Asp Mvc with MySql. Migrating an existing project runs fine but if I create a LINQ query with Take & Skip it fails.
First Test (OK)
var post = _db.Posts.FirstOrDefaultAsync(a => a.id == 1234);
Second Test (OK)
var post = _db.Posts.Include(a => a.Comments);
var result = await post.Select(a => new TRDPostViewModel
{
Created = a.Created,
Body = a.Body,
Comments = a.Comments.Select(d => new TRDCommentViewModel
{
Body = d.Body,
Id = d.Id,
}).Where(m => m.Trash == false)
.OrderByDescending(f => f.Created)
.ToList(),
}).FirstOrDefaultAsync();
Third Test (FAIL)
var result = await post.Select(a => new TRDPostViewModel
{
Created = a.Created,
Body = a.Body,
Comments = a.Comments.Select(d => new TRDCommentViewModel
{
Body = d.Body,
Id = d.Id,
}).Where(m => m.Trash == false)
.OrderByDescending(f => f.Created)
.Skip(33)
.Take(10)
.ToList(),
}).FirstOrDefaultAsync();
And here is the Trace:
Unknown column 'Extent1.Id' in 'where clause'MySql.Data.MySqlClient.MySqlException
Makes no sense at all. Same code with MsSql is working fine. Using latest MySql.Data.Entity.EF6, Version=6.9.7.0
Am I missing something? Spend hours to solve but without success.

Are you sure your second query is really OK?
1) Id = d.Id, <= Why this comma (not really important)? ('ID =' is redundant)
2) .Where(m => m.Trash == false) <= 'Trash' is not in the select, so this property is not know at this time
3) .OrderByDescending(f => f.Created) <= idem for 'Created'
4) Why a comma after .ToList()?
I have simplified your DDL (which is not a MWE) with generated data.
I have reproduced your problem in VS2013.
I have also test your query with LINQPad directly against the database and I have the same problem with the third test, probably a bug in the driver mysql:
trdposts.Select(a => new {
Created = a.Created,
Body = a.Body,
Comments = a.Posttrdcomments
.Select(d => new { Body = d.body, Id = d.Id, d.Created, d.Trash})
.Where(m => m.Trash == 1)
.OrderByDescending(f => f.Created)
.Skip(33)
.Take(10)
.ToList()
})
Give a shorter SQL query:
SELECT t1.PostId, t1.body, t1.Id, t1.Created, t1.Trash
FROM trdposts AS t0
OUTER APPLY (
SELECT t2.body, t2.Created, t2.Id, t2.PostId, t2.Trash
FROM trdcomments AS t2
WHERE ((t2.PostId = t0.Id) AND (t2.Trash = 1))
ORDER BY t2.Created DESC
) AS t1
ORDER BY t1.Created DESC
Without .Skip() and .Take(), we get good 'LEFT OUTER JOIN'

That kind of query is kind of impossible to do with MySQL. How would you write it in SQL if you wrote the query yourself? The problem is that MySQL has no support for what the SQL standard calls lateral joins, in MsSQL the keyword "APPLY" is used. The .NET driver for PostgreSQL and MsSQL supports those kind of queries, but not MySQL.

Its a known issue with MySQL EF and from other posts maybe MySQL itself.
See:
https://bugs.mysql.com/bug.php?id=78610
and other SO post:
Unknown column 'Project2.Name' in 'where clause'
it was posted 3 years ago and marked critical if that tells you anything. It exists in MySQL connector 6.9.11.0 and later. I've just had to work around it as best i can. I don't expect it to get fixed.