We Keep Coding

.NET Core 3.1 LINQ expression from could not be translated for lists within a list

I have an List containing ids for posts on a blog. The list is generated through a separate system that allows the user to filter posts from a number of parameters. I have tested it, and it works. My objective is to take this List and perform a LINQ Query that grabs the users who have made the posts, and sort the posts according to who posted them. Query as follows:
List<UserPostDTO> UserPosts =
await (from x in _context.UserPosting //UserPosting links User and Post tables in the db. It has both user id and post id
where PostedByList.Contains(x.Posting.PostingId) //PostedByList is a List<int>
&& x.Approved == "A" //only show posts that have been approved
&& x.User.Active == true //only show posts from active users
group x by new { x.User.FirstName, x.User.LastName, x.User.UserId } into xGroup
select new UserPostDTO //Object containing First/Last name, email, and List<PostPOCO>
{
FirstName = xGroup.Key.FirstName, //string
LastName = xGroup.Key.LastName, //string
UserEmail = xGroup.Key.UserId, //int
//PostsByUser = (from y in xGroup //List of PostPOCO objects
// select new PostPOCO
// {
// PostTitle = y.Posting.Title, //string
// PostId = y.Posting.PostingId //int
// }).ToList()
}).ToListAsync(); //I have read this should solve the issue. It does not.
When I uncomment out the part of the query for PostsByUser, it gives the following error upon hitting the query:
The LINQ expression [...] could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync(). See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
If that part of the code is left commented, it runs without any errors.
I can't figure a way around this. I have tried replacing the second line with _context.UserPosting.AsEnumerable() , but this conflicts with the .ToListAsync(), which was supposed to be part of the client/server Evaluation workaround anyway. So no luck there.
This is a bit of an issue since the sub list depends on the main query.
Is there anyway to fix this? I have tried it in LinqPad and it works. It compiles in Visual Studio without error. It just crashed when actually trying to run.

I figured it out. Turns out the limitations imposed by Microsoft's new changes in EF Core altered what works in the Group clause: https://learn.microsoft.com/en-us/ef/core/what-is-new/ef-core-3.x/breaking-changes#linq-queries-are-no-longer-evaluated-on-the-client
To get this to work, it needs a fresh query on the database. Possibly not ideal, but there it is:
List<UserPostDTO> UserPosts =
await (from x in _context.UserPosting
where PostedByList.Contains(x.Posting.PostingId)
&& x.Approved == "A"
&& x.User.Active == true
group x by new { x.User.FirstName, x.User.LastName, x.User.UserId } into xGroup
select new UserPostDTO
{
FirstName = xGroup.Key.FirstName,
LastName = xGroup.Key.LastName,
UserEmail = xGroup.Key.UserId,
PostsByUser = (from y in _context.UserPosting
where xGroup.Key.UserId == y.UserId
select new PostPOCO
{
PostTitle = y.Posting.Title,
PostId = y.Posting.PostingId
}).ToList()
}).ToListAsync();

Why is my object being disposed?

I'm trying to access SQL server using LINQ with c# in asp.net framework.
I'm writing a simple log-in form on a webpage. The following is my c# code to check if it is in the database.
public int getUserId(string un, string pw)
{
IEnumerable<int> query;
using (var context = new IngredientsLinqDataContext())
{
query = from c in context.USERs
where c.Username == un && c.Password == pw
select c.UserID;
}
if(query.Count() >= 1)
{
return query.Min();//a very serious kludge. Need to fix this
}
else { return -1; }
}
}
An object-disposed-exception is thrown when checking if the count of the query is greater than or equal to one. I had thought that, since query gets declared outside of the curly braces, this shouldn't be an issue of scope.
Additionally, what is a good way to fix the kludge? It should only be returning a single int, not a list of ints. How can I treat it that way? Thanks!

Change
query = from c in context.USERs
where c.Username == un && c.Password == pw
select c.UserID;
to
query = (from c in context.USERs
where c.Username == un && c.Password == pw
select c.UserID).ToList();
LINQ is using deferred execution, so you query is not evaluated immediately, but it is evaluated only at query.Count(). The context at that moment is already disposed. That's why you receive error.

context is disposed because you are trying to retrieve the data from database out of the context using block;
using (var context = new IngredientsLinqDataContext())
{
query = from c in context.USERs
where c.Username == un && c.Password == pw
select c.UserID;
if (query.Count() >= 1)
{
return query.Min();//a very serious kludge. Need to fix this
}
else { return -1; }
}

You have to be aware that a Linq query can beAsEnumerable or AsQueryable.
If your query is AsEnumerable it holds all information needed to access the elements in your query: You can ask for the first element in your sequence (if there is one), and once you've got an element you can ask for the next one (if there is one). If your query uses other functions in your process, it has all information to access these other functions. In short: AsEnumerable is processed in local memory.
If your query is AsQueryable it holds an Expression and a Provider. The Provider knows which process is designated to process the Expression. It knows which language this process uses. It is the task of the Provider to convert the Expression into the language of the destination processor and to send it to this processor.
For Entity Framework this is usually a database like SQL or MySQL, but it could also be a spreadsheet or a CSV file. The nice thing about a DbContext is that you don't need to know which language it uses to communicate with the other process. You could use the same IDbSet classes to communicate with a completely different process.
So an IQueryable does not hold the information to enumerate over the elements in the query. It only knows how to translate it into the language of the designated process and who to ask to execute this query.
The translation and execution is not done until you ask for elements. This means that you have to keep the providing object that your query uses alive until you don't need any more results from this providing object anymore.
Executing the query is done whenever you ask for a function that doesn't use deferred execution (as is described in the remarks section of every linq function). These are functions like ToList(), FirstOrDefault(), Single(), Count(), etc.
Functions like Where, Select, GroupBy and others that use deferred execution only change the Expression.
Conclusion
Make sure you've fetched all data that you need from your provider before
you Dispose() it

May be this will fix your issue.
public int getUserId(string un, string pw)
{
int query;
using (var context = new IngredientsLinqDataContext())
{
--change here
query = (from c in context.USERs
where c.Username == un && c.Password == pw
select c.UserID).FirstOrDefault();
}
if(query != 0) --change here
{
return query;//a very serious kludge. Need to fix this
}
else { return -1; }
}

Your use of query in this construct instructs LINQ to wait until the object is accessed before executing the actual query against the database. In this case query is accessed outside your using block, so that's why you're getting the error.
Try this:
result = (from c in context.USERs
where c.Username == un && c.Password == pw
select c.UserID).Min();

How to structure a partially dynamic LINQ statement?

I currently have a method on my repository like this:
public int GetMessageCountBy_Username(string username, bool sent)
{
var query = _dataContext.Messages.AsQueryable();
if (sent)
query = query.Where(x => x.Sender.ToLower() == username.ToLower());
else
query = query.Where(x => x.Recipient.ToLower() == username.ToLower());
return query.Count();
}
It currently builds one of two queries based on the sent boolean. Is this the best way to do this or is there a way to do this within the query itself? I want to check if x.Sender is equal to username if sent equals true. But I want to check if x.Recipient is equal to username if sent equals false.
I then want this LINQ expression to translate into SQL within Entity Framework, which I believe it is doing.
I just want to avoid repeating as much code as possible.

You could do something like this :
public int GetMessageCountBy_Username(string username, bool sent)
{
Func<Message, string> userSelector = m => sent ? m.Sender : m.Recipient;
var query =
_dataContext.Messages.AsQueryable()
.Where(x => userSelector(x).ToLower() == username.ToLower());
return query.Count();
}
Thus the choosing of the right user (the sender or the recipient) is done before the linq part, saving you from repeating it twice.

Yes, I believe this is correct way to do it. Because it is easy to create complex queries without repeating whole parts of queries.
And your thinking about translating to SQL is correct too. But beware, this is done at the moment, when data or agregation is requested. In your case, the SQL will be generated and executed when you call Count().

LINQ returns 0 results if using nullable int variable, accurate results if using "null"

I have a table called "test", which only has 1 column, "NullableInt" (nullable int type)
The records are: 1, 2, null
int? nullableInt = null;
var t = db.tests.Where(x => x.NullableInt == null).ToList(); // returns 1 record
var t2 = db.tests.Where(x => x.NullableInt == nullableInt).ToList(); // returns 0 records
For some reason, t2 returns 0 records, even tho it's using "nullableInt" variable, which has a value of null, just like t, which is comparing against "null"
Any help would be greatly appreciated!

Yep - it's a bug in LINQ-to-SQL / Entity Framework. IS NULL queries will only be generated if you hardcode null into the query, instead of a variable that happens to currently be null.
The second query will generate
SELECT .......
WHERE NullableInt == #someParam
WHERE #someParam is null.
Where the first will generate the appropriate IS NULL in the WHERE clause.
If you're using LINQ-to-SQL, you can log your queries to Console.Out to see for yourself, and if you're using EF, then ToTraceString() should show you the same info (or SQL Server profiler)

tl;dr
If you use DbContext in EF6 this is fixed.
If you're using EF5 (or ObjectContext in EF6) you need to set ObjectContext.ContextOptions.UseCSharpNullComparisonBehavior to true. To do that on DbContext use this:
((IObjectContextAdapter)db).ObjectContext.ContextOptions.UseCSharpNullComparisonBehavior = true;
.
More details
The root cause of this issue is a difference in how the database compares null values and how C# compares null values. Because you write your query in C# you want to use the semantics of C#.
In EF5 we introduced ObjectContext.ContextOptions.UseCSharpNullComparisonBehavior, which allowed you to opt in to using C# semantics instead of database semantics. The default is false (so that existing queries don't magically start returning different results when you upgrade to EF5). But you can set it to true and both your queries will return rows.
If you are using DbContext in EF5 you need to drop down to the ObjectContext to set it:
((IObjectContextAdapter)db).ObjectContext.ContextOptions.UseCSharpNullComparisonBehavior = true;
If you are using EF6, then it's already set to true on DbContext so you are good to go. We decided this causes so much confusion it was worth taking the potential impact on existing queries.

Queries could be built in this way:
var q = db.tests;
if(nullableInt.HasValue)
{
q = q.Where(x => x.NullableInt == nullableInt.Value);
}
else
{
q = q.Where(x => x.NullableInt == null);
}
var t2 = q.ToList();

There is another solution that will always work, albeit with a small caveat:
int? nullableInt = null;
var t2 = db.tests.Where(x => object.Equals(x.NullableInt, nullableInt)).ToList();
When the value is null you will get the proper IS NULL query, however when its not null you will get something like:
SELECT ...
WHERE ([t0].[NullableInt] IS NOT NULL) AND ([t0].[NullableInt] = #p0)
Obviously it has a condition extra (the source of which is kind of puzzling). That being said, SQL Server's query optimizer should detect that, since #p0 is a non-null value, the first condition is a superset and will cut the where clause.

Would doing:
var t2 = db.tests.Where(x => x.NullableInt == nullableInt ?? null).ToList();
Work?
It seems like utter madness though.

Calling functions from within linq statement

Just wondering if this is the most efficient method of doing this? is there a way of having all the linq within one statement instead of calling a method, like a subselect or something?
newEmployee = (from emp
in db.employees
select new
{
a.EmployeeID,
a.Username,
Status = emp.GetEmployeeCurrentStatus(a.Username)
}).ToList();
This is the GetEmployeeCurrentStatus which returns the status of the employee:
public string GetEmployeeCurrentStatus(string username)
{
using (Entities db = new Entities())
{
var times = (from d in db.TimeTables
where d.DateTime == DateTime.Today &&
d.Employee.Username == username
select d)
.OrderByDescending(d => d.TimeID).FirstOrDefault();
return (x.ClockOut == null ? "IN" : "OUT");
}
}

how about:
newEmployee = (db.employees.Select(emp => new
{
emp.EmployeeID,
emp.Username,
Status = db.TimeTables
.Where(d => d.Employee.Username == emp.Username
&& d.DateTime == DateTime.Today)
.Select(x => x.ClockOut == null ? "IN" : "OUT")
.FirstOrDefault()
})).ToList();
Your attempt may appear cleaner and is functionally ok. However, it is firing up a secondary db call. This will be bad for scalability and performance. The version i've posted uses the same initial db connection and will make the join 1-1. This will result in tighter, faster queries as well as lower resource usage.

You cannot really call a custom method inside a query (or a part of a query that will be executed using the databse). You have essentially two options:
Call ToList before performing the select that needs to call the method (this way, the method will be called on in-memory data)
Compose the query such that it can all run on the SQL server if it is possible. This can be done using AsExpandable extension in predicate builder. For more information on how this works, see also my blog post.

its fine for small data (employees count) but since each GetEmployeeCurrentStatus requires an sql new connection so its not that best practice.
I personally will get all employees (one trip to database) and then get all employees status (one trip to database) so i cashed them all, now i'll join them locally
Hope this helped

Regardless of efficiency, having GetEmployeeCurrentStatus(...) as a method makes the code clearer and more reusable.

Assuming you are using LINQ to SQL or EF, I would refactor your query to use a Join. That way, you will execute a single efficient SQL query on the database, instead of two separate queries.