We Keep Coding

LinQ to SQL throws Stackoverflow exception when using Any()

I'm using a LinQ query that looks like this
public List<TEntity> GetEntities<TEntity>(int[] ids)
{
var someDbSet = new DbSet<TEntity>();
var resultQ = someDbSet.Where(t => !ids.Any() || ids.Contains(t.ID)); //<= crashing line
return resultQ.toList();
}
It usually works, but for some case when ids size is ~ 7000 items it crashes.
The thrown exception message is "Exception of type 'System.StackOverflowException' was thrown.".
It has no stack trace or InnerException.
I also get this info: "EntityFramework.pdb not loaded... contains the debug information required to find the source for the module EntityFramework.dll"
Is this a known bug or can someone explain why it doesn't work when the array is bigger?
I'm using .NET Framework 4.5, EntityFramework 6.1.3, EntityFramework6.Npgsql 3.0.3

If we pass an array with only two values int[] ids = {1, 2} to your method GetEntities EntityFramework will generate the next query:
SELECT
[Extent1].[Id] AS [Id],
...
FROM [dbo].[Entity] AS [Extent1]
WHERE ( NOT EXISTS (SELECT
1 AS [C1]
FROM (SELECT
1 AS [C0]
FROM ( SELECT 1 AS X ) AS [SingleRowTable1]
UNION ALL
SELECT
1 AS [C0]
FROM ( SELECT 1 AS X ) AS [SingleRowTable2]) AS [UnionAll1]
)) OR (1 = [Extent1].[Id]) OR (2 = [Extent1].[Id])
If we increase the number of elements in ids array this query becomes more complex with more levels of nestings. I think that EntityFramework uses some recursive algorithm to generate SQL-code for !ids.Any() expression. When the number of elements in ids array increases the depth of the recursion also increases. Therefore it generates StackOverflowException when the number of elements in ids array (and also the depth of the recursion) is large.
If we delete !ids.Any() expression the next query will be generated:
SELECT
[Extent1].[Id] AS [Id],
...
FROM [dbo].[Entity] AS [Extent1]
WHERE [Extent1].[Id] IN (1,2)
Such query does not generate StackOverflowException when the number of elements in the ids array is large. Therefore it would be better to extract !ids.Any() expression out of LINQ query:
public List<TEntity> GetEntities<TEntity>(int[] ids)
{
var someDbSet = new DbSet<TEntity>();
if (!ids.Any())
return someDbSet.ToList();
var resultQ = someDbSet.Where(t => ids.Contains(t.ID));
return resultQ.toList();
}
You should also take to account that there is a limitation on number of items for WHERE IN condition: Limit on the WHERE col IN (...) condition.
ionutnespus wrote:
Yes, extracting the condition outside Where() is working. Still, I
couldn't find any explanation why EF would use such a complicated
algorithm for such a simple condition. Any thoughts on that?
I've decided to answer this question by extending this post because the asnwer is large and contains code.
I don't know for sure why EF generates such complex query but I've made some research and here are my thoughts. If we modify your GetEntites method and use next condition in LINQ query:
someDbSet.Where(t => !ids.Any(i => i == 3) || ids.Contains(t.ID));
the next SQL-query will be generated if ids = {1, 2}:
SELECT
[Extent1].[Id] AS [Id],
...
FROM [dbo].[Entity] AS [Extent1]
WHERE ( NOT EXISTS (
SELECT 1 AS [C1]
FROM (
SELECT 1 AS [C0] FROM ( SELECT 1 AS X ) AS [SingleRowTable1] WHERE 3 = 1
UNION ALL
SELECT 1 AS [C0] FROM ( SELECT 1 AS X ) AS [SingleRowTable2] WHERE 3 = 2
) AS [UnionAll1]
)) OR (1 = [Extent1].[Id]) OR (2 = [Extent1].[Id])
Here you can see that NOT EXISTS condition contains two subqueries each of which checks if the next element of the ids array equals required value. I think that it is logical to use NOT EXISTS SQL-condition to represent Any() method. But why does EF generates one subquery for each array element? In my opinion EF does so because because EF Team tried to write algorithm that generates queries that are not depend on database type. But this is only my opinion. May be it would be better to ask this question EF Team on github.

Can you try like this?
public List<TEntity> GetEntities<TEntity>(int[] ids)
{
var someDbSet = new DbSet<TEntity>();
var resultQ = new List<your_list_type>();
foreach( var id in ids) {
resultQ.Add(someDbSet.Where(prm => prm.ID == id).FirstOrDefault());
}
return resultQ;
}

As per your error message, The exception that is thrown when the execution stack overflows because it contains too many nested method calls.
As MSDN
The default maximum size 2 gigabytes (GB) of an array.
In a 64-bit environment, you can avoid the size restriction by setting the enabled attribute of the gcAllowVeryLargeObjects configuration element to true in the run-time environment.
Moreover your ids exceeds to 2 gb limits. i think this might be the cause

Create complex query to execute queries stored in column?

I am trying to get some information from a table, when a (one or more than one) condition(s) is(are) met but this conditions are stored in other table.
Here is a screenshot of the 3 tables:
So I need a query to do this:
Given a Status (IDStatus) and an Action (IDAction)
If there is an IDCondition (it can be NULL, one, or more than one)
A. If there is one, use the IDocField as a column from a table called IDoc, using ConditionOperator as the condition (read this as =, <, >) and the value on ConditionValue.
B. If there are more than one, the same as before, but use each condition with AND.
C. If it is NULL, directly go to point 3
Get Subject, SendTo, CC, CCO, and FileTemplate that matched all of the above explained.
I want to try this because it would reduce my C# code, but if this is too complex or not plausible, I will do multiple queries like what I explained above.
I am asking this, because I am not sure how to get multiple conditions in a SELECT. I mean by this:
If there is more than one condition, how can I execute them or get all of them in a single query but that has all the returning things on point 3 ?
Example:
SELECT *
FROM WF_MailCondition
WHERE IDCondition = 1
this will return all conditions (keys can repeat) that has IDCondition equal to 1. But then I have to use all the three columns to create a condition like ID = 1 been ID in IDocField, = in ConditionOperator and 1 in ConditionValue.
I know there is some inconsistencies in the image, like IDCondition is NN (not null) but I said that it can be null (I am fixing this things).
My SQL:
SELECT
Subject, SendTo, CC, CCO, FileTemplate
FROM
WF_Mail AS M
INNER JOIN
WF_MailStatusAction AS S ON S.IDMail = M.IDMail
WHERE
(SELECT "HERE SELECT THE 3 COLUMNS TO USE A CONDITION" FROM WF_MailCondition WHERE IDCondition = S.IDCondition)
AND (S.IDAction = "ACTION FROM OUTSIDE" AND S.IDStatus = "STATUS FROM OUTSIDE")
How to do the SELECT "HERE SELECT 3 COLUMNS..." and if the IDCondition is NULL directly get the columns requested in the first SELECT (Subject, SendTo, etc.)
The "ACTION FROM OUTSIDE" is a parametrized statement.
Forget about foreign keys, we are not using it (boss' decisions...).
I am doing this because is for an "event" thing. I call a method in my C# code that will see if there are mails to send. But this mails can have none, one or more than one conditions. This is to have more options than all have to send a mail. Maybe in the workflow there is no need to send email but in code is "hardcoded" so with this I am totally free to do what I need about that feature.
Maybe all of this can be a function or a stored procedure but I am very ignorant about that and performance with them.
If you need more info, please tell me.

From what I understand, I would write a query like this:
select m.subject, m.cc
from wf_mail m
where exists (select 'true'
from wf_mailstatusaction s, wf_mailcondition c
where s.idmail=m.idmail
and s.idcondition = c.idcondition
and c.idocfield||c.conditionoperator||c.conditionValue = 'id>500'
group by s.idcondition
having count(s.idcondition) = 1)
or exists (select 'true'
from wf_mailstatusaction s, wf_mailcondition c
where s.idmail=m.idmail
and s.idcondition = c.idcondition
and c.idocfield||c.conditionoperator||c.conditionValue IN ('id>500', 'id<200')
group by s.idcondition
having count(s.idcondition) > 1)
or exists (select 'true'
from wf_mailstatusaction s
where s.idmail=m.idmail
and s.idcondition is null)

I would suggest the following:
- Create a QueryBuilder Class
Where you have methods like:
* AddSelect()
* AddFrom()
* AddWhere()
* ToString()
...
And then use the "middlelayer" to handle the query cunstruction based on the conditions
That way you have the most control over everything going on

How to resolve Subquery returned more than 1 value error

I'm trying to execute this stored procedure
ELSE IF(#Item_Type='Frames')
BEGIN
SELECT
ID.Selling_Price, --0
ID.Purchase_Price,--1
#Discount AS Discount,--2
IH.Item_Name AS Item_Group_Name,--3
#Item_Group_Code AS Item_Group_Code,--4
FD.Item_Code,
ID.Item_Name,
FD.Material,
FD.Color,
FD.RIM,
FD.Frame_Code
FROM
Frame_Details AS FD
LEFT JOIN
Item_Details AS ID ON FD.Item_Code = ID.Item_Code
LEFT JOIN
Item_Header AS IH ON IH.Item_G_Code = ID.Item_G_Code
WHERE
FD.Status = '0'
AND FD.Item_Code = #Item_Code
GROUP BY
ID.Selling_Price, ID.Purchase_Price, IH.Item_Name,
FD.Item_Code, ID.Item_Name, FD.Material, FD.Color, FD.RIM, FD.Frame_Code
END
it will not execute successfully..this is the error message that I got
Subquery returned more than 1 value. This is not permitted when the subquery follows =, !=, <, <= , >, >= or when the subquery is used as an expression.
I have tried several examples given before, but they don't works for me.

It seems that the subquery returns more than one records. In this case, the result is a set, not a single value, while which is not allowed in your scenario. To avoid this issue, you may have a try to join the result of this subquery, or use TOP 1 or max or sum or AVG to return only one value.
Thanks

Average numbers in my SQL query

I have a question regarding my SQL query. Below you will see my database:
And I have the following query right now:
SELECT enquete_vraag,enquete_antwoord,docent,vak,semesterstart
FROM ENQUETE_ANTWOORD
LEFT JOIN KDV ON ENQUETE_ANTWOORD.kdv_ID = KDV.kdv_ID
LEFT JOIN DOCENT ON KDV.docent_ID = DOCENT.docent_ID
LEFT JOIN VAK ON KDV.vak_ID = VAK.vak_ID
LEFT JOIN ENQUETE_VRAAG ON ENQUETE_ANTWOORD.enquete_vraag_ID = ENQUETE_VRAAG.enquete_vraag_ID
WHERE DOCENT.docent_ID = variableDocentID AND VAK.vak = variableVak
And I display the returned data in a datagridview:
Now the datagridview shows all questions that are being answered by all students. What I want is the average for each question and only show that. So you have 1 row that has question 6 with the average answer and question 7 with the average answer and so on, how do I achieve that in my SQL query?

hi you should do something like this
SELECT enquete_vraag,AVG(enquete_antwoord) enquete_antwoord,docent,vak,semesterstart
FROM ENQUETE_ANTWOORD
LEFT JOIN KDV ON ENQUETE_ANTWOORD.kdv_ID = KDV.kdv_ID
LEFT JOIN DOCENT ON KDV.docent_ID = DOCENT.docent_ID
LEFT JOIN VAK ON KDV.vak_ID = VAK.vak_ID
LEFT JOIN ENQUETE_VRAAG ON ENQUETE_ANTWOORD.enquete_vraag_ID = ENQUETE_VRAAG.enquete_vraag_ID
WHERE DOCENT.docent_ID = variableDocentID AND VAK.vak = variableVak
GROUP BY enquete_vraag, docent,vak,semesterstart

SELECT enquete_vraag,AVG(enquete_antwoord) as [enquete_antwoord]
FROM ...
GROUP BY enquete_vraag
The problem then, of course, becomes which vak etc to choose.... Because of translation, it is not easy for me to guess at which value means what, so it is hard to advise on that. You might be able to include the extra values in the GROUP BY clause (if they are the same for all the matching rows); or you might be able to take a MIN / MAX.

just add avg function in your code.
...AVG(enquete_antwoord)...
write above code it gives you the correct answer

Is this LINQ Query "correct"?

I have the following LINQ query, that is returning the results that I expect, but it does not "feel" right.
Basically it is a left join. I need ALL records from the UserProfile table.
Then the LastWinnerDate is a single record from the winner table (possible multiple records) indicating the DateTime the last record was entered in that table for the user.
WinnerCount is the number of records for the user in the winner table (possible multiple records).
Video1 is basically a bool indicating there is, or is not a record for the user in the winner table matching on a third table Objective (should be 1 or 0 rows).
Quiz1 is same as Video 1 matching another record from Objective Table (should be 1 or 0 rows).
Video and Quiz is repeated 12 times because it is for a report to be displayed to a user listing all user records and indicate if they have met the objectives.
var objectiveIds = new List<int>();
objectiveIds.AddRange(GetObjectiveIds(objectiveName, false));
var q =
from up in MetaData.UserProfile
select new RankingDTO
{
UserId = up.UserID,
FirstName = up.FirstName,
LastName = up.LastName,
LastWinnerDate = (
from winner in MetaData.Winner
where objectiveIds.Contains(winner.ObjectiveID)
where winner.Active
where winner.UserID == up.UserID
orderby winner.CreatedOn descending
select winner.CreatedOn).First(),
WinnerCount = (
from winner in MetaData.Winner
where objectiveIds.Contains(winner.ObjectiveID)
where winner.Active
where winner.UserID == up.UserID
orderby winner.CreatedOn descending
select winner).Count(),
Video1 = (
from winner in MetaData.Winner
join o in MetaData.Objective on winner.ObjectiveID equals o.ObjectiveID
where o.ObjectiveNm == Constants.Promotions.SecVideo1
where winner.Active
where winner.UserID == up.UserID
select winner).Count(),
Quiz1 = (
from winner2 in MetaData.Winner
join o2 in MetaData.Objective on winner2.ObjectiveID equals o2.ObjectiveID
where o2.ObjectiveNm == Constants.Promotions.SecQuiz1
where winner2.Active
where winner2.UserID == up.UserID
select winner2).Count(),
};

You're repeating join winners table part several times. In order to avoid it you can break it into several consequent Selects. So instead of having one huge select, you can make two selects with lesser code. In your example I would first of all select winner2 variable before selecting other result properties:
var q1 =
from up in MetaData.UserProfile
select new {up,
winners = from winner in MetaData.Winner
where winner.Active
where winner.UserID == up.UserID
select winner};
var q = from upWinnerPair in q1
select new RankingDTO
{
UserId = upWinnerPair.up.UserID,
FirstName = upWinnerPair.up.FirstName,
LastName = upWinnerPair.up.LastName,
LastWinnerDate = /* Here you will have more simple and less repeatable code
using winners collection from "upWinnerPair.winners"*/

The query itself is pretty simple: just a main outer query and a series of subselects to retrieve actual column data. While it's not the most efficient means of querying the data you're after (joins and using windowing functions will likely get you better performance), it's the only real way to represent that query using either the query or expression syntax (windowing functions in SQL have no mapping in LINQ or the LINQ-supporting extension methods).
Note that you aren't doing any actual outer joins (left or right) in your code; you're creating subqueries to retrieve the column data. It might be worth looking at the actual SQL being generated by your query. You don't specify which ORM you're using (which would determine how to examine it client-side) or which database you're using (which would determine how to examine it server-side).
If you're using the ADO.NET Entity Framework, you can cast your query to an ObjectQuery and call ToTraceString().
If you're using SQL Server, you can use SQL Server Profiler (assuming you have access to it) to view the SQL being executed, or you can run a trace manually to do the same thing.
To perform an outer join in LINQ query syntax, do this:
Assuming we have two sources alpha and beta, each having a common Id property, you can select from alpha and perform a left join on beta in this way:
from a in alpha
join btemp in beta on a.Id equals btemp.Id into bleft
from b in bleft.DefaultIfEmpty()
select new { IdA = a.Id, IdB = b.Id }
Admittedly, the syntax is a little oblique. Nonetheless, it works and will be translated into something like this in SQL:
select
a.Id as IdA,
b.Id as Idb
from alpha a
left join beta b on a.Id = b.Id

It looks fine to me, though I could see why the multiple sub-queries could trigger inefficiency worries in the eyes of a coder.
Take a look at what SQL is produced though (I'm guessing you're running this against a database source from your saying "table" above), before you start worrying about that. The query providers can be pretty good at producing nice efficient SQL that in turn produces a good underlying database query, and if that's happening, then happy days (it will also give you another view on being sure of the correctness).