We Keep Coding

entity framwork core generates weird sql

I have an existing LINQ query that I am trying to optimize. I have the following entity Types (simplified)
public class Account
{
public int Id { get; set; }
public IEnumerable<OpportunityInfo> Opportunities { get; set; }
}
public class Opportunity
{
public int Id { get; set; }
public string Name { get; set; }
public bool Active { get; set; }
public IEnumerable<Quote> Quotes { get; set; }
}
public class Quote
{
}
It is a standard hierarchy of Account to Opportunity to Quote. Nothing Special. I have the following query that I am using on an ASP.NET Core controller index method. I am starting from Quote and working backwards because there is dynamic query logic between the query and opportunityQuotes that must be Quote based. Otherwise I would start from the top direction.
var query = from o in Quotes select o;
additional query logic (filtering and sorting)
var opportunityQuotes = from o in query
group o by new
{
accountId = o.Opportunity.AccountId,
accountName = o.Opportunity.Account.Name,
active = o.Opportunity.Account.Active,
}
into p
select new
{
Id = p.Key.accountId,
Name = p.Key.accountName,
Active = p.Key.active,
Opportunities =
(from q in p
group q by new
{
Id = q.Opportunity.Id,
Name = q.Opportunity.Name,
Active = q.Opportunity.Active
}
into r
select new
{
Name = r.Key.Name,
Id = r.Key.Id,
Active = r.Key.Active,
Quotes = r
})
};
opportunityQuotes.Dump();
This query generates the following SQL.
SELECT [o].[Id], [o].[ARRValue], [o].[AccountId], [o].[AdjustedArr], ...
FROM [Quotes] AS [o]
LEFT JOIN [Opportunities] AS [o.Opportunity] ON [o].[OpportunityId] = [o.Opportunity].[Id]
INNER JOIN [Accounts] AS [o.Account] ON [o].[AccountId] = [o.Account].[Id]
ORDER BY [o].[AccountId], [o.Account].[Name], [o.Account].[Active]
GO
SELECT [q.Opportunity0].[Id], [q.Opportunity0].[Name], [q.Opportunity0].[Active]
FROM [Opportunities] AS [q.Opportunity0]
GO
SELECT [q.Opportunity0].[Id], [q.Opportunity0].[Name], [q.Opportunity0].[Active]
FROM [Opportunities] AS [q.Opportunity0]
GO
SELECT [q.Opportunity0].[Id], [q.Opportunity0].[Name], [q.Opportunity0].[Active]
FROM [Opportunities] AS [q.Opportunity0]
GO
In reality it generates on query for each opportunity, but I left that out for brevity sake. In my opinion EF should not generate a separate query for each quote. In fact if I comment out the .Name and .Active key parameters in the query as shown below:
group q by new
{
Id = q.Opportunity.Id,
// Name = q.Opportunity.Name,
// Active = q.Opportunity.Active
}
and comment out the correspond variables in the select clause it generates much cleaner sql.
SELECT [o].[Id], [o].[ARRValue], [o].[AccountId], ...
FROM [Quotes] AS [o]
LEFT JOIN [Opportunities] AS [o.Opportunity] ON [o].[OpportunityId] = [o.Opportunity].[Id]
INNER JOIN [Accounts] AS [o.Account] ON [o].[AccountId] = [o.Account].[Id]
ORDER BY [o].[AccountId], [o.Account].[Name], [o.Account].[Active]
GO
The reason I am confused is that .Name and .Active are in the exact same object, they are grouped in the key in the same way as the .Id field, and therefore I don't see why EF would change its behavior just by adding additional group values. Can someone explain the behavior?

Let's take a step back and look at it from a different perspective: If you were to write the SQL query manually, and wanted to fetch all the data required in one query, you would get a lot of duplicate data for the opportunities and account. You could also do this here:
var query = from o in Quotes select o;
var oppQuotes = from o in query
select new
{
AccountId = o.Opportunity.Account.Id,
AccountName = o.Opportunity.Account.Name,
// ... and so on, with all the fields you expect to use.
OpportunityId = o.Opportunity.Id,
OpportunityName = o.Opportunity.Name,
// ... and so on, with all the fields you expect to use.
QuoteId = o.Id,
QuoteName = o.Name,
// ... and again, you get the point.
};
Then, just do an .AsEnumerable() on it, and perform the grouping in your C# code. The database won't be able to optimize anything anyways.
var opportunityQuotes = from q in oppQuotes.AsEnumerable()
group q by new { q.AccountId, q.AccountName }
into accounts
// ... and so on.
For your question, why EF is creating the strange query, I'm at a loss.
In any case, it is always good to be thinking about how YOU would create the sql code to get the data you want most efficiently and not rely on EF to "do the right thing". In many cases it will, in others it will completely blow up in your face. When you want a query, think of the SQL and then translate that to EF code. If you tell it specifically, what you want, you will get it.

Get another class from LINQ to SQL query

I am a little weak in LINQ to SQL so will try to explain my problem.
I have a method as follows (simplified to explain it better):
public static List<ic_ProductData> GetCompleteSimilarProductsWithApplyButton(InfoChoiceAdminDataContext db)
{
var products = (from
p in db.ic_ProductDatas
join proddef in db.ic_ProductDefs on p.ProductDefId equals proddef.ProductDefId
select p
).ToList();
return products;
}
ic_ProductData and ic_ProductDefs are tables in my database
The ic_ProductData class contains a manually created property as:
public ic_ProductDef RelatedProductDef { get; set; }
I want to modify the above LINQ to SQL query so that I can populate this property.
Please note I do not want another call to the database.
Also there are a lot of properties in ic_ProductData so I want to avoid mapping each and every property
Something to the effect of the following (obviously the below is wrong):
public static List<ic_ProductData> GetCompleteSimilarProductsWithApplyButton(InfoChoiceAdminDataContext db)
{
var products = (from
p in db.ic_ProductDatas
join proddef in db.ic_ProductDefs on p.ProductDefId equals proddef.ProductDefId
//trying to change here
select new ic_ProductData
{
//do something with p here so that all the properties of new object gets filled
// avoid mapping of properties here
RelatedProductDef = proddef
}
).ToList();
return products;
}
With my limited knowledge I am stuck here.
Please help!
Thanks in advance!

You can do something like this:
var query = (from p in db.ic_ProductDatas
join proddef in db.ic_ProductDefs on p.ProductDefId equals proddef.ProductDefId
select new
{
ProductData = p,
Def = proddef
}).ToList();
List<ic_ProductData> products = new List<ic_ProductData>();
foreach( var product in query)
{
product.ProductData.RelatedProductDef = product.Def;
products.Add(product);
}
Basicly, you first need to do the one query to the database, this returns an anonymous type containing both your product and its Def.
Finally, you loop (in memory, no db-calls!) over these, creating your final objects with their RelatedProductDef properties populated.

Linq to entities - SQL Query - Where list contains object with 2 properties (or more)

Having the following example:
var myIds = db.Table1.Where(x=>x.Prop2 == myFilter).Select(x=>x.Id).ToList();
var results = db.Table2.Where(x=> myIds.Contains(x.T1)).ToList();
This part is straight forward.
However, now I am facing a "slight" change where my "filter list" has 2 properties instead of only one:
// NOTE: for stackoverflow simplification I use a basic query to
// get my "myCombinationObject".
// In reality this is a much more complex case,
// but the end result is a LIST of objects with two properties.
var myCombinationObject = db.Table3.Where(x=>x.Prop3 == myFilter)
.Select(x=> new {
Id1 = x.T1,
Id2 = x.T2
}).ToList();
var myCombinationObjectId1s = myCombinationObject.Select(x=>xId1).ToList();
var myCombinationObjectId2s = myCombinationObject.Select(x=>xId2).ToList();
// step#1 - DB SQL part
var resultsRaw = db.Tables.Where( x=>
myCombinationObjectId1s.Contains(x.Prop1)
|| myCombinationObjectId2s.Contains(x.Prop2))
.ToList();
// step#2 - Now in memory side - where I make the final combination filter.
var resultsFiltered = resultsRaw.Where( x=>
myCombinationObject.Contains(
new {Id1 = x.Prop1, Id2 = x.Prop2 }
).ToList();
My question: is it even possible to merge the step#2 in the step#1 (query in linq to entities) ?

I've managed once to do what you want, however it is pretty hard and requires changing entity model a bit. You need an entity to map type
new {Id1 = x.Prop1, Id2 = x.Prop2 }
So you need enity having 2 properties - Id1 and Id2. If you have one - great, if not then add such entity to your model:
public class CombinationObjectTable
{
public virtual Guid Id1 { get; set; }
public virtual Guid Id2 { get; set; }
}
Add it to your model:
public DbSet<CombinationObjectTable> CombinationObjectTable { get; set; }
Create new migration and apply it database (database will have now additional table CombinationObjectTable). After that you start to build a query:
DbSet<CombinationObjectTable> combinationObjectTable = context.Set<CombinationObjectTable>();
StringBuilder wholeQuery = new StringBuilder("DELETE * FROM CombinationObjectTable");
foreach(var obj in myCombinationObject)
{
wholeQuery.Append(string.Format("INSERT INTO CombinationObjectTable(Id1, Id2) VALUES('{0}', '{1}')", obj.Id1, obj.Id2);
}
wholeQuery.Append(
db.Tables
.Where( x=>
myCombinationObjectId1s.Contains(x.Prop1)
|| myCombinationObjectId2s.Contains(x.Prop2))
.Where( x=>
combinationObjectTable.Any(ct => ct.Id1 == x.Id1 && ct.Id2 == x.Id2)
).ToString();
);
var filteredResults = context.Tables.ExecuteQuery(wholeQuery.ToString());
Thanks to this your main query stays written in linq. If you do not want to add new table to your db this is as well achievable. Add new class CombinationObjectTable to model, generate new migration to add it and afterwards remove code creating that table from migration code. After that apply migration. This way the db schema won't be changed but EF will think that there is CombinationObjectTable in database. Instead of it you will need to create a temporary table to hold data:
StringBuilder wholeQuery = new StringBuilder("CREATE TABLE #TempCombinationObjectTable(Id1 uniqueidentifies, Id2 uniqueidentifier);");
And when you invoke ToString method on your linq query change CombinationObjectTable to #TempCombinationObjectTable:
...
.ToString()
.Replace("CombinationObjectTable", "#TempCombinationObjectTable")
Other thing worth considering would be using query parameters to pass values in INSERT statements instead of just including them in query yourself - this is of course achievable with EF as well. This solution is not fully ready to apply, rather some hint in which direction you may go for the solution.

Can you do something like this:
var result=
db.Tables
.Where(t=>
db.Table3
.Where(x=>x.Prop3 == myFilter)
.Any(a=>a.T1==t.Prop1 || a.T2==t.Prop2)
).ToList();

If you simply want to avoid the intermediate result (and also creating a second intermediary list) you can do the following
var resultsFiltered = db.Tables.Where( x=>
myCombinationObjectId1s.Contains(x.Prop1)
|| myCombinationObjectId2s.Contains(x.Prop2))
.AsEnumerable() // everything past that is done in memory but isn't materialized immediately, keeping the streamed logic of linq
.Where( x=>
myCombinationObject
.Contains(new {Id1 = x.Prop1, Id2 = x.Prop2 })
.ToList();

How to add to linq2sql entity set for master detail if I can't explicitly define the detail?

Maybe I'm going about this the wrong way...
I have an Order table and an OrderItem table. I create a new Order using linq2sql generated classes.
I then attempt to get all orderable items out of my database using a query that goes after various tables.
I try to then create a new list of OrderItem from that query, but it squawks that I can't explicitly create the object.
Explicit construction of entity type OrderItem in query is not allowed.
Here is the query:
return (from im in dc.MasterItems
join c in dc.Categories
on im.CATEGORY equals c.CATEGORY1
select new OrderItem()
{
OrderItemId = im.ItemId
});
The idea is to populate the database with all orderable items when a new order is created, and then display them in a grid for updates. I'm taking the results of that query and attempting to use AddRange on Order.OrderItems
Is there a proper strategy for accomplishing this using linq2sql?
Thanks in advance for your help.

From my understanding of L2S, I don't think you can use explicit construction (in other words new SomeObj() { ... }) in a query because you aren't enumerating the results yet. In other words, the query has just been built, so how are you supposed to do this:
SELECT new OrderItem() FROM MasterItems im JOIN Categories c on c.CATEGORY1 = im.CATEGORY
This is what you're trying to do, which doesn't work because you can't return a POCO (unless you join the OrderItem back somehow and do OrderItem.* somewhere). Ultimately, what you would have to do is just enumerate the collection (either in a foreach loop or by calling ToList()) on the query first and then build your OrderItem objects.
var query = (from im in dc.MasterItems
join c in dc.Categories
on im.CATEGORY equals c.CATEGORY1
select new { MasterItem = im, Category = c});
List<OrderItem> returnItems = new List<OrderItem>();
foreach(var item in query)
{
returnItems.Add(new OrderItem() { OrderItemId = item.MasterItem.ItemId });
}
return returnItems;
OR
return (from im in dc.MasterItems
join c in dc.Categories
on im.CATEGORY equals c.CATEGORY1
select new { MasterItem = im, Category = c})
.ToList()
.Select(tr => new OrderItem() { OrderItemId = tr.MasterItem.ItemId });
Try that out and let me know if that helps.

Expand the order class by creating a partial file where that class OrderItem now has property(ies) which lend itself to business logic needs, but don't need to be saved off to the database.
public partial class OrderItem
{
public int JoinedOrderItemId { get; set; }
public bool HasBeenProcessed { get; set; }
}

LINQ query with SELECT and two GROUP-BY condition

What's the equivalent LINQ instruction for a Datatable of the following SQL query:
SELECT code_direction, count(TP) AS CN
FROM table1
WHERE cod_time = 'A011'
GROUP BY TP,code_direction;
and how to get the result into a new datatable?
I tried to convert it but I there're some errors. Someone could take a look on this:
var query = from t in table1.AsEnumerable()
group t by new { t.TP, t.code_direction }
into grp
select new
{
grp.Key.code_direction,
CN = grp.Count(t.TP)
};
foreach (var x in query)
{
Console.Write(x.code_direction);
Console.Write(x.CN);
}

As far as your first question goes. The LINQ equivalent of the SQL query is:
var query = from t in table1.AsEnumerable()
where t.cod_time == "A011"
group t by new { t.TP, t.code_direction }
into grp
select new
{
grp.Key.code_direction,
CN = grp.Count()
};
Note that you don't have to pass any argument to grp.Count(). (For the obvious reason that in SQL COUNT(TP) is the same as COUNT(*), i.e. just count the number of rows. The story would be different if you'd use COUNT(DISTINCT TP) or similar.)
As far as the second question goes, if your query just returned an IEnumerable<T> where T is DataRow (i.e. a query like table1.AsEnumerable().Where(r => r.cod_time == "A011")) then you could just the DataTableExtensions.CopyToDataTable extension method. As your query returns an anonymous type however, you will have to follow these instructions found on MSDN.

I Have been using LINQ to work on a JSON object returned from a remote sharepoint web service. I have posted this because most of the answers I found online were slightly different from what I needed.
a json list of daily activities is returned from a remote sharepoint list & is then summarised using LINQ
The simplified version of a custom object definition is shown below( & which is defined in the models area of an MVC application)
public class MyCustomObjectList
{
public string eventdate { get; set; }
public string userid { get; set; }
public string action { get; set; }
}
The JSON object is serialised into a MyCustomObjectList array.
var customobject = serializer.Deserialize<MyCustomObjectList>(jsonobject);
I wanted to work out how many actions of each type happened on a given day. NB eventdate is stored as a string in format yyyy-mm-dd hh:MM:ss. This was to simplify conversions between c#, JSON & Jquery ( where required I create DateTime objects elsewhere in the code using the
eventdate.
Some will argue this is inefficient, but I prefer to split processes into a sequential set of really simple operations, for the sake of easier debugging & to help other people follow my code. Thats why there are 2 Linq queries .
querya strips out the time component from the eventdate This ensures our later grouping happens by day, & not by second. To be doubly sure that there is no caching, I create it in a new field called actionday. I also rename action to activity, because intellisense was getting confused!! The other columns are copied as is.
var querya =
from c in customobject.rows
select new { actionday = c.eventdate.Substring(0, 10), activity = c.action, c.userid,
c.eventdate };
/* queryb produces a grouped count of querya, grouped on actionday & activity, creating new columns actionkey,ActionCount,Dte,action & DetailList ( which is a summary for debugging purposes)
*/
var queryb=
from p in querya group p by new { p.actionday, p.activity} into idGroup
actionkey = idGroup.Key,
ActionCount = idGroup.Count(),
Dte = idGroup.Key.actionday,
action = idGroup.Key.activity,
DetailList = idGroup
};
Here’s a version that sumarises by 3 columns
var queryc = from p in querya
group p by new { p.actionday, p.userid, p.activity} into idGroup
select new
{
actionday = idGroup.Key,
ActionCount = idGroup.Count(),
userid = idGroup.Key.userid,
Dte = idGroup.Key.actionday,
action = idGroup.Key.activity,
DetailList = idGroup
};