We Keep Coding

LINQ - Creating Subquery based on Where clause

I'm seeing behavior in LINQ that I just don't understand and I really want to prevent it from happening because it's causing my queries to run much slower than they should.
The following C# code produces the results I'm expecting but when I add an additional dynamic filter to it things start to go south.
resultChildrenTC = resultChildrenTC.Where(x => x.parentId != 0 && x.clientId == this.clientId);
var me = resultChildrenTC.ToList();
Good results from C# above, and what I would expect.
...
CASE WHEN (1 = [Extent12].[accept_revisions_ind]) THEN 1 ELSE 0 END AS [C14],
[Extent4].[client_id] AS [client_id]
FROM [dbo].[automation_sequence_executions] AS [Extent1]
INNER JOIN [dbo].[automation_sequence_status] AS [Extent2] ON [Extent1].[automation_sequence_status_id] = [Extent2].[id]
INNER JOIN [dbo].[automation_sequences] AS [Extent3] ON [Extent1].[automation_sequence_id] = [Extent3].[id]
INNER JOIN [dbo].[project] AS [Extent4] ON [Extent3].[project_id] = [Extent4].[id]
LEFT OUTER JOIN [dbo].[automation_sequence_execution_results] AS [Extent5] ON [Extent1].[id] = [Extent5].[auto_seq_exec_id]
LEFT OUTER JOIN [dbo].[automation_sequence_test_case_status] AS [Extent6] ON [Extent1].[runtime_case_grp_status] = [Extent6].[id]
LEFT OUTER JOIN [dbo].[users] AS [Extent7] ON [Extent1].[executed_by_id] = [Extent7].[id]
INNER JOIN [dbo].[users] AS [Extent8] ON [Extent3].[created_by_id] = [Extent8].[id]
LEFT OUTER JOIN [dbo].[users] AS [Extent9] ON [Extent3].[last_modified_by_id] = [Extent9].[id]
LEFT OUTER JOIN [dbo].[machines] AS [Extent10] ON [Extent1].[machine_id] = [Extent10].[id]
LEFT OUTER JOIN [dbo].[execution_sync_queue] AS [Extent11] ON ([Extent1].[id] = [Extent11].[execution_id]) AND (0 = [Extent11].[status])
LEFT OUTER JOIN [dbo].[execution_schedule] AS [Extent12] ON [Extent1].[schedule_id] = [Extent12].[id]
WHERE (0 <> (CASE WHEN ([Extent1].[parent_group_exec_id] IS NULL) THEN 0 ELSE [Extent1].[parent_group_exec_id] END)) AND (1 = [Extent4].[status]) AND ([Extent4].[client_id] = #p__linq__0)',N'#p__linq__0 int',#p__linq__0=0
Now once I edit the C# code from above to take in the additional filter things change...
resultChildrenTC = resultChildrenTC.Where(x => x.parentId != 0 && x.clientId == this.clientId).Where(filterExpression);
var me = resultChildrenTC.ToList();
When this runs LINQ decides to create a subquery. This is a big deal because later on I need to run an EXISTS on this query and would like to extend on this filter even more but because it's always creating this random subquery it breaks up my filtering which causes poor performance. What gives and is there a way to turn this off?
[Filter1].[client_id1] AS [client_id]
FROM (SELECT [Extent1].[id] AS [id1], [Extent1].[machine_id] AS [machine_id], [Extent1].[executed_by_id] AS [executed_by_id], [Extent1].[external_test_mgmt_id] AS [external_test_mgmt_id1], [Extent1].[schedule_id] AS [schedule_id], [Extent1].[parent_group_exec_id] AS [parent_group_exec_id], [Extent1].[patriarch_id] AS [patriarch_id], [Extent1].[runtime_case_grp_status] AS [runtime_case_grp_status], [Extent1].[execution_start_time] AS [execution_start_time], [Extent2].[id] AS [id3], [Extent2].[name] AS [name2], [Extent2].[fail_alert_ind] AS [fail_alert_ind], [Extent2].[hold_alert_ind] AS [hold_alert_ind], [Extent2].[complete_ind] AS [complete_ind], [Extent3].[id] AS [id2], [Extent3].[name] AS [name1], [Extent3].[created_by_id] AS [created_by_id], [Extent3].[last_modified_by_id] AS [last_modified_by_id], [Extent3].[case_group_ind] AS [case_group_ind], [Extent4].[name] AS [name3], [Extent4].[client_id] AS [client_id1]
FROM (SELECT [Extent1].[id] AS [id1], [Extent1].[machine_id] AS [machine_id], [Extent1].[executed_by_id] AS [executed_by_id], [Extent1].[external_test_mgmt_id] AS [external_test_mgmt_id1], [Extent1].[schedule_id] AS [schedule_id], [Extent1].[parent_group_exec_id] AS [parent_group_exec_id], [Extent1].[patriarch_id] AS [patriarch_id], [Extent1].[runtime_case_grp_status] AS [runtime_case_grp_status], [Extent1].[execution_start_time] AS [execution_start_time], [Extent2].[id] AS [id3], [Extent2].[name] AS [name2], [Extent2].[fail_alert_ind] AS [fail_alert_ind], [Extent2].[hold_alert_ind] AS [hold_alert_ind], [Extent2].[complete_ind] AS [complete_ind], [Extent3].[id] AS [id2], [Extent3].[name] AS [name1], [Extent3].[created_by_id] AS [created_by_id], [Extent3].[last_modified_by_id] AS [last_modified_by_id], [Extent3].[case_group_ind] AS [case_group_ind], [Extent4].[name] AS [name3], [Extent4].[client_id] AS [client_id1]
FROM [dbo].[automation_sequence_executions] AS [Extent1]
INNER JOIN [dbo].[automation_sequence_status] AS [Extent2] ON [Extent1].[automation_sequence_status_id] = [Extent2].[id]
INNER JOIN [dbo].[automation_sequences] AS [Extent3] ON [Extent1].[automation_sequence_id] = [Extent3].[id]
INNER JOIN [dbo].[project] AS [Extent4] ON [Extent3].[project_id] = [Extent4].[id]
WHERE (0 <> (CASE WHEN ([Extent1].[parent_group_exec_id] IS NULL) THEN 0 ELSE [Extent1].[parent_group_exec_id] END)) AND ([Extent3].[name] LIKE ''%UAT%'') AND (1 = [Extent4].[status]) ) AS [Filter1]
LEFT OUTER JOIN [dbo].[automation_sequence_execution_results] AS [Extent5] ON [Filter1].[id1] = [Extent5].[auto_seq_exec_id]
LEFT OUTER JOIN [dbo].[automation_sequence_test_case_status] AS [Extent6] ON [Filter1].[runtime_case_grp_status] = [Extent6].[id]
INNER JOIN [dbo].[users] AS [Extent7] ON [Filter1].[executed_by_id] = [Extent7].[id]
INNER JOIN [dbo].[users] AS [Extent8] ON [Filter1].[created_by_id] = [Extent8].[id]
INNER JOIN [dbo].[users] AS [Extent9] ON [Filter1].[last_modified_by_id] = [Extent9].[id]
LEFT OUTER JOIN [dbo].[machines] AS [Extent10] ON [Filter1].[machine_id] = [Extent10].[id]
LEFT OUTER JOIN [dbo].[execution_sync_queue] AS [Extent11] ON (0 = [Extent11].[status]) AND ([Filter1].[id1] = [Extent11].[execution_id])
LEFT OUTER JOIN [dbo].[execution_schedule] AS [Extent12] ON [Filter1].[schedule_id] = [Extent12].[id]
WHERE [Filter1].[client_id1] = #p__linq__0',N'#p__linq__0 int',#p__linq__0=0
UPDATE - Where clause impact
This is occurring whenever I perform a LIKE on a string field. It seems like Contains is creating this odd behavior.
Subquery:
resultChildrenTC = resultChildrenTC.Where(x => x.parentId != 0 && x.clientId == this.clientId).Where(x=>x.Name.Contains("wes"));
No subquery:
resultChildrenTC = resultChildrenTC.Where(x => x.parentId != 0 && x.clientId == this.clientId).Where(x=>x.Name == "wes");
UPDATE - Possible reasoning
I've created a number of tickets that may all be related, which I didn't know at the time.
From what I've learned about LINQ-to-entities it's a mistake to use the included FK in your queries. A more detailed view into this can be found in the following post:
LINQ - SQL Script Changing based on JOIN Order

If you already know that you would run the same query multi times, you can just cache it somehow to reduce the server's workload.
var mycatch = resultChildrenTC.Where(x => x.parentId != 0 && x.clientId == this.clientId).ToList();
var result1 = mycache .Where(filterExpression1)
var result2 = mycache .Where(filterExpression2)
var result3 = mycache .Where(filterExpression2)

Linq to SQL assistance with joins

Please can you help me translate the following SQL query into LINQ
Its mainly the joins that i am having issues with.
The result will be LINQ to SQL code which gets executed.
many thanks
SELECT * FROM unit INNER JOIN
unit_measurement_total ON unit.prime_measurement_uri = unit_measurement_total.uri RIGHT OUTER JOIN
property_expense_schedule
INNER JOIN
unit_apportionment ON property_expense_schedule.uri = unit_apportionment.property_expense_schedule_uri
INNER JOIN
unit_apportionment_date ON unit_apportionment.uri = unit_apportionment_date.unit_apportionment_uri ON
unit_measurement_total.property_ref = unit_apportionment.property_ref
WHERE (property_expense_schedule.property_ref = ...)
This is the LINQ I have at the moment, but its not generating the same results as the SQL query. So i am going wrong with the joins and the right outer joins.
var query = (from units in context.units
join unitmestot in context.unit_measurement_total on units.prime_measurement_uri equals unitmestot.uri
from pes in context.property_expense_schedule
join unitapp in context.unit_apportionment on new { A = pes.uri, B = unitmestot.property_ref, C = unitmestot.unit_ref } equals new { A = unitapp.property_expense_schedule_uri, B = unitapp.property_ref, C = unitapp.unit_ref}
join unitappdate in context.unit_apportionment_date on unitapp.uri equals unitappdate.unit_apportionment_uri
select new Apportionment()
{
PropertyRef = units.property_ref.ToString(),
ScheduleName = pes.name,
ScheduleRef = pes.#ref.ToString(),
PropertyExpenseScheduleUri = pes.uri,
UnitRef = units.#ref.ToString(),
UnitName = "(" + units.#ref.ToString() + ")" + units.name,
ObseleteUnit = units.obsolete_unit,
ApportionmentPercentage = unitappdate.apportionment_percentage,
ToDate = unitappdate.to_date,
MeasurementBasis = unitmestot.measurement_basis,
MeasuredIn = unitmestot.measured_in,
MeasurementImperialTotal = unitmestot.measurement_imperial_total,
MeasurementMetricTotal = unitmestot.measurement_metric_total
}).Where(filter);
the latest code you supplied generates the following SQL and returns 0 rows.
SELECT [Extent1].[uri] AS [uri], CASE WHEN ([Join3].[property_ref1] IS NULL) THEN N'' ELSE CAST( [Join3].[property_ref1] AS nvarchar(max)) END AS [C1], [Extent3].[name] AS [name], CAST( [Extent3].[ref] AS nvarchar(max)) AS [C2], [Extent3].[uri] AS [uri1], CASE WHEN ([Join3].[ref] IS NULL) THEN N'' ELSE CAST( [Join3].[ref] AS nvarchar(max)) END AS [C3], N'(' + CASE WHEN (CASE WHEN ([Join3].[ref] IS NULL) THEN N'' ELSE CAST( [Join3].[ref] AS nvarchar(max)) END IS NULL) THEN N'' WHEN ([Join3].[ref] IS NULL) THEN N'' ELSE CAST( [Join3].[ref] AS nvarchar(max)) END + N')' + CASE WHEN ([Join3].[name] IS NULL) THEN N'' ELSE [Join3].[name] END AS [C4], [Join3].[obsolete_unit] AS [obsolete_unit], [Extent2].[apportionment_percentage] AS [apportionment_percentage], [Extent2].[to_date] AS [to_date], CASE WHEN ([Join3].[measurement_basis] IS NULL) THEN N'' ELSE [Join3].[measurement_basis] END AS [C5], CASE WHEN ([Join3].[measured_in] IS NULL) THEN N'' ELSE [Join3].[measured_in] END AS [C6], [Join3].[measurement_imperial_total] AS [measurement_imperial_total], [Join3].[measurement_metric_total] AS [measurement_metric_total] FROM [tramps].[unit_apportionment] AS [Extent1] INNER JOIN [tramps].[unit_apportionment_date] AS [Extent2] ON [Extent1].[uri] = [Extent2].[unit_apportionment_uri] INNER JOIN [tramps].[property_expense_schedule] AS [Extent3] ON [Extent1].[property_expense_schedule_uri] = [Extent3].[uri] LEFT OUTER JOIN (SELECT [Extent4].[property_ref] AS [property_ref2], [Extent4].[measurement_basis] AS [measurement_basis], [Extent4].[measured_in] AS [measured_in], [Extent4].[measurement_imperial_total] AS [measurement_imperial_total], [Extent4].[measurement_metric_total] AS [measurement_metric_total], [Extent5].[property_ref] AS [property_ref1], [Extent5].[ref] AS [ref], [Extent5].[name] AS [name], [Extent5].[obsolete_unit] AS [obsolete_unit] FROM [tramps].[unit_measurement_total] AS [Extent4] INNER JOIN [tramps].[unit] AS [Extent5] ON [Extent4].[uri] = [Extent5].[prime_measurement_uri] ) AS [Join3] ON [Extent1].[property_ref] = [Join3].[property_ref2] WHERE (N'101329' = (CASE WHEN ([Join3].[property_ref1] IS NULL) THEN N'' ELSE CAST( [Join3].[property_ref1] AS nvarchar(max)) END)) AND ( NOT (('Y' = [Join3].[obsolete_unit]) AND ([Join3].[obsolete_unit] IS NOT NULL)))

LINQ does not support Right Outer Join, so it should be simulated by swapping the left and right parts and performing a Left Outer Join (which is not so natural, but at least the pattern in known - join clause (C# Reference)).
With that being said, I think the equivalent LINQ query should be something like this:
from unitapp in context.unit_apportionment
join unitappdate in context.unit_apportionment_date on unitapp.uri equals unitappdate.unit_apportionment_uri
join pes in context.property_expense_schedule on unitapp.property_expense_schedule_uri equals pes.uri
join unitmestot in context.unit_measurement_total on unitapp.property_ref equals unitmestot.property_ref
into unitapp_unitmesstot from unitmestot in unitapp_unitmesstot.DefaultIfEmpty() // Left Outer Join
join units in context.units on unitmestot.uri equals units.prime_measurement_uri
select new Apportionment()
{
// ...
}
EDIT: Looks like EF is generating additional criteria for the inner join right after the left outer join which is causing differences with the original SQL query. In such case you can try grouping the right part into a subquery in attempt to change the join order:
from unitapp in context.unit_apportionment
join unitappdate in context.unit_apportionment_date on unitapp.uri equals unitappdate.unit_apportionment_uri
join pes in context.property_expense_schedule on unitapp.property_expense_schedule_uri equals pes.uri
join right in (
from unitmestot in context.unit_measurement_total
join units in context.units on unitmestot.uri equals units.prime_measurement_uri
select new { unitmestot, units }
) on unitapp.property_ref equals right.unitmestot.property_ref
into outerJoin from right in outerJoin.DefaultIfEmpty() // Left Outer Join
let unitmestot = right.unitmestot
let units = right.units
select new Apportionment()
{
// ...
}

Creating Linq from SQL with OrderBy and GroupBy

I have the following table structure.
TableA TableB TableC
- MID - PID - PID
- NAME - INIT_DATE - MID
This is the SQL Query that I need to translate into Linq
SELECT TOP 10 TableA.NAME,
COUNT(TableB.INIT_DATE) AS [TOTALCOUNT]
FROM TableC
INNER JOIN TableA ON TableC.MID = TableA.MID
LEFT OUTER JOIN TableB ON TableC.PID = TableB.PID
GROUP BY TableA.NAME
ORDER BY [TOTALCOUNT] DESC
I tried to reproduce the above query with this Linq query:
iqModel = (from tableC in DB.TableC
join tableA in DB.TableA on tableC.MID equals tableA.MID
select new { tableC, tableA } into TM
join tableB in DB.TableB on TM.tableC.PID equals J.PID into TJ
from D in TJ.DefaultIfEmpty()
select new { TM, D } into MD
group MD by MD.TM.tableA.NAME into results
let TOTALCOUNT = results.Select(item=>item.D.INIT_DATE).Count()
orderby TOTALCOUNT descending
select new SelectListItem
{
Text = results.Key.ToString(),
Value = TOTALCOUNT.ToString()
}).Take(10);
But I think I am doing something wrong.
The Output of the LINQ and SQL is not same. I think up to JOIN or GROUPBY it is Correct.
EDIT :-
I have also tried the following Linq query but still it's not working correctly.
var iqModel = (from c in DB.TableC
join a in DB.TableA on c.MID equals a.MID
join b in DB.b on c.PID equals b.PID into b_join
from b in b_join.DefaultIfEmpty()
select new SelectListItem { Text = a.NAME, Value = b.INIT_DATE != null ? b.INIT_DATE.ToString() : string.Empty });
var igModel = iqModel.GroupBy(item => item.Text);
var result = igModel.OrderByDescending(item => item.Select(r => r.Value).Count());
I want to understand what am I doing wrong and how can it be fixed.
I am newbie to LINQ to SQL I think in above LINQ I really made it complicated by adding more select.

I think the difference is caused by the fact that the SQL COUNT(field) function does not include NULL values. There is no direct equivalent construct in LINQ, but it could be simulated with Count(e => e.Field != null) or like this (which seems to produce better SQL):
var query =
(from a in db.TableA
join c in db.TableC on a.MID equals c.MID
join b in db.TableB on c.PID equals b.PID into joinB
from b in joinB.DefaultIfEmpty()
group b by a.Name into g
let TOTALCOUNT = g.Sum(e => e.INIT_DATE != null ? 1 : 0)
orderby TOTALCOUNT descending
select new SelectListItem { Text = g.Key, Value = TOTALCOUNT }
).Take(10);
which generates the following SQL
SELECT TOP (10)
[Project1].[C2] AS [C1],
[Project1].[Name] AS [Name],
[Project1].[C1] AS [C2]
FROM ( SELECT
[GroupBy1].[A1] AS [C1],
[GroupBy1].[K1] AS [Name],
1 AS [C2]
FROM ( SELECT
[Join2].[K1] AS [K1],
SUM([Join2].[A1]) AS [A1]
FROM ( SELECT
[Extent1].[Name] AS [K1],
CASE WHEN ([Extent3].[INIT_DATE] IS NOT NULL) THEN 1 ELSE 0 END AS [A1]
FROM [dbo].[TableAs] AS [Extent1]
INNER JOIN [dbo].[TableCs] AS [Extent2] ON [Extent1].[MID] = [Extent2].[MID]
LEFT OUTER JOIN [dbo].[TableBs] AS [Extent3] ON [Extent2].[PID] = [Extent3].[PID]
) AS [Join2]
GROUP BY [K1]
) AS [GroupBy1]
) AS [Project1]
ORDER BY [Project1].[C1] DESC

I assume, that you not see "group by" command at resulting query, instead of it "distinct" command is used. Am I right?
First query makes distinct by TableA.NAME and then calculates COUNT(TableB.INIT_DATE) with the help of subquery like this:
select distinct1.Name, (select count() from *join query* where Name = distinct1.Name)
from (select distinct Name from *join query*) as distinct1
If so, not worry about it. Because conversion from linq to real t-sql script sometimes very unpredictable (you can not force them to be equal, only when query is very simple), but both queries are equivalent one to another and return same results (compare them to make sure).

LINQ query optimization for slow grouping

I have a LINQ query that gets data via Entity Framework Code First from an SQL database. This works, but it works very very slow.
This is the original query:
var tmpResult = from mdv in allMetaDataValues
where mdv.Metadata.InputType == MetadataInputType.String && mdv.Metadata.ShowInFilter && !mdv.Metadata.IsHidden && !string.IsNullOrEmpty(mdv.ValueString)
group mdv by new
{
mdv.ValueString,
mdv.Metadata
} into g
let first = g.FirstOrDefault()
select new
{
MetadataTitle = g.Key.Metadata.Title,
MetadataID = g.Key.Metadata.ID,
CollectionColor = g.Key.Metadata.Collection.Color,
CollectionID = g.Key.Metadata.Collection.ID,
MetadataValueCount = 0,
MetadataValueTitle = g.Key.ValueString,
MetadataValueID = first.ID
};
This is the generated SQL from the original query:
{SELECT
0 AS [C1],
[Project4].[Title] AS [Title],
[Project4].[ID] AS [ID],
[Extent9].[Color] AS [Color],
[Project4].[Collection_ID] AS [Collection_ID],
[Project4].[ValueString] AS [ValueString],
[Project4].[C1] AS [C2]
FROM (SELECT
[Project2].[ValueString] AS [ValueString],
[Project2].[ID] AS [ID],
[Project2].[Title] AS [Title],
[Project2].[Collection_ID] AS [Collection_ID],
(SELECT TOP (1)
[Filter4].[ID1] AS [ID]
FROM ( SELECT [Extent6].[ID] AS [ID1], [Extent6].[ValueString] AS [ValueString], [Extent7].[Collection_ID] AS [Collection_ID1], [Extent8].[ID] AS [ID2], [Extent8].[InputType] AS [InputType], [Extent8].[ShowInFilter] AS [ShowInFilter], [Extent8].[IsHidden] AS [IsHidden1]
FROM [dbo].[MetadataValue] AS [Extent6]
LEFT OUTER JOIN [dbo].[Media] AS [Extent7] ON [Extent6].[Media_ID] = [Extent7].[ID]
INNER JOIN [dbo].[Metadata] AS [Extent8] ON [Extent6].[Metadata_ID] = [Extent8].[ID]
WHERE ( NOT (([Extent6].[ValueString] IS NULL) OR (( CAST(LEN([Extent6].[ValueString]) AS int)) = 0))) AND ([Extent7].[IsHidden] <> cast(1 as bit))
) AS [Filter4]
WHERE (2 = CAST( [Filter4].[InputType] AS int)) AND ([Filter4].[ShowInFilter] = 1) AND ([Filter4].[IsHidden1] <> cast(1 as bit)) AND ([Filter4].[Collection_ID1] = #p__linq__0) AND (([Project2].[ValueString] = [Filter4].[ValueString]) OR (([Project2].[ValueString] IS NULL) AND ([Filter4].[ValueString] IS NULL))) AND (([Project2].[ID] = [Filter4].[ID2]) OR (1 = 0))) AS [C1]
FROM ( SELECT
[Distinct1].[ValueString] AS [ValueString],
[Distinct1].[ID] AS [ID],
[Distinct1].[Title] AS [Title],
[Distinct1].[Collection_ID] AS [Collection_ID]
FROM ( SELECT DISTINCT
[Filter2].[ValueString] AS [ValueString],
[Filter2].[ID3] AS [ID],
[Filter2].[InputType1] AS [InputType],
[Filter2].[Title1] AS [Title],
[Filter2].[ShowInFilter1] AS [ShowInFilter],
[Filter2].[IsHidden2] AS [IsHidden],
[Filter2].[Collection_ID2] AS [Collection_ID]
FROM ( SELECT [Filter1].[ValueString], [Filter1].[Collection_ID3], [Filter1].[IsHidden3], [Filter1].[ID3], [Filter1].[InputType1], [Filter1].[Title1], [Filter1].[ShowInFilter1], [Filter1].[IsHidden2], [Filter1].[Collection_ID2]
FROM ( SELECT [Extent1].[ValueString] AS [ValueString], [Extent2].[Collection_ID] AS [Collection_ID3], [Extent4].[IsHidden] AS [IsHidden3], [Extent5].[ID] AS [ID3], [Extent5].[InputType] AS [InputType1], [Extent5].[Title] AS [Title1], [Extent5].[ShowInFilter] AS [ShowInFilter1], [Extent5].[IsHidden] AS [IsHidden2], [Extent5].[Collection_ID] AS [Collection_ID2]
FROM [dbo].[MetadataValue] AS [Extent1]
LEFT OUTER JOIN [dbo].[Media] AS [Extent2] ON [Extent1].[Media_ID] = [Extent2].[ID]
INNER JOIN [dbo].[Metadata] AS [Extent3] ON [Extent1].[Metadata_ID] = [Extent3].[ID]
LEFT OUTER JOIN [dbo].[Metadata] AS [Extent4] ON [Extent1].[Metadata_ID] = [Extent4].[ID]
LEFT OUTER JOIN [dbo].[Metadata] AS [Extent5] ON [Extent1].[Metadata_ID] = [Extent5].[ID]
WHERE ( NOT (([Extent1].[ValueString] IS NULL) OR (( CAST(LEN([Extent1].[ValueString]) AS int)) = 0))) AND ([Extent2].[IsHidden] <> cast(1 as bit)) AND (2 = CAST( [Extent3].[InputType] AS int)) AND ([Extent3].[ShowInFilter] = 1)
) AS [Filter1]
WHERE [Filter1].[IsHidden3] <> cast(1 as bit)
) AS [Filter2]
WHERE [Filter2].[Collection_ID3] = #p__linq__0
) AS [Distinct1]
) AS [Project2] ) AS [Project4]
LEFT OUTER JOIN [dbo].[Collection] AS [Extent9] ON [Project4].[Collection_ID] = [Extent9].[ID]}
If we remove the "let first = g.FirstOrDefault()" and change "MetadataValueID = first.ID" to "MetadataValueID = 0" so that we just have a fixed ID = 0 for testing purposes, then the data loads very fast and the generated query itself is half the size compared to the original
So it seems that this part is making the query very slow:
let first = g.FirstOrDefault()
...
MetadataValueID = first.ID
};
How can this be rewritten?
If I try to rewrite the code, it is still slow:
MetadataValueID = g.Select(x => x.ID).FirstOrDefault()
or
let first = g.Select(x => x.ID).FirstOrDefault()
...
MetadataValueID = first
};
Any suggestions?

Using EF I have allways felt that it has problems efficiently translating stuff like g.Key.Metadata.Collection, so I try to join more explicitly and to include only fields, that are neccessary for your result. You can use include instead of join using repository pattern.
Then your query would look like this:
from mdv in allMetaDataValues.Include("Metadata").Include("Metadata.Collection")
where mdv.Metadata.InputType == MetadataInputType.String &&
mdv.Metadata.ShowInFilter &&
!mdv.Metadata.IsHidden &&
!string.IsNullOrEmpty(mdv.ValueString)
group mdv by new
{
MetadataID = mdv.Metadata.ID,
CollectionID = mdv.Metadata.Collection.ID,
mdv.Metadata.Title,
mdv.Metadata.Collection.Color,
mdv.ValueString
} into g
let first = g.FirstOrDefault().ID
select new
{
MetadataTitle = g.Key.Title,
MetadataID = g.Key.MetadataID,
CollectionColor = g.Key.Color,
CollectionID = g.Key.CollectionID,
MetadataValueCount = 0,
MetadataValueTitle = g.Key.ValueString,
MetadataValueID = first
}
Good tool for playing with linq is LinqPad.
The problem is also that:
let first = g.FirstOrDefault().ID
cannot be easily translated to SQL see this answer. But this rewrite simplifies the underlying query for it at least. It remains to me unclear, why you need first ID from a set without using orderby.
It could be rewriten like this:
let first = (from f in allMetaDataValues
where f.Metadata.ID == g.Key.MetadataID &&
f.ValuesString == g.Key.ValuesString select f.ID)
.FirstOrDefault()
This way you do not let EF write the query for you and you can specify exactly how to do the select.
To speed up the query you can also consider adding indexes to database according to the generated query - namely index using both colums used in where clause of this let first query.

Try the following solution.
Replace FirstOrDefault() with .Take(1). FirstOrDefault() is not lazy loaded.
var tmpResult = from mdv in allMetaDataValues
where mdv.Metadata.InputType == MetadataInputType.String && mdv.Metadata.ShowInFilter && !mdv.Metadata.IsHidden && !string.IsNullOrEmpty(mdv.ValueString)
group mdv by new
{
mdv.ValueString,
mdv.Metadata
} into g
let first = g.Take(1)
select new
{
MetadataTitle = g.Key.Metadata.Title,
MetadataID = g.Key.Metadata.ID,
CollectionColor = g.Key.Metadata.Collection.Color,
CollectionID = g.Key.Metadata.Collection.ID,
MetadataValueCount = 0,
MetadataValueTitle = g.Key.ValueString,
MetadataValueID = first.ID
};

Joining int column to string column using LINQ

Have tried all the various combinations from quite a few searches, and it still doesn't seem to work. I have the following query:
var r = from a in ctx.DGApprovedLinks
join h in ctx.DGHosts on a.HostID equals h.ID
join c in ctx.DGConfigs on SqlFunctions.StringConvert((double)a.ResponseCode) equals c.SubType
where c.Type == "HTTP Status"
select new
{
a.ID,
a.HostID,
h.URL,
a.SourceURL,
a.TargetURL,
c.Value,
a.ExtFlag
};
but it fails to return any results. I get the following in SQL Profiler:
SELECT
1 AS [C1],
[Extent1].[ID] AS [ID],
[Extent1].[HostID] AS [HostID],
[Extent2].[URL] AS [URL],
[Extent1].[SourceURL] AS [SourceURL],
[Extent1].[TargetURL] AS [TargetURL],
[Extent3].[Value] AS [Value],
[Extent1].[ExtFlag] AS [ExtFlag]
FROM [dbo].[DGApprovedLink] AS [Extent1]
INNER JOIN [dbo].[DGHost] AS [Extent2] ON [Extent1].[HostID] = [Extent2].[ID]
INNER JOIN [dbo].[DGConfig] AS [Extent3] ON ((STR( CAST( [Extent1].[ResponseCode] AS float))) = [Extent3].[SubType]) OR ((STR( CAST( [Extent1].[ResponseCode] AS float)) IS NULL) AND ([Extent3].[SubType] IS NULL))
WHERE N'HTTP Status' = [Extent3].[Type]
Any assistance would be much appreciated.