We Keep Coding

Slow LINQ query because of Where+ToDictionary

I have this query that I need to run:
IEnumerable<MerchantWithParameters> merchants =
from i in (
from m in d.GetTable<Merchant>()
join mtp in d.GetTable<MerchantToParameters>() on m.Id equals mtp.MerchantId into mtps
from mtp in mtps.DefaultIfEmpty()
join cp in d.GetTable<ContextParameters>() on mtp.ContextParametersId equals cp.Id into cps
from cp in cps.DefaultIfEmpty()
select new {Merchant = m, ContextParameter = cp}
)
group i by new { i.Merchant.Id } into ig
select new MerchantWithParameters()
{
Id = ig.Key.Id,
Parameters = ig.Where(g => g.ContextParameter != null).ToDictionary(g => g.ContextParameter.Key, g => g.ContextParameter.Text)
};
For some reason it takes really long time for this query to be completed.
I believe that it has something to do with
Parameters = ig.Where(g => g.ContextParameter != null).ToDictionary(g => g.ContextParameter.Key, g => g.ContextParameter.Text)
Because when I remove this line, query starts to execute really fast.
Could you please show me what am I doing wrong?
UPDATE:
I am using ToList() to extract data from the database.

It is known SQL limitation. You cannot get grouped items, only grouping key or aggregation result. Since you need all records, we can do grouping on the client side, but previously maximally limit retrieved data.
var query =
from m in d.GetTable<Merchant>()
from mtp in d.GetTable<MerchantToParameters>().LeftJoin(mtp => m.Id == mtp.MerchantId)
from cp in d.GetTable<ContextParameters>().LeftJoin(cp => mtp.ContextParametersId == cp.Id)
select new
{
MerchantId = m.Id,
ContextParameterKey = (int?)cp.Key,
ContextParameterText = cp.Text
};
var result =
from q in query.AsEnumerable()
group q by q.MerchantId into g
select new MerchantWithParameters
{
Id = g.Key,
Parameters = g.Where(g => g.ContextParameterKey != null)
.ToDictionary(g => g.ContextParameterKey.Value, g => g.ContextParameterText)
};
var merchants = result.ToList();

joining two from with an expression<func<>>

Is it possible to join two from based on a local expression variable?
ex;
var query = from t in context.table1
from a in context.anothertable1.Where(x => t.id == a.id)
select new {a,t};
on line 2, the Where clause .Where(x => t.id == a.id) how would you move it into an expression?
I know i can do this;
Expression<Func<anothertable1, bool>> test = x => x.field1 == 1;
and It would work here;
var query = from t in context.table1
from a in context.anothertable1
.Where(x => t.id == a.id)
.Where(test)
select new {a,t};
and everything work and the sql query generated is as expected.
I can't figure out how to do the same with the other where.
EDIT
a more complex example, i anonymized it so it might not compile
var listOfMinMaxtable1 = (from n in context.table1.Where(table1Filter)
group n by n.table1_Number into grp
select new MinMaxtable1()
{
table1_Id_Max = grp.Max(x => x.table1_Id),
table1_Id_Min = grp.Min(x => x.table1_Id),
table1_Number = grp.Key
});
var listtable2 = (from t in context.table2
group t by t.table2_Id into grp
select new table2()
{
table2 = grp,
table2_Id = grp.Key
});
var query = from MinMax in listOfMinMaxtable1
//inner join **reference 1**
from table3 in context.table3
.Where(x => x.table_Number == MinMax.table_Number)
.Where(noticeMasterFilter) //a working expression<func<>>
//inner join **reference 2**
from Lasttable1 in context.table1
.Where(x => x.table_Id == MinMax.table_Id_Max)
//left join **reference 3**
from Firsttable1 in context.table1
.Where(x => x.table_Id == MinMax.table_Id_Min)
.Where(firstNoticeFilter) //a working expression<func<>>
.DefaultIfEmpty()
//left join **reference 4**
from Lasttable2 in listtable2
.Where(x => x.table_Id == MinMax.table_Id_Max)
.SelectMany(x => x.table2)
.Where(x => x.table2_Id == 123)
.OrderByDescending(x => x.table_Id)
.Take(1)
.DefaultIfEmpty()
if you find //left join reference 3 in the code above
that where clause; .Where(x => x.table_Id == MinMax.table_Id_Min)
might be sometime; .Where(x => x.table_Id == MinMax.table_Id_Max)
I could just copy/paste the whole from and change the where clause while adding noop pattern (an expression that return false and this make entity framework remove the whole thing so it doesn't affect the generated sql/result) with an expression on both from
for reference(this is noise to the question), the noop expression that i'm talking about is;
Expression<Func<table1, bool>> includeFrom= x => false;
and would be used like
//left join **reference 3**
from Firsttable1 in context.table1
.Where(x => x.table_Id == MinMax.table_Id_Min)
.Where(firstNoticeFilter) //a working expression<func<>>
.Where(includeFrom) //<--- this line make it a noop if the expression stay false
.DefaultIfEmpty()
but I don't want to do this if it's possible to make a custom expression that would go into the .Where()

Instead of creating an expression based on one type, you can create a combined type and use that for your where expression.
Two Table Combined Type
public class TwoTableDto
{
public Table1 t { get; set; }
public Table2 a { get; set; }
}
Query without expression
var query = (from t in context.table1
from a in context.anothertable1
select new TwoTableDto { t = t, a = a })
.Where(x => x.t.id == x.a.id);
Expression
Expression<Func<TwoTableDto, bool>> expr = x => x.t.id == x.a.id;
Query with expression
var query = (from t in context.table1
from a in context.anothertable1
select new TwoTableDto { t = t, a = a })
.Where(expr);

Linq Dynamic Query With Group By

I need extra where clause for my Linq query. For example if customer choose a date filter so i need to date filter to my query etc... When i try to myQuery.Where predicate there is visible just group by's field.
How can i append new where condition to my query.
//for example i need dynamically append o.OrderDate==Datetime.Now or another where clause
var myQuery =(from o in _db.Orders
join l in _db.OrderLines.Where(x => x.ParaBirimi == model.ParaBirimi) on o.orderId equals
l.OrderId
where o.OrderDate.Value.Year == year1
group o by new {o.OrderDate.Value.Month}
into g
select
new
{
Month = g.Key.Month,
Total = g.Select(t => t.OrderLines.Sum(s => s.OrderTotal)).FirstOrDefault()
});

You are too late at the end of the query to add new Where. You have already grouped the data, and projected it, removing nearly all the fields.
Try:
var baseQuery = from o in _db.Orders
join l in _db.OrderLines.Where(x => x.ParaBirimi == model.ParaBirimi) on o.orderId equals l.OrderId
where o.OrderDate.Value.Year == year1
select new { Order = o, OrderLine = l };
if (something)
{
baseQuery = baseQuery.Where(x => x.Order.Foo == "Bar");
}
var myQuery = (from o in baseQuery
group o by new { o.Order.OrderDate.Value.Month }
into g
select
new
{
Month = g.Key.Month,
Total = g.Sum(t => t.OrderLine.OrderTotal)
});
Clearly you can have multiple if. Each .Where() is in && (AND) with the other conditions.
Note how the result of the join is projected in an anonymous class that has two properties: Order and OrderLine

To many nested statements

I have a c# code as follows
var deptSalesQuery =
from d in db.DashboardFigures
join s in outlets.Split(',').Select(x => int.Parse(x)) on d.OutletNo equals s
where (d.TypeOfinformation == "SALES")
group d by new
{
d.Number
} into newGroupedresult
select new DeptSales()
{
Dn = (int)newGroupedresult.Key.Number,
Qs = (double)newGroupedresult.Sum(d => d.Value_4),
Se = (double)newGroupedresult.Sum(d => d.Value_2),
Si = (double)newGroupedresult.Sum(d => d.Value_3)
+ (double)newGroupedresult.Sum(d => d.Value_2)
};
When I pass in Outlets = "1,2,3,4,....all the way upto 110", the software crashes telling me that there are too many nested statements.
Is there any way that I can remove the JOIN and add something onto the WHERE clause to help the situation?
Thanks

As written by juharr, before the query do:
int[] splitted = outlets.Split(',').Select(int.Parse);
and in the query:
where splitted.Contains(d.OutletNo) && d.TypeOfinformation == "SALES"

Linq SelectMany

Hi I am coding my way through the MS 101 linq examples.
The "JoinOperators" are giving me a hard time since I am trying to refactor the query expressions to lambda syntax and vice versa.
Anyway, on example 105 I see this query expression:
var supplierCusts =
from sup in suppliers
join cust in customers on sup.Country equals cust.Country into cs
from c in cs.DefaultIfEmpty() // DefaultIfEmpty preserves left-hand elements that have no matches on the right side
orderby sup.SupplierName
select new
{
Country = sup.Country,
CompanyName = c == null ? "(No customers)" : c.CompanyName,
SupplierName = sup.SupplierName
};
And I tried implementing it as a lambda this way:
// something is not right here because the result keeps a lot of "Join By" stuff in the output below
var supplierCusts =
suppliers.GroupJoin(customers, s => s.Country, c => c.Country, (s, c) => new { Customers = customers, Suppliers = suppliers })
.OrderBy(i => i.Suppliers) // can't reference the "name" field here?
.SelectMany(x => x.Customers.DefaultIfEmpty(), (x, p) => // does the DefaultIfEmpty go here?
new
{
Country = p.Country,
CompanyName = x == null ? "(No customers)" : p.CompanyName,
SupplierName = p // not right: JoinOperators.Program+Customer ... how do I get to supplier level?
});
For some reason I can't access the supplier-level information this way. When I switch out the customers with suppliers I can't access the customer-level information.
Is there some overload of SelectMany() that lets me pull from the field-level of both objects?
Also, I don't understand why the GroupJoin() appears to return an object with 2 collections (suppliers and customers). Isn't it supposed to join them somehow?
I guess I don't understand how GroupJoin() works.

You have wrong result selector in group join, that's where problems started. Here is fixed query:
var supplierCusts =
suppliers
.GroupJoin(customers,
sup => sup.Country,
cust => cust.Country,
(sup, cs) => new { sup, cs })
.OrderBy(x => x.sup.Name)
.SelectMany(x => x.cs.DefaultIfEmpty(), (x, c) =>
new
{
Country = x.sup.Country,
CompanyName = c == null ? "(No customers)" : c.CompanyName,
SupplierName = x.sup.Name
});

If you want to learn translating the query expressions into lambda's, I suggest you check out LinqPad which can do that by default. For example, your query is translated as follows:
Suppliers
.GroupJoin (
Customers,
sup => sup.Country,
cust => cust.Country,
(sup, cs) =>
new
{
sup = sup,
cs = cs
}
)
.SelectMany (
temp0 => temp0.cs.DefaultIfEmpty (),
(temp0, c) =>
new
{
temp0 = temp0,
c = c
}
)
.OrderBy (temp1 => temp1.temp0.sup.CompanyName)
.Select (
temp1 =>
new
{
Country = temp1.temp0.sup.Country,
CompanyName = (temp1.c == null) ? "(No customers)" : temp1.c.CompanyName,
SupplierName = temp1.temp0.sup.CompanyName
}
)
That being said, I typically find SelectMany to be easier to code and maintain using the query syntax instead of the lambda syntax.
The GroupJoin in this example is used to accomplish the left join (via the .DefaultIfEmpty clause).

Try this:
var supplierCusts =
suppliers.GroupJoin(customers, s => s.Country, c => c.Country, (s, c) => new { Supplier = s, Customers = c })
.OrderBy(i => i.Supplier.SupplierName)
.SelectMany(r => r.Customers.DefaultIfEmpty(), (r, c) => new
{
Country = r.Supplier.Country,
CompanyName = c == null ? "(No customers)" : c.CompanyName,
SupplierName = r.Supplier.SupplierName
});