We Keep Coding

Using lambdas LINQ query in C#

I am still in the midst of converting our web app from vb to C# and this complex query, that I just finished converting and have tested and it works properly, needs refactoring.
This section could probably perform a little better if lambdas were used (I've read in numerous places that lambda expressions perform faster)
foreach (var datapersymbol in (from symId in symblist
select (
from q in ctx.HistoricalPriceData.Where(q => q.SymbolId == symId)
join x in ctx.Symbols on q.SymbolId equals x.SymbolId
orderby q.Date descending
select new {q.Date, q.LastPrice, x.Symbol1})).ToList())
As you can see I've added lambdas where I thought appropriate but Im certain they can be in other parts of this expressions as well. I'm still getting up to speed in C# so Im not an expert here.

Assuming you have a navigation property on HistoricalPriceData to Symbols called "Symbol":
foreach(var datapersymbol in ctx
.HistoricalPriceData
.Where(h=>symblist.Contains(h.SymbolId))
.OrderByDescending(h=>h.Date)
.Select(h=>new {h.Date,h.LastPrice,h.Symbol.Symbol1}))
As Rahul pointed out, there is no performance difference between lamda/method syntax and query syntax, however, this will perform slightly better because it drops the .ToList(), which will allow the loop to begin processing as soon as it gets the first record from the database rather than waiting for the entire result (and it doesn't need to create a List). I've also simplified it somewhat by using .Contains instead of a subquery as well, and switched to using navigation properties rather than a explicit join which makes it easier to read/more maintainable.

For the clarity of the code I will rather refactor it like below;
var datapersymbols = from symId in symblist
select (
from q in ctx.HistoricalPriceData.Where(q => q.SymbolId == symId)
join x in ctx.Symbols on q.SymbolId equals x.SymbolId
orderby q.Date descending
select new {q.Date, q.LastPrice, x.Symbol1});
foreach (var datapersymbol in datapersymbols)
{
}
As far as I can see, in this case, there is not much that you can gain from the C# side. You have to check the query plan for the SQL query and then add necessary indexes. The SymbolId field should be an indexed field.

LINQ to Dynamics CRM query optimisation

I'm working with a website that links into a Dynamics CRM Online. I'm new to both of these but find the best way to learn is to put yourself under pressure.
Anyway, I have the following LINQ query that I've built using LinqPad:
from m in py3_membershipSet
join c in ContactSet on m.py3_Member.Id equals c.ContactId
where m.statuscode.Value == 1
orderby m.py3_name
select m
However, this gives an out of memory exception. It runs ok if I use Take(100) but I expect there to be about 1200 results to retrieve in total. Whether the memory issue is a LinqPad related problem I don't know but either way, I am assuming the above query isn't the most efficient way to pull these results.
I could really do with some help on making this more efficient, if it is as much of a memory hog as it appears via LinqPad.

An OutOfMemory exception,
...is thrown when there is not enough memory to continue the execution of
a program.
So I don't think it is anything in particular with the Linq you have written - apart from that it returning more data than your client can cope with. I suspect this is an issue more to do with your client than CRM or Linq.
This might be something do with LinqPad (not used it myself), have you tried running that script from a console app (to rule out any LinqPad issues)?
1200 doesn't sound like and awful lot of data, I often retrieve 1000~ records without issue but I have happily retrieved far more (5000~).
Paging might avoid the problem; Page Large Result Sets with LINQ.
Related reading: Troubleshooting Exceptions: System.OutOfMemoryException

Because the query does not know what fields will be needed later, all columns are returned from the entity when only the entity is specified in the select clause. In order to specify only the fields you will use, you must return a new object in the select clause, specifying the fields you want to use.
So instead of this:
from m in py3_membershipSet
join c in ContactSet on m.py3_Member.Id equals c.ContactId
where m.statuscode.Value == 1
orderby m.py3_name
select m
Use this:
from m in py3_membershipSet
join c in ContactSet on m.py3_Member.Id equals c.ContactId
where m.statuscode.Value == 1
orderby m.py3_name
select new py3_membership()
{
py3_membershipid = m.py3_membershipid,
py3_name = m.py3_name
}
Check out this post more details.
To Linq or not to Linq

IQueryable<T>.GroupBy() On Reference Type

Using Linq to SQL I'm writing queries that are taking advantage of the IQueryable.GroupBy method.
Even though my query involves many tables and left joins lets say for illustration that we are only working with two tables. TableA has a one to many relationship to TableB.
var queryResults = from db.TableA
.Join(db.TableB, tA => tA.ID, tB => tB.TableA_ID, (ta, tb) => tb)
.GroupBy(tb => tb.TableA);
This will give me an
<IQueryable<IGrouping<TableA, TableB>>
On the surface this seems to work however I'm worried because I'm calling the GroupBy method and passing in a reference type for the keySelector argument.
Please help me understand why this is or isn't a safe thing to do.

Hard to tell what you're trying to do, but I think you might want to use comprehension syntax here to make it easier to write/understand. In particular, do your grouping after you've done whatever joins or other operations you want to do instead of mixing them together.
var query = ... (can be chained methods, whatever)
var grouped = from row in query
group row by row.SomeProperty;
Then you'll likely have an easier time (I would think) writing and reasoning about the query.

Try this:
var queryResults =
from ta in db.TableA
join tb in db.TableB on ta.ID equals tb.TableA_ID
group tb by ta;
And yes, this is safe to do on reference types. The query generated by linq-to-sql has nothing to do with reference types - it's just SQL.

Working with Cross Context Joins in LINQ-to-SQL

Initially I had written this query using LINQ-to-SQL
var result = from w in PatternDataContext.Windows
join cf in PatternDataContext.ControlFocus on w.WindowId equals cf.WindowId
join p in PatternDataContext.Patterns on cf.CFId equals p.CFId
join r in ResultDataContext.Results on p.PatternId equals r.PatternId
join fi in ResultDataContext.IclFileInfos on r.IclFileId equals fi.IclFileId
join sp in sessionProfileDataContext.ServerProfiles on fi.ServerProfileId equals sp.ProfileId
join u in infrastructure.Users on sp.UserId equals u.Id
where w.Process.Equals(processName)
select u.DistributedAppId;
And when I executed it, and saw result in the QuickWatch.., it showed this message:
the query contains references to items defined on a different data context
On googling, I found this topic at Stackoverflow itself, where I learned simulating cross context joins and as suggested there, I changed my query a bit to this:
var result = from w in PatternDataContext.Windows
join cf in PatternDataContext.ControlFocus on w.WindowId equals cf.WindowId
join p in PatternDataContext.Patterns on cf.CFId equals p.CFId
join r in SimulateJoinResults() on p.PatternId equals r.PatternId
join fi in SimulateJoinIclFileInfos() on r.IclFileId equals fi.IclFileId
join sp in SimulateJoinServerProfiles() on fi.ServerProfileId equals sp.ProfileId
join u in SimulateJoinUsers() on sp.UserId equals u.Id
where w.Process.Equals(processName)
select u.DistributedAppId;
This query is using these SimulateXyz methods:
private static IQueryable<Result> SimulateJoinResults()
{
return from r in SessionDataProvider.Instance.ResultDataContext.Results select r;
}
private static IQueryable<IclFileInfo> SimulateJoinIclFileInfos()
{
return from f in SessionDataProvider.Instance.ResultDataContext.IclFileInfos select f;
}
private static IQueryable<ServerProfile> SimulateJoinServerProfiles()
{
return from sp in sessionProfileDataContext.ServerProfiles select sp;
}
private static IQueryable<User> SimulateJoinUsers()
{
return from u in infrastructureDataContext.Users select u;
}
But even this approach didn't solve the problem. I'm still getting this message in QuickWatch...:
the query contains references to items defined on a different data context
Any solution for this problem? Along with the solution, I would also want to know why the problem still exists, and how exactly the new solution removes it, so that from next time I could solve such problems myself. I'm new to LINQ, by the way.

I've had to do this before, and there are two ways to do it.
The first is to move all the servers into a single context. You do this by pointing LINQ-to-SQL to a single server, then, in that server, create linked servers to all the other servers. Then you just create views for any tables you're interested from the other servers, and add those views to your context.
The second is to manually do the joins yourself, by pulling in data from one context, and using just the properties you need to join into another context. For example,
int[] patternIds = SessionDataProvider.Instance.ResultDataContext.Results.Select(o => o.patternId).ToArray();
var results = from p in PatternDataContext.Patterns
where patternIds.Contains(p.PatternId)
select p;
Though the first is easier to work with, it does have its share of problems. The problem is that you're relying on SQL Server to be performant with linked servers, something it is notoriously bad at. For example, consider this query:
var results = from p in DataContext.Patterns
join r in DataContext.LinkedServerResults on p.PatternId equals r.PatternId
where r.userId = 10;
When you enumerate this query, the following will occur (let's call the normal and linked servers MyServer and MyLinkedServer, respectively)
MyServer asks MyLinkedServer for the Results
MyLinkedServer sends the Results back to MyServer
MyServer takes those Results, joins them on the Patterns table, and returns only the ones with Results.userId = 10
So now the question is: When is the filtering done - on MyServer or MyLinkedServer? In my experience, for such a simple query, it will usually be done on MyLinkedServer. However, once the query gets more complicated, you'll suddenly find that MyServer is requesting the entire Results table from MyLinkedServer and doing the filtering after the join! This wastes bandwidth, and, if the Results tables is large enough, could turn a 50ms query into a 50 second query!
You could fix unperformant cross-server joins using stored procedures, but if you do a lot of complex cross-server joins, you may end up writing stored procedures for most of your queries, which is a lot of work and defeats part of the purpose of using L2SQL in the first place (not having to write a lot of SQL).
In comparison, the following code would always perform the filtering on the server containing the Results table:
int[] patternIds = (from r in SessionDataProvider.Instance.ResultDataContext.Results
where r.userId = 10
select r.PatternId).ToArray();
var results = from p in PatternDataContext.Patterns
where patternIds.Contains(p.PatternId)
select p;
Which is best for your situation is up to your best judgement.
Note that there is a third potential solution which I did not mention, as it is not really a programmer-solution: you could ask your server admins to set up a replication task to copy the necessary data from MyLinkedServer to MyServer once a day/week/month. This is only an option if:
Your program can work with slightly stale data from MyLinkedServer
You only need to read, never write, to MyLinkedServer
The tables you need from MyLinkedServers are not exorbitantly huge
You have the space/bandwidth available
Your database admins are not stingy/lazy

Your SimulateJoins can't work because they return IQueryable. Your current solution is exactly the same as your former one and that is the reason why you get the same exception. If you check the linked question again you will see that their helper methods return IEnumerable which is the only way to make cross context operations. As you probably already know it means that join will be performed in memory on the application server instead of the database server = it will pull all data from your partial queries and execute join as linq-to-objects.
Cross context join on database level is IMO not possible. You can have different connections, different connection strings with different servers, etc. Linq-to-sql does not handle this.

You could work around it by "escaping from" Linq to SQL on the second context, i.e., calling for instance .ToList() on ResultDataContext.Results and ResultDataContext.IclFileInfos so that your query ended up looking like:
var result = from w in PatternDataContext.Windows
join cf in PatternDataContext.ControlFocus on w.WindowId equals cf.WindowId
join p in PatternDataContext.Patterns on cf.CFId equals p.CFId
join r in ResultDataContext.Results.ToList()
on p.PatternId equals r.PatternId
join fi in ResultDataContext.IclFileInfos.ToList()
on r.IclFileId equals fi.IclFileId
join sp in sessionProfileDataContext.ServerProfiles on
fi.ServerProfileId equals sp.ProfileId
join u in infrastructure.Users on sp.UserId equals u.Id
where w.Process.Equals(processName)
select u.DistributedAppId;
Or AsEnumerable() as long as you "get out" of Linq to SQL and into Linq to Objects for the "offending" context.

Old question, but as I happened to have the same problem, my solution was to pass the manually crafted T-SQL cross-server query (with linked servers) directly to the provider through the ExecuteQuery method of the first context:
db.ExecuteQuery(Of cTechSupportCall)(strSql).ToList
This just saves you from having to create a view server side, and Linq to SQL still maps the results to the proper type. This is useful when there is that one query that is just impossible to formulate in Linq.

Why would Entity Framework not be able to use ToString() in a LINQ statement?

This works in LINQ-to-SQL:
var customersTest = from c in db.Customers
select new
{
Id = c.Id,
Addresses = from a in db.Addresses where c.Id.ToString() ==
a.ReferenzId select a
};
foreach (var item in customersTest)
{
Console.WriteLine(item.Id);
}
But a similar example in Entity Framework gets an error message that says basically that it can't "translate it to SQL", here is the original error message in German:
"'LINQ to Entities' erkennt die
Methode 'System.String ToString()'
nicht, und diese Methode kann nicht in
einen Speicherausdruck übersetzt
werden."
Translation:
"'LINQ to Entities' does not recognize
Method 'System.String ToString()',
this method can not be translated into
a memory expression.
Can anyone shed any light on how we could get this kind of statement to work in Entity Framework or explain why it gets this error?

Simply put: LINQ to Entities doesn't know about the conversion from your ID type to a string.
What is the type of c.ID? Is there any reason why it's one type for ID, but another for ReferenzId? If at all possible, make them the same type, at which point you won't have a problem any more. I don't know if there are other ways of performing conversions in LINQ to Entities - there may be - but aligning the types would be cleaner.
By the way, this really looks like it's a join:
var query = from c in db.Customers
join a in db.Addresses on c.Id equals a.ReferenzId into addresses
select new { Id = c.Id, Addresses = addresses };
EDIT: To respond to your comment - ToString appears in IntelliSense because the compiler has no real idea what your query is going to mean or how it will be translated. It's perfectly valid C#, and can generate a valid expression tree - it's just that EF doesn't know how to convert that expression tree into SQL.
You could try using Convert.ToString(c.Id) instead of just calling c.Id.ToString()...

Updated answer:
If you followed the link I gave you at the beginning of my answer, this missing feature has meanwhile received 75 votes and is now (finally!) implemented by Microsoft in EF 6.1. To all who participated: Thank you for voting! Your voice was heard.
For example:
var query = from e in context.Employees where e.EmployeeID.ToString() == "1" select e;
will now be translated to:
DECLARE #p0 NVarChar(1000) = '1'
SELECT [t0].[EmployeeID], [t0].[LastName], [t0].[FirstName], [t0].[Title],
[t0].[TitleOfCourtesy], [t0].[BirthDate], [t0].[HireDate], [t0].[Address],[t0].[City],
[t0].[Region], [t0].[PostalCode], [t0].[Country], [t0].[HomePhone], [t0].[Extension],
[t0].[Photo], [t0].[Notes], [t0].[ReportsTo], [t0].[PhotoPath]
FROM [Employees] AS [t0]
WHERE (CONVERT(NVarChar,[t0].[EmployeeID])) = #p0
i.e. e.EmployeeID.ToString() translates to (CONVERT(NVarChar,[t0].[EmployeeID])).
Original answer:
It makes no sense to me why Linq2EF does not translate .ToString() into a proper SQL statement, as Linq2SQL does, only the Microsoft dev team knows the reason why they did not implement it yet. :-(
But you can raise the priority to implement it if you vote for this feature by following this link.
Luckily there are also 2 workarounds available, both of them I used recently in EF queries:
I) What helped me to get around this limitation was to change the query into a list, like so:
var customersList = (from c in db.Customers
select c).ToList(); // converts to IEnumerable<T> ...
var customersTest = (from c in customersList
select new {Id=c.ID.ToString()}); // ... which allows to use .ToString()
The statement .ToList() converts to IEnumerable<T>, where .ToString() is available. Note that, Depending on the requirements, you can use .AsEnumerable() as well, which has the advantage that deferred execution is supported which is better if you have multiple linq queries depending on each other or if you're using different parameter values (many thanks to Divega for this hint!).
Afterwards you can use this query as you wish , e.g.:
var customersTest2 = from c in customersTest
select new
{
Id = c.Id,
Addresses = from a in db.Addresses where c.Id == a.ReferenzId select a
};
Of course if you need you can add more properties to the objects of customersTest as required. You can also optimize the query above, I have only used 3 steps for readability of this example.
II) For simple conversions, and if you have to reuse the generated query in further subqueries (and it needs to remain IQueryable), use SqlFunctions from System.Data.Objects.SqlClient, they will be translated into SQL queries correctly.
Example 1: Date conversion (you have to use dateparts as below shown)
var customersTest = from c in db.Customers
select new {
strDate=SqlFunctions.DateName("dd", c.EndDate)
+"."+SqlFunctions.DateName("mm", c.EndDate)
+"."+SqlFunctions.DateName("yyyy", c.EndDate)
}
Example 2: Numeric to string conversion
var customersTest = from c in db.Customers
select new {
strID=SqlFunctions.StringConvert((double)c.ID)
}
This should help you out of most situations where conversions into strings are required.

Entity Framework 6.1 RTM which was just released now support .ToString()

LINQ to Entities as far as I understand it (for v1) is very primative. In otherwords it doesn't know how to take the extension method "ToString()" and generate the SQL for it.
In LINQ to SQL, it executes the extension method "ToString()" before generating the SQL. The difference is that LINQ to Entities uses IQueryable instead of IEnumerable.
BUT, from what I remember casting should work (because casting is a data type and SQL knows about CAST()).
So
c.Id.ToString() should really be (string)c.Id
(also, make sure it is (string) and not (String)).
One of the downfalls I would say about using Lambda (in Entity Framework) to generate the SQL expression instead of pure LINQ.
Keep in mind too, that using CAST on the left side of the equals sign in SQL is a bit ill performing :-)