We Keep Coding

More complex queries in LINQ than SQL

I am new to LINQ queries. I have read/researched about all advantages of LINQ queries over SQL but i have one basic question why do we need to use these queries as i feel their syntax is more complicated than traditional sql queries?
For example look at below example for simple Left Outer Join
var q=(from pd in dataContext.tblProducts
join od in dataContext.tblOrders on pd.ProductID equals od.ProductID into t
from rt in t.DefaultIfEmpty()
orderby pd.ProductID
select new
{
//To handle null values do type casting as int?(NULL int)
//since OrderID is defined NOT NULL in tblOrders
OrderID=(int?)rt.OrderID,
pd.ProductID,
pd.Name,
pd.UnitPrice,
//no need to check for null since it is defined NULL in database
rt.Quantity,
rt.Price,
})
.ToList();

So, the point of LINQ (Language Integrated Queries) is to provide easy ways of working with enumerable collections in executing memory. Contrast to SQL, which is a language for determining what the user gets from a set of data in a database.
Because of the SQL-like syntax, it's easy to confuse LINQ code with SQL, and think that they're 'alike' - they're really not. SQL gets a subset of data from a superset; LINQ is 'syntactic sugar' that hides common operations involving foreach loops.
For instance, this is a common programming pattern:
foreach(Thing thing in things)
{
if(thing.SomeProperty() == "Some Value")
return true;
}
...this is done rather easily in LINQ:
return things.Any(t => t.SomeProperty() == "Some Value");
The two code are functionally the same, and I'm pretty sure even compile to roughly the same IL code. The difference is how it looks to you.
You don't have to use LINQ; you can choose to use a standard foreach, and there are times, such as complex loops, where it is useful to do so. Ultimately it is a question of readability - my counter-question to you is, is the LINQ version of your foreach loop more, or less, readable than the original foreach loop?
If the answer is 'less', then I suggest converting it back to a foreach.

I'm by no means an sql or a linq expert, I use them both.
There is a trend to either make linq into something bad or a silver bullet depending on what side are you.
You need to seriously consider your project requirements in order to choose. The choice is not mutually exclusive. Take what is good from them both .
Advantages
Quick turn around for development
Queries can be dynamically
Tables are automatically created into class
Columns are automatically created into properties
Relationship are automatically appeaded to classes
Lambda expressions are awesome
Data is easy to setup and use
Disadvantages
No clear outline for Tiers
No good way of view permissions
Small data sets will take longer to build the query than execute
There is an overhead for creating queries
When queries are moved from sql to application side, joins are very slow
DBML concurrency issues
Hard to understand advance queries using Expressions
I found that programmers used to Sql will have a hard time figuring out the tricks with LINQ. But programmers with Sql knowledge, but haven't done a ton of work with it, will pick up linq quicker.

The main issue when people start using LINQ is that they keep thinking in the SQL way, they design the SQL query first and then translate it to LINQ. You need to learn how to think in the LINQ way and your LINQ query will become neater and simpler. For instance, in your LINQ you don't need joins. You should use Associations/Navigation Properties instead. Check this post for more details.

Entity Framework and large queries. What's practical?

I'm from old school where DB had all data access encapsulated into views, procedures, etc. Now I'm forcing myself into using LINQ for most of the obvious queries.
What I'm wondering though, is when to stop and what practical? Today I needed to run query like this:
SELECT D.DeviceKey, D.DeviceId, DR.DriverId, TR.TruckId, LP.Description
FROM dbo.MBLDevice D
LEFT OUTER JOIN dbo.DSPDriver DR ON D.DeviceKey = DR.DeviceKey
LEFT OUTER JOIN dbo.DSPTruck TR ON D.DeviceKey = TR.DeviceKey
LEFT OUTER JOIN
(
SELECT LastPositions.DeviceKey, P.Description, P.Latitude, P.Longitude, P.Speed, P.DeviceTime
FROM dbo.MBLPosition P
INNER JOIN
(
SELECT D.DeviceKey, MAX(P.PositionKey) LastPositionKey
FROM dbo.MBLPosition P
INNER JOIN dbo.MBLDevice D ON P.DeviceKey = D.DeviceKey
GROUP BY D.DeviceKey
) LastPositions ON P.PositionKey = LastPositions.LastPositionKey
) LP ON D.DeviceKey = LP.DeviceKey
WHERE D.IsActive = 1
Personally, I'm not able to write corresponing LINQ. So, I found tool online and got back 2 page long LINQ. It works properly-I can see it in profiler but it's not maintainable IMO. Another problem is that I'm doing projection and getting Anonymous object back. Or, I can manually create class and project into that custom class.
At this point I wonder if it is better to create View on SQL Server and add it to my model? It will break my "all SQL on cliens side" mantra but will be easier to read and maintain. No?
I wonder where you stop with T-SQL vs LINQ ?
EDIT
Model description.
I have DSPTrucks, DSPDrivers and MBLDevices.
Device can be attached to Truck or to Driver or to both.
I also have MBLPositions which is basically pings from device (timestamp and GPS position)
What this query does - in one shot it returns all device-truck-driver information so I know what this device attached to and it also get's me last GPS position for those devices. Response may look like so:
There is some redundant stuff but it's OK. I need to get it in one query.

In general, I would also default to LINQ for most simple queries.
However, when you get at a point where the corresponding LINQ query becomes harder to write and maintain, then what's the point really? So I would simply leave that query in place. It works, after all. To make it easier to use it's pretty straight-forward to map a view or cough stored procedure in your EF model. Nothing wrong with that, really (IMO).

You can firstly store Linq queries in variables which may help to make it not only more readable, but also reusable.
An example maybe like the following:
var redCars = from c in cars
where c.Colour == "red"
select c;
var redSportsCars = from c in redCars
where c.Type == "Sports"
select c;
Queries are lazily executed and not composed until you compile them or iterate over them so you'll notice in profiler that this does produce an effecient query
You will also benifit from defining relationships in the model and using navigation properties, rather than using the linq join syntax. This (again) will make these relationships reusable between queries, and more readable (because you don't specify the relationships in the query like the SQL above)
Generally speaking your LINQ query will be shorter than the equivalent SQL, but I'd suggest trying to work it out by hand rather than using a conversion tool.
With the exception of CTEs (which I'm fairly sure you can't do in LINQ) I would write all queries in LINQ these days

I find when using LINQ its best to ignore whatever sql it generates as long as its retrieving the right thing and is performant, only when one of those doesn't work do I actually look at what its generating.
In terms of the sql it generates being maintainable, you shouldn't really worry about the SQL being maintainable but more the LINQ query that is generating the SQL.
In the end if the sql is not quite right I believe there are various things you can do to make LINQ generate SQL more along the lines you want..to some extent.
AFAIK there isn't any inherent problem with getting anonymous objects back, however if you are doing it it multiple places you may want to create a class to keep things neater.

IQueryable<T>.GroupBy() On Reference Type

Using Linq to SQL I'm writing queries that are taking advantage of the IQueryable.GroupBy method.
Even though my query involves many tables and left joins lets say for illustration that we are only working with two tables. TableA has a one to many relationship to TableB.
var queryResults = from db.TableA
.Join(db.TableB, tA => tA.ID, tB => tB.TableA_ID, (ta, tb) => tb)
.GroupBy(tb => tb.TableA);
This will give me an
<IQueryable<IGrouping<TableA, TableB>>
On the surface this seems to work however I'm worried because I'm calling the GroupBy method and passing in a reference type for the keySelector argument.
Please help me understand why this is or isn't a safe thing to do.

Hard to tell what you're trying to do, but I think you might want to use comprehension syntax here to make it easier to write/understand. In particular, do your grouping after you've done whatever joins or other operations you want to do instead of mixing them together.
var query = ... (can be chained methods, whatever)
var grouped = from row in query
group row by row.SomeProperty;
Then you'll likely have an easier time (I would think) writing and reasoning about the query.

Try this:
var queryResults =
from ta in db.TableA
join tb in db.TableB on ta.ID equals tb.TableA_ID
group tb by ta;
And yes, this is safe to do on reference types. The query generated by linq-to-sql has nothing to do with reference types - it's just SQL.

Sort Linq list with one column

I guess it should be really simple, but i cannot find how to do it.
I have a linq query, that selects one column, of type int, and i need it sorted.
var values = (from p in context.Products
where p.LockedSince == null
select Convert.ToInt32(p.SearchColumn3)).Distinct();
values = values.OrderBy(x => x);
SearchColumn3 is op type string, but i only contains integers. So i thought, converting to Int32 and ordering would definitely give me a nice 1,2,3 sorted list of values. But instead, the list stays ordered like it were strings.
199 20 201
Update:
I've done some tests with C# code and LinqPad.
LinqPad generates the following SQL:
SELECT [t2].[value]
FROM (
SELECT DISTINCT [t1].[value]
FROM (
SELECT CONVERT(Int,[t0].[SearchColumn3]) AS [value], [t0].[LockedSince], [t0].[SearchColumn3]
FROM [Product] AS [t0]
) AS [t1]
WHERE ([t1].[LockedSince] IS NULL)
) AS [t2]
ORDER BY [t2].[value]
And my SQL profiler says that my C# code generates this piece of SQL:
SELECT DISTINCT a.[SearchColumn3] AS COL1
FROM [Product] a
WHERE a.[LockedSince] IS NULL
ORDER BY a.[SearchColumn3]
So it look like C# Linq code just omits the Convert.ToInt32.
Can anyone say something useful about this?

[Disclaimer - I work at Telerik]
You can solve this problem with Telerik OpenAccess ORM too. Here is what i would suggest in this case.
var values = (from p in context.Products
where p.LockedSince == null
orderby "cast({0} as integer)".SQL<int>(p.SearchColumn3)
select "cast({0} as integer)".SQL<int>(p.SearchColumn3)).ToList().Distinct();
OpenAccess provides the SQL extension method, which gives you the ability to add some specific sql code to the generated sql statement.
We have started working on improving this behavior.
Thank you for pointing this out.
Regards
Ralph

Same answer as one my other questions, it turns out that the Linq provider i'm using, the one that comes with Telerik OpenAccess ORM does things different than the standard Linq to SQL provider! See the SQL i've posted in my opening post! I totally wasn't expecting something like this, but i seem that the Telerik OpenAccess thing still needs a lot of improvement. So be careful before you start using it. It looks nice, but it has some serious shortcomings.

I can't replicate this problem. But just make sure you're enumerating the collection when you inspect it. How are you checking the result?
values = values.OrderBy(x => x);
foreach (var v in values)
{
Console.WriteLine(v.ToString());
}
Remember, this won't change the order of the records in the database or anywhere else - only the order that you can retrieve them from the values enumeration.

Because your values variable is a result of a Linq expression, so that it doest not really have values until you calling a method such as ToList, ToArray, etc.
Get back to your example, the variable x in OrderBy method, will be treated as p.SearchColumn3 and therefore, it's a string.
To avoid that, you need to let p.SearchColumn3 become integer before OrderBy method.
You should add a let statement in to your code as below:
var values = (from p in context.Products
where p.LockedSince == null
let val = Convert.ToInt32(p.SearchColumn3)
select val).Distinct();
values = values.OrderBy(x => x);
In addition, you can combine order by statement with the first, it will be fine.

Why would Entity Framework not be able to use ToString() in a LINQ statement?

This works in LINQ-to-SQL:
var customersTest = from c in db.Customers
select new
{
Id = c.Id,
Addresses = from a in db.Addresses where c.Id.ToString() ==
a.ReferenzId select a
};
foreach (var item in customersTest)
{
Console.WriteLine(item.Id);
}
But a similar example in Entity Framework gets an error message that says basically that it can't "translate it to SQL", here is the original error message in German:
"'LINQ to Entities' erkennt die
Methode 'System.String ToString()'
nicht, und diese Methode kann nicht in
einen Speicherausdruck übersetzt
werden."
Translation:
"'LINQ to Entities' does not recognize
Method 'System.String ToString()',
this method can not be translated into
a memory expression.
Can anyone shed any light on how we could get this kind of statement to work in Entity Framework or explain why it gets this error?

Simply put: LINQ to Entities doesn't know about the conversion from your ID type to a string.
What is the type of c.ID? Is there any reason why it's one type for ID, but another for ReferenzId? If at all possible, make them the same type, at which point you won't have a problem any more. I don't know if there are other ways of performing conversions in LINQ to Entities - there may be - but aligning the types would be cleaner.
By the way, this really looks like it's a join:
var query = from c in db.Customers
join a in db.Addresses on c.Id equals a.ReferenzId into addresses
select new { Id = c.Id, Addresses = addresses };
EDIT: To respond to your comment - ToString appears in IntelliSense because the compiler has no real idea what your query is going to mean or how it will be translated. It's perfectly valid C#, and can generate a valid expression tree - it's just that EF doesn't know how to convert that expression tree into SQL.
You could try using Convert.ToString(c.Id) instead of just calling c.Id.ToString()...

Updated answer:
If you followed the link I gave you at the beginning of my answer, this missing feature has meanwhile received 75 votes and is now (finally!) implemented by Microsoft in EF 6.1. To all who participated: Thank you for voting! Your voice was heard.
For example:
var query = from e in context.Employees where e.EmployeeID.ToString() == "1" select e;
will now be translated to:
DECLARE #p0 NVarChar(1000) = '1'
SELECT [t0].[EmployeeID], [t0].[LastName], [t0].[FirstName], [t0].[Title],
[t0].[TitleOfCourtesy], [t0].[BirthDate], [t0].[HireDate], [t0].[Address],[t0].[City],
[t0].[Region], [t0].[PostalCode], [t0].[Country], [t0].[HomePhone], [t0].[Extension],
[t0].[Photo], [t0].[Notes], [t0].[ReportsTo], [t0].[PhotoPath]
FROM [Employees] AS [t0]
WHERE (CONVERT(NVarChar,[t0].[EmployeeID])) = #p0
i.e. e.EmployeeID.ToString() translates to (CONVERT(NVarChar,[t0].[EmployeeID])).
Original answer:
It makes no sense to me why Linq2EF does not translate .ToString() into a proper SQL statement, as Linq2SQL does, only the Microsoft dev team knows the reason why they did not implement it yet. :-(
But you can raise the priority to implement it if you vote for this feature by following this link.
Luckily there are also 2 workarounds available, both of them I used recently in EF queries:
I) What helped me to get around this limitation was to change the query into a list, like so:
var customersList = (from c in db.Customers
select c).ToList(); // converts to IEnumerable<T> ...
var customersTest = (from c in customersList
select new {Id=c.ID.ToString()}); // ... which allows to use .ToString()
The statement .ToList() converts to IEnumerable<T>, where .ToString() is available. Note that, Depending on the requirements, you can use .AsEnumerable() as well, which has the advantage that deferred execution is supported which is better if you have multiple linq queries depending on each other or if you're using different parameter values (many thanks to Divega for this hint!).
Afterwards you can use this query as you wish , e.g.:
var customersTest2 = from c in customersTest
select new
{
Id = c.Id,
Addresses = from a in db.Addresses where c.Id == a.ReferenzId select a
};
Of course if you need you can add more properties to the objects of customersTest as required. You can also optimize the query above, I have only used 3 steps for readability of this example.
II) For simple conversions, and if you have to reuse the generated query in further subqueries (and it needs to remain IQueryable), use SqlFunctions from System.Data.Objects.SqlClient, they will be translated into SQL queries correctly.
Example 1: Date conversion (you have to use dateparts as below shown)
var customersTest = from c in db.Customers
select new {
strDate=SqlFunctions.DateName("dd", c.EndDate)
+"."+SqlFunctions.DateName("mm", c.EndDate)
+"."+SqlFunctions.DateName("yyyy", c.EndDate)
}
Example 2: Numeric to string conversion
var customersTest = from c in db.Customers
select new {
strID=SqlFunctions.StringConvert((double)c.ID)
}
This should help you out of most situations where conversions into strings are required.

Entity Framework 6.1 RTM which was just released now support .ToString()

LINQ to Entities as far as I understand it (for v1) is very primative. In otherwords it doesn't know how to take the extension method "ToString()" and generate the SQL for it.
In LINQ to SQL, it executes the extension method "ToString()" before generating the SQL. The difference is that LINQ to Entities uses IQueryable instead of IEnumerable.
BUT, from what I remember casting should work (because casting is a data type and SQL knows about CAST()).
So
c.Id.ToString() should really be (string)c.Id
(also, make sure it is (string) and not (String)).
One of the downfalls I would say about using Lambda (in Entity Framework) to generate the SQL expression instead of pure LINQ.
Keep in mind too, that using CAST on the left side of the equals sign in SQL is a bit ill performing :-)