We Keep Coding

Querying with many (~100) search terms with Entity Framework

I need to do a query on my database that might be something like this where there could realistically be 100 or more search terms.
public IQueryable<Address> GetAddressesWithTown(string[] towns)
{
IQueryable<Address> addressQuery = DbContext.Addresses;
addressQuery.Where( x => towns.Any( y=> x.Town == y ) );
return addressQuery;
}
However when it contains more than about 15 terms it throws and exception on execution because the SQL generated is too long.
Can this kind of query be done through Entity Framework?
What other options are there available to complete a query like this?

Sorry, are we talking about THIS EXACT SQL?
In that case it is a very simple "open your eyes thing".
There is a way (contains) to map that string into an IN Clause, that results in ONE sql condition (town in ('','',''))
Let me see whether I get this right:
addressQuery.Where( x => towns.Any( y=> x.Town == y ) );
should be
addressQuery.Where ( x => towns.Contains (x.Town)
The resulting SQL will be a LOT smaller. 100 items is still taxing it - I would dare saying you may have a db or app design issue here and that requires a business side analysis, I have not me this requirement in 20 years I work with databases.

This looks like a scenario where you'd want to use the PredicateBuilder as this will help you create an Or based predicate and construct your dynamic lambda expression.
This is part of a library called LinqKit by Joseph Albahari who created LinqPad.
public IQueryable<Address> GetAddressesWithTown(string[] towns)
{
var predicate = PredicateBuilder.False<Address>();
foreach (string town in towns)
{
string temp = town;
predicate = predicate.Or (p => p.Town.Equals(temp));
}
return DbContext.Addresses.Where (predicate);
}

You've broadly got two options:
You can replace .Any with a .Contains alternative.
You can use plain SQL with table-valued-parameters.
Using .Contains is easier to implement and will help performance because it translated to an inline sql IN clause; so 100 towns shouldn't be a problem. However, it also means that the exact sql depends on the exact number of towns: you're forcing sql-server to recompile the query for each number of towns. These recompilations can be expensive when the query is complex; and they can evict other query plans from the cache as well.
Using table-valued-parameters is the more general solution, but it's more work to implement, particularly because it means you'll need to write the SQL query yourself and cannot rely on the entity framework. (Using ObjectContext.Translate you can still unpack the query results into strongly-typed objects, despite writing sql). Unfortunately, you cannot use the entity framework yet to pass a lot of data to sql server efficiently. The entity framework doesn't support table-valued-parameters, nor temporary tables (it's a commonly requested feature, however).
A bit of TVP sql would look like this select ... from ... join #townTableArg townArg on townArg.town = address.town or select ... from ... where address.town in (select town from #townTableArg).
You probably can work around the EF restriction, but it's not going to be fast and will probably be tricky. A workaround would be to insert your values into some intermediate table, then join with that - that's still 100 inserts, but those are separate statements. If a future version of EF supports batch CUD statements, this might actually work reasonably.
Almost equivalent to table-valued paramters would be to bulk-insert into a temporary table and join with that in your query. Mostly that just means you're table name will start with '#' rather than '#' :-). The temp table has a little more overhead, but you can put indexes on it and in some cases that means the subsequent query will be much faster (for really huge data-quantities).
Unfortunately, using either temporary tables or bulk insert from C# is a hassle. The simplest solution here is to make a DataTable; this can be passed to either. However, datatables are relatively slow; the over might be relevant once you start adding millions of rows. The fastest (general) solution is to implement a custom IDataReader, almost as fast is an IEnumerable<SqlDataRecord>.
By the way, to use a table-valued-parameter, the shape ("type") of the table parameter needs to be declared on the server; if you use a temporary table you'll need to create it too.
Some pointers to get you started:
http://lennilobel.wordpress.com/2009/07/29/sql-server-2008-table-valued-parameters-and-c-custom-iterators-a-match-made-in-heaven/
SqlBulkCopy from a List<>

Why does LINQ-to-Entities put this query in a sub-select?

I have the following LINQ query:
var queryGroups = (from p in db.cl_contact_event
select new Groups { inputFileName = p.input_file_name }).Distinct();
Which translates to the following when run:
SELECT
[Distinct1].[C1] AS [C1],
[Distinct1].[input_file_name] AS [input_file_name]
FROM ( SELECT DISTINCT
[Extent1].[input_file_name] AS [input_file_name],
1 AS [C1]
FROM [mel].[cl_contact_event] AS [Extent1]
) AS [Distinct1]
Now I'm pretty sure that the reason there is a sub-select is because I have the base LINQ query surrounded by () and then perform .Distinct() but I don't know enough about LINQ to be sure of this. If that's indeed the case is there a way to restructure/code my query so that a sub-select doesn't occur?
I know that it probably seems that I'm just nit-picking here but I'm just curious.

In this I suspect that the actual root cause of the subquery is the anonymous type constructor. Because you are not selecting a known entity, but rather an arbitrary object constructed from other entity values, the EF parser needs to make sure it can produce the exact set of fields -- whether from a single table, joined tables, calculated fields, other sub-queries, etc. The expression tree parser is very good at writing SQL statements out of LINQ queries whenever possible, but it's not omniscient. It processes the queries in a systematic way, that will always produce correct results (in the sense that you get what you asked for), though not always optimal results.
As far as rewriting the query to eliminate the sub-select, first off: I don't see an obvious way to do so that eliminates the anonymous type and produces correct results. More importantly, though, I wouldn't bother. Modern SQL servers like Sybase are very smart -- often smarter than the developer -- and very good at producing an optimal query plan out of a query. Besides that, EF loves sub-queries, because they are very good ways to write complex queries in an automated fashion. You often find them even when your LINQ query did not appear use them. Trying to eliminate them all from your queries will quickly become an exercise in futility.

I wouldn't worry about this particular situation at all. SQL Server (and most likely any enterprise database) will optimize away the outer Select statement anyway. I would theorize that the reason this SQL statement is generated is because this is the most generic and reusable statement. From my experience, this always happens on Distinct().

Entity Framework and large queries. What's practical?

I'm from old school where DB had all data access encapsulated into views, procedures, etc. Now I'm forcing myself into using LINQ for most of the obvious queries.
What I'm wondering though, is when to stop and what practical? Today I needed to run query like this:
SELECT D.DeviceKey, D.DeviceId, DR.DriverId, TR.TruckId, LP.Description
FROM dbo.MBLDevice D
LEFT OUTER JOIN dbo.DSPDriver DR ON D.DeviceKey = DR.DeviceKey
LEFT OUTER JOIN dbo.DSPTruck TR ON D.DeviceKey = TR.DeviceKey
LEFT OUTER JOIN
(
SELECT LastPositions.DeviceKey, P.Description, P.Latitude, P.Longitude, P.Speed, P.DeviceTime
FROM dbo.MBLPosition P
INNER JOIN
(
SELECT D.DeviceKey, MAX(P.PositionKey) LastPositionKey
FROM dbo.MBLPosition P
INNER JOIN dbo.MBLDevice D ON P.DeviceKey = D.DeviceKey
GROUP BY D.DeviceKey
) LastPositions ON P.PositionKey = LastPositions.LastPositionKey
) LP ON D.DeviceKey = LP.DeviceKey
WHERE D.IsActive = 1
Personally, I'm not able to write corresponing LINQ. So, I found tool online and got back 2 page long LINQ. It works properly-I can see it in profiler but it's not maintainable IMO. Another problem is that I'm doing projection and getting Anonymous object back. Or, I can manually create class and project into that custom class.
At this point I wonder if it is better to create View on SQL Server and add it to my model? It will break my "all SQL on cliens side" mantra but will be easier to read and maintain. No?
I wonder where you stop with T-SQL vs LINQ ?
EDIT
Model description.
I have DSPTrucks, DSPDrivers and MBLDevices.
Device can be attached to Truck or to Driver or to both.
I also have MBLPositions which is basically pings from device (timestamp and GPS position)
What this query does - in one shot it returns all device-truck-driver information so I know what this device attached to and it also get's me last GPS position for those devices. Response may look like so:
There is some redundant stuff but it's OK. I need to get it in one query.

In general, I would also default to LINQ for most simple queries.
However, when you get at a point where the corresponding LINQ query becomes harder to write and maintain, then what's the point really? So I would simply leave that query in place. It works, after all. To make it easier to use it's pretty straight-forward to map a view or cough stored procedure in your EF model. Nothing wrong with that, really (IMO).

You can firstly store Linq queries in variables which may help to make it not only more readable, but also reusable.
An example maybe like the following:
var redCars = from c in cars
where c.Colour == "red"
select c;
var redSportsCars = from c in redCars
where c.Type == "Sports"
select c;
Queries are lazily executed and not composed until you compile them or iterate over them so you'll notice in profiler that this does produce an effecient query
You will also benifit from defining relationships in the model and using navigation properties, rather than using the linq join syntax. This (again) will make these relationships reusable between queries, and more readable (because you don't specify the relationships in the query like the SQL above)
Generally speaking your LINQ query will be shorter than the equivalent SQL, but I'd suggest trying to work it out by hand rather than using a conversion tool.
With the exception of CTEs (which I'm fairly sure you can't do in LINQ) I would write all queries in LINQ these days

I find when using LINQ its best to ignore whatever sql it generates as long as its retrieving the right thing and is performant, only when one of those doesn't work do I actually look at what its generating.
In terms of the sql it generates being maintainable, you shouldn't really worry about the SQL being maintainable but more the LINQ query that is generating the SQL.
In the end if the sql is not quite right I believe there are various things you can do to make LINQ generate SQL more along the lines you want..to some extent.
AFAIK there isn't any inherent problem with getting anonymous objects back, however if you are doing it it multiple places you may want to create a class to keep things neater.

When/why would you use Linq in an application? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
i know how linq works but i am not getting idea where exactly i should use linq?

The question is rather like asking "I know how plus and minus work, but when do I use addition? When do I use subtraction?"
You use addition when you want to know the sum of two things and subtraction when you want to know their difference.
Essentially all LINQ does is add operators to the C# and VB languages for sorting, filtering, projecting, joining and grouping. When do you use LINQ? When you want to sort, filter, project, join or group something.

I say use LINQ.. Absolutely anywhere that it will help accomplish your goal in a concise and maintainable way.
There is no broad use or don't use restrictions, you really have to make a judgement for the piece of code at hand to whether or not it will benefit from LINQ, in general some of the old overly verbose pieces of code (nested looping etc) can be more clearly expressed in LINQ and might be beneficial.
And it should be noted that LINQ is NOT synonymous with LINQ to SQL, LINQ stands for Language Integrated Query and is extraordinarily useful outside a database context.

LINQ is used generally when you have to work with IEnumerable (or IQueryable in case of operations against a database). It makes your code cleaner and more functional.
You could use it for example to replace a for loop:
from i in Enumerable.Range(0, n)
where i % 2 == 0
select i;
instead of:
for (int i = 0; i < n; i++)
{
if (i % 2 == 0)
{
something.Add(i);
}
}

You should use LINQ where you find it appropriate -because it can be both more expressive/concise than conventional common procedural programming idioms.

You can utilize it for a number of scenarios, e.g:
Retrieving data from Entity Framework or using Linq to SQL
Perform operations on collections (sorting, selecting, etc)
Traversing XML documents
These operations all require significantly less code using Linq.

LINQ is a standard. Custom layers you make are not, not unless your team pre-built and agreed upon a custom layer. If your team won't make custom layers, then using LINQ might alleviate the problem of each people having their own way of making requests to the database.

Anytime you want to filter data within your application without having to run a query. Here's a good place to look at examples of how you may want to use it.
http://msdn.microsoft.com/en-us/vcsharp/aa336746.aspx

For me it makes my code more concise and easier to maintain. use it anywhere you have to work with accessing or manipulting Collection of data like Databases, XML, Lists, etc.
But just keep in mind some of it quirks.

At places where it makes code more readable and maintainable.
In places where you work with collections of data and you want to transform, filter, connect or sort it.
You do not have to use LINQ, it is not the cure for all problems. It may actually perform worse than the procedual programmed way in some cases.
Start using it and with the time and experience you get the feeling where it makes sense and is most useful.

thanks for answer.
the answer might take pages why linq? but in short after reading ceratin books,
if i say The main goal of LINQ and LINQ to SQL is to
get rid of, or at least reduce An impedance mismatch between Relational database structure and object oriented . it’s easier to adapt the C# language than to change SQL or XML.
With LINQ, the aim is toward deeply integrating the capabilities of data query and
manipulation languages into programming languages.
LINQ removes many of the barriers among objects, databases, and XML. It
enables us to work with each of these paradigms using the same language-integrated
facilities.
i got answer from book *Linq In Action. *
in this first chapter clears why linq.
this is really good book.

Let's take an example. You want to write a simple query that retrieves customers as follows:
SELECT UPPER(Name)
FROM Customer
WHERE Name LIKE 'A%'
ORDER BY Name
That doesn't look too bad, right? But now suppose these results are feeding a web page, and we want to retrieve just rows 21-30. Suddenly, you need a subquery:
SELECT UPPER(Name) FROM
(
SELECT *, RN = row_number()
OVER (ORDER BY Name)
FROM Customer
WHERE Name LIKE 'A%'
) A
WHERE RN BETWEEN 21 AND 30
ORDER BY Name
Here's same query in LINQ. The gain in simplicity is clear:
var query =
from c in db.Customers
where c.Name.StartsWith ("A")
orderby c.Name
select c.Name.ToUpper();
var thirdPage = query.Skip(20).Take(10);
Only when we enumerate thirdPage will the query actually execute. In the case of LINQ to SQL or Entity Framework, the translation engine will convert the query (that we composed in two steps) into a single SQL statement optimized for the database server to which it's connected.

Linq2Sql - Storing Complex Linq Queries for future dynamic execuction - raw text - possible?

I am having a lot of fun with Linq2Sql. Expression Trees have been great, and just the standard Linq2Sql syntax has been a lot of fun.
I am now down to part of my application where I have to somehow store queries in a database, that are custom for different customers that use the same database and same tables (well, view, but you know what I mean). Basically, I cant hard-code anything, and I have to leave the query language clear text so someone can write a new where-clause type query.
So, if that description was harsh, let me clarify:
In a previous version of our application, we used to do direct SQL calls to the db using raw SQL. Yea. it was fun, dirty, and it worked. We would have a database table fulled of different criteria like
(EventType = 6 and Total > 0)
or a subquery style
(EventType = 7
AND Exists (
select *
from events as e1
where events.EventType = e1.EventType
and e1.objectNumber = 89)
)
(sql injection anyone?)
In Linq2Sql, this is a little more challenging. I can make all these queries no problem in the CLR, but being able to pass dynamic where criterias to Linq is a little more challenging, especially if I want to perform a sub query (like the above example).
Some ideas I had:
Get the raw expression, and store it --- but I have no idea how to take the raw text expression and reverse it back to executable to object expression.
Write a SQl like language, and have it parse the code and generate Linq Expression -- wow, that could be a lot of fun
I am quite sure there is no SomeIqueryable.Where("EventType = 6 and Total > 54"). I was reading that it was available in beta1, but I don't see how you can do that now.
var exp2 = context.POSDataEventView.Where("EmployeeNumber == #0", 8310);
This would be the easiest way for me to deploy.. I think.
Store serialized Expressions -- wow.. that would be confusing to a user trying to write a query --- hell, I'm not sure I could even type it all out.
So, I am looking for some ideas on how I can store a query in some kind of clear text, and then execute it against my Linq2Sql objects in some fashion without calling the ExecuteSQL. I want to use the LinqObjects.
P.S. I am using pLinqo for this application if that helps. Its still linq2sql though.
Thanks in advance!

Perhaps the Dynamic LINQ Library (in the MSDN samples) would help?
In particular, usage like:
This should work with any IQueryable<T> source - including LINQ-to-Objects simply by calling .AsQueryable() on the sequence (typically IEnumerable<T>).