SQL server is able to translate EF .First() using its function TOP(1). But when using Entity Framework's .Last() function, it throws an exception. SQL server does not recognize such functions, for obvious reasons.
I used to work it around by sorting descending and taking the first corresponding line :
var v = db.Table.OrderByDescending(t => t.ID).FirstOrDefault(t => t.ClientNumber == ClientNumberDetected);
This does it with a single query, but sorting the whole table (million rows) before querying...
Do I have good reasons to think there will be speed issues if I abuse of this technique ?
I thought of something similar... but it requires two query :
int maxID_of_Client = db.Where(t => t.ClientNumber == ClientNumberDetected).Max(t => t.ID);
var v = db.First(t => t.ID == maxID_of_Client);
It's consisting of retrieving the max ID of the client, then use this ID to retrieve the last line of the client.
It doesn't seems faster to query two times...
There must be a way to optimize this and use a single query without sorting millions of datas.
Unless there is something I don't understand, I'm probably not the first to think about this problem and I want to solve it for good !
Thanks in advance.
The assumption driving this question is that result sets with no ordering clause come back from your DB in any predictable order at all.
In reality, result sets that come back from SQL have no implicit ordering and none should be assumed.
Therefore, the result of
db.Table.FirstOrDefault(t => t.ClientNumber == ClientNumberDetected)
is actually indeterminate.
Whether you're taking first or last, without ordering it's all meaningless anyway.
Now, what goes to SQL where you add an ordering clause to your LINQ? It will be something similar to...
SELECT TOP(1) something FROM somewhere WHERE foo=bar ORDER BY somevalue
or, in the the descending/last case
SELECT TOP(1) something FROM somewhere WHERE foo=bar ORDER BY somevalue DESC
From SQL's POV, there's no significant difference here and your DB will be optimized for this sort of query. The index can be scanned in either direction, and the cost of each query above is the same.
TL;DR :
db.Table.OrderByDescending(t => t.ID)
.FirstOrDefault(t => t.ClientNumber == ClientNumberDetected)
is just fine.
I am using two different ways to implement an advanced search where each approach has their advantages and disadvantages.
The user can perform an advanced search using 3 parameters (name, familyname, and mobile).
The first way I tried is to check which of those parameters are provided by the user; that requires me to do 8 if & else checks (2 ^ number of parameters) and in each conditions I write a separate query which accepts the corresponding parameters, for example if the user has entered name & family name the where clause of the query will look like this:
where(x=>x.name.contains(name) && x.familyname.contains(familyname))
or in another case if the user has entered only the mobile the where clause of the query will look like this :
where(x=>x.mobile==mobile)
the advantage of this way is that I hit the databse once but the disadvantage is that I have to write a lot more code.
The second way I tried is that I declared an IQueryable<> object and I feed data into It with no condition at first and then I check search parameters one by one and if any of them has value I filter the IQueryable with that value and at last I perform a .ToList(). This way has the advantage of much less code but hits the database twice which is a problem.
Here is a code sample of the second way:
List<ShoppingCardForGridView> list_ShoppingCardForGridView =
new List<ShoppingCardForGridView>();
IQueryable<ShoppingCardForGridView> outQuery =
from x in db.ShoppingCards
orderby x.TFDateBackFromBank descending
where x.TFIsPaymentCompleted == true
select new ShoppingCardForGridView
{
Id = x.Id,
TFCustomerFullName =
x.Customer.TFName + " " + x.Customer.TFFamilyName,
TFDateBackFromBank = x.TFDateBackFromBank.Value,
TFIsDelivered = x.TFIsDelivered,
TFItemsPriceToPay = x.TFItemsPriceToPay,
TFDateBackFromBankPersian = x.TFDateBackFromBankPersian
};
if (!string.IsNullOrEmpty(CustomerFullName))
{
outQuery = outQuery.Where(x =>
x.TFCustomerFullName.Contains(CustomerFullName));
}
if (!string.IsNullOrEmpty(IsDelivered))
{
bool deliveryStatus = Convert.ToBoolean(IsDelivered);
outQuery = outQuery.Where(x => x.TFIsDelivered == deliveryStatus);
}
list_ShoppingCardForGridView = outQuery.ToList();
I wonder if there is any better way or a best practice to perform an advanced search using entityframework?
You're not hitting the database multiple times with your second solution. Remember that an IQueryable object is a query itself, not the results of a query. Execution is deferred until the query is actually iterated. Conditionally appending multiple Where clauses based on various if checks is changing the query itself, not processing the results of the query (since there are no results at that point in time).
You can use a profiler on your database to verify that only one query is being executed, and that the one query contains all of the filtering in that one go.
This is a reasonably common pattern; I've used it on a number of instances for making custom search pages.
I have a little program that needs to do some calculation on a data range. The range maybe contain about half a millon of records. I just looked to my db and saw that a group by was executed.
I thought that the result was executed on the first line, and later I just worked with data in RAM. But now I think that the query builder combine the expression.
var Test = db.Test.Where(x => x > Date.Now.AddDays(-7));
var Test2 = (from p in Test
group p by p.CustomerId into g
select new { UniqueCount = g.Count() } );
In my real world app I got more subqueries that is based on the range selected by the first query. I think I just added a big overhead to let the DB make different selects.
Now I bascilly just call .ToList() after the first expression.
So my question is am I right about that the query builder combine different IQueryable when it builds the expression tree?
Yes, you are correct. LINQ expressions are lazily evaluated at the moment you evaluate them (via .ToList(), for example). At that point in time, Entity Framework will look at the total query and build an SQL statement to represent it.
In this particular case, it's probably wiser to not evaluate the first query, because the SQL database is optimized for performing set-based operations like grouping and counting. Rather than forcing the database to send all the Test objects across the wire, deserializing the results into in-memory objects, and then performing the grouping and counting locally, you will likely see better performance by having the SQL database just return the resulting Counts.
I'd like some expert advice on this. I've used compiled queries before, but for this particular case, i'm not sure whether it's appropriate.
It's a search form where the query changes and is dependent on what is being searched on.
static Func<DBContext, int, IQueryable<Foo>> Search = CompiledQuery.Compile(
(DBContext db, int ID) =>
db.Person
.Where(w => w.LocationID = ID)
.Select(s =>
new Foo
{
Name = s.PersonName,
Age = s.Age,
Location = s.LocationName,
Kin = s.Kin
}));
Now if someone fills in the search box, i want to extend the query by adding another Where statement to the query:
var query = Search(context, 123);
query = query.Where(w => w.Name.Contains(searchString));
So my question is, is it returning all the results where LocationID == 123, then checking the results for a searchString match? Or is it actually extending the compiled query?
If it's the former (which i suspect it is), should scrap the CompiledQuery and just create a method that extends the query then return it as a list?
Also, what are the best practices for CompiledQuery usage and is there a guideline of when they should be used?
Note: I'm using the above in an ASP.NET website with Linq to SQL. Not sure if that makes any difference.
Thanks
The problem is that the compiled query is set in stone; it knows what SQL it will run against the database. The lambda expression is lazy loaded however, and cannot modify the compile query as it is being run during run time. The bad news is that it will return all of the records from the database, but it will query those records in memory to further refine them.
If you want to compile the query then I would suggest writing two queries with different signatures.
As far as I know, it is good practice to compile your query once, that is the whole point of pre-compiled query(and that's why your pre-compiled query is static), it saves time to compile that query into SQL. If it extend that pre-compiled query, then it is compiling that query again, which you loose gains.
Query result on result (your query variable) is no longer LINQ to SQL.
Just include your additional condition in your compiled query.
DB.Person.Where(w => w.LocationID == ID
& (searchString=="" || w.Name.Contains(searchString)))
If i am right then you need some dynamic where clause in linq. So for that i would suggest go this way
IEnumerable list;
if(condition1)
{
list = Linq Statement;
}
if(condition2)
{
list = from f in list where con1=con && con2=con select f;
}
if(condition3)
{
list = from n in list con1=con && con2=con select f;
}
I hope you got my words.
I know a typical way is like this:
IQueryable query = from staff in dataContext.Staffs;
if(name1 != null)
{
query = from staff in query where (staff.name == name1);
}
However, from a program we took over from other developers, we saw code like this:
IQueryable query = from staff in dataContext.Staffs;
query = from staff in query where (name1 == null || staff.name == name1);
If this is a normal SQL statement, I would definitely say that the 2nd one is a bad practice. Because it adds a meaningless where clause to the query when name1 is null.
But I am new to LINQ, so I am not sure if LINQ is different?
you can write it like
IQueryable query = from staff in dataContext.Staffs;
query = from staff in query where (name1 != null && staff.name == name1);
This way second part of your condition will not be evaluated if your first condition evaluates to false
Update:
if you write
IQueryable query = from staff in dataContext.Staffs;
query = from staff in query where (name1 == null || staff.name == name1);
and name1 is null second part of your condition will not be evaluated since or condition only requires one condition to return true
plz see this link for further detail
Often this sort of thing feels smoother to write using the fluent syntax, rather than the query syntax.
e.g.
IQueryable query = dataContext.Staffs;
if(name1 != null)
{
query = query.Where(x => x.name == name1);
}
So if name1 is null, you just don't do any Where() call. If you have multiple different filters, all of which may or may not be required, and perhaps various different sort orders, I find this becomes a lot more manageable.
Edit for alex: OK, I was answering the question about adding a where clause only when a value is not null. In response to the other part of the question, I tried this out with Entity Framework 4 to see what SQL that LINQ produced. You do this by casting query to an ObjectQuery and calling .ToTraceString(). The results were that the WHERE clause came out as follows:
WHERE #p__linq__0 IS NULL OR [Extent1].[name] = #p__linq__1
So, yes, it's classic bad SQL, if you have an index on the name column, don't expect it to be used.
Edit #2: Tried this again using LINQ to SQL rather than Entity Framework, with rather different results. This time, trying the query with name1 being null results in no WHERE clause at all, as you'd hope; trying it with name1 being "a" resulted in a simple WHERE [t0].[name] = #p0 and #p0 sent as "a". Entity Framework does not seem to optimize thus. That's a bit worrying.
The best way to do this is to create yourself an extension method that will take in a conditional statement and a where expression. If the condition is true then it will use the where expression else it will not use it. This can dramatically clean up your code, eliminating the need for if statements.
public static class LinqExtensions
{
public static IQueryable<T> WhereIf<T>(this IQueryable<T> query, bool condition, Expression<Func<T, bool>> whereClause)
{
if (condition)
{
return query.Where(whereClause);
}
return query;
}
}
Now you can write your code like this:
IQueryable<Staffs> query = dataContext.Staffs.AsQueryable().WhereIf(name1 != null, x => x.Name == name1);
So I tried the .Where(..., x => ...) extension method listed here as an answer but it doesn't work against Entity Framework as Linq To Entities doesn't know how to translate that into TSQL.
So here's my solution getting my Func on:
Expression<Func<SomeEfPoco, bool>> columnBeingFilteredPredicate = x => true; // Default expression to just say yes
if (!string.IsNullOrWhiteSpace(someColumnBeingFilteredValue))
{
columnBeingFilteredPredicate = x => x.someColumnBeingFiltered == someColumnBeingFilteredValue;
}
_context.SomeEfPocos.Where(x => ..... &&
..... &&
..... &&)
.Where(columnBeingFilteredPredicate);
someColumnBeingFilteredValue in my case is a string parameter on the encapsulating method with a default value of NULL.
LINQ is diffrent in some other causes (not in this causes),
LINQ is the way to get data in the "Faster way" with a littel code and clear cod as possible, there a many benefits of LINQ:
Makes it easier to transform data into objects. I'm sure you've heard the term "Impedence Mismatch" being used quite often, meaning that LINQ reduces the amount of work you must do to translate between object-oriented code and data paradigms such as hierarchical, flat-file, messages, relational, and more. It doesn't eliminate the "Impedence Mismatch" because you must still reason about your data in its native form, but the bridge from here to there is (IMO) much shorter.
A common syntax for all data. Once you learn query syntax, you can use it with any LINQ provider. I think this is a much better development paradigm than the Tower of Babel that has grown over the years with data access technologies. Of course, each LINQ provider has unique nuances that are necessary, but the basic approach and query syntax is the same.
Strongly typed code. The C# (or VB.NET) query syntax is part of the language and you code with C# types, which are translated into something a provider understands. This means that you gain the productivity of having your compiler find errors earlier in the development lifecycle than elsewhere. Granted, many errors in stored proc syntax will generate errors when you save, but LINQ is more general than SQL Server. You have to think of all the other types of data sources that generate runtime errors because their queries are formed with strings or some other loosely typed mechanism.
Provider integration. Pulling together data sources is very easy. For example, you can use LINQ to Objects, LINQ to SQL, and LINQ to XML together for some very sophisticated scenarios. I think it's very elegant.
Reduction in work. Before LINQ, I spent a lot of time building DALs, but now my DataContext is the DAL. I've used OPFs too, but now I have LINQ that ships with multiple providers in the box and many other 3rd party providers, giving me the benefits from my previous points. I can set up a LINQ to SQL DataContext in a minute (as fast as my computer and IDE can keep up).
Performance in the general case doesn't become an issue. SQL Server optimizes queries quite well these days, just like stored procs. Of course, there are still cases where stored procs are necessary for performance reasons. For example, I've found it smarter to use a stored proc when I had multiple interactions between tables with additional logic inside of a transaction. The communications overhead of trying to do the same task in code, in addition to getting the DTC involved in a distributed transaction made the choice for a stored proc more compelling. However, for a query that executes in a single statement, LINQ is my preferred choice because even if there was a small performance gain from a stored proc, the benefits in previous points (IMO) carry more weight.
Built-in security. One reason I preferred stored procs before LINQ was that they forced the use of parameters, helping to reduce SQL injection attacks. LINQ to SQL already parameterizes input, which is just as secure.
LINQ is declarative. A lot of attention is paid to working with LINQ to XML or LINQ to SQL, but LINQ to Objects is incredibly powerful. A typical example of LINQ to Objects is reading items from a string[]. However, that's just a small example. If you think about all of the IEnumerable collections (you can also query IEnumerable) that you work with every day, the opportunities are plentiful. i.e. Searching an ASP.NET ListBox control for selected items, performing set operations (such as Union) on two collections, or iterating through a List and running a lambda in a ForEach of each item. Once you begin to think in LINQ, which is declarative in nature, you can find many of your tasks to be simpler and more intuitive than the imperative techniques you use today.
I could probably go on, but I'd better stop there. Hopefully, this will provide a more positive view of how you could be more productive with LINQ and perhaps see it as a useful technology from a broader perspective.
I've seen this pattern in standard SQL, and it seems useful if you have several parameters that may be NULL. For example:
SELECT * FROM People WHERE ( #FirstName IS NULL OR FirstName = #FirstName )
AND ( #LastName IS NULL OR LastName = #LastName )
If you see this in LINQ, it's possible they just blindly translated their old SQL-queries.
I like use the Expression
e.g.
Expression<Func<Persons, bool>> expresionFinal = c => c.Active == true;
if (DateBirth.HasValue)
{
Expression<Func<Persons, bool>> expresionDate = c => (EntityFunctions.TruncateTime(c.DateBirth) == DateBirth);
expresionFinal = PredicateBuilder.And(expresionFinal, expresionDate);
}
IQueryable query = dataContext.Persons;
query = query.Where(expresionFinal);
For EF Core I broke it up like this:
IQueryable<Partners> recs = contextApi.Partners;
if (status != -1)
{
recs = recs.Where(i => i.Status == status);
}
recs = recs.OrderBy(i => i.Status).ThenBy(i => i.CompanyName);
foreach (var rec in recs)
{
}
I had to be explicit with my typing instead of relying on var.
I like the idea with Extension
public static IQueryable<T> WhereIf<T>(this IQueryable<T> query, bool condition, Expression<Func<T, bool>> whereClause)
=> condition ? query.Where(whereClause) : query;
No, I am not strongly agree with you.
here you just gave a simple logic
if(name1 != null)
// do your stuff
but what will happen if you do something different with the name1 that have null value..!!
Ok, now consider this situation.
In this example you shows how to handle possible null values in source collections.
An object collection such as an IEnumerable<T> can contain elements whose value is null.
If a source collection is null or contains an element whose value is null,
and your query does not handle null values, a NullReferenceException will be thrown when you execute the query.
Probably this could be a issue...
I use the extension method below. It's less flexible than the WhereIf extension from the other answers, but it's shorter to use.
public static IQueryable<T1> FilterBy<T1, T2>(this IQueryable<T1> query, T2 expectedValue, Expression<Func<T1, T2>> propertyAccessor)
{
if (propertyAccessor == null) throw new ArgumentNullException(nameof(propertyAccessor));
if (expectedValue == null) return query;
var equalExpr = Expression.Equal(propertyAccessor.Body, Expression.Constant(expectedValue, typeof(T2)));
var lambda = Expression.Lambda<Func<T1, bool>>(equalExpr, propertyAccessor.Parameters);
return query.Where(lambda);
}
It can be used like:
var query = dataContext.Staffs.FilterBy(name, s => s.Name);