We Keep Coding

Trim function within a Where Clause

Say you have a Collection of items
IList<string> items = List<string>();
And you want to search each element against some search term: term. If you performed a Trim() within the Where clause as follows, would the trim operation occur for each element in the sequence, or would it be compiled once and then used to check against the elements?
items.Where(o => o.Contains(term.Trim())).ToList();
This question was based on a LINQ to SQL statement but I have simplified it. (Unsure if compiling to SQL would make any difference).

If it's in memory, then yes, it is called every time. A simple test:
public static class BLah
{
public static string DoTrim(this string item)
{
Console.WriteLine("called");
return item.Trim();
}
}
IList<string> items = new List<string> { "a", "b", "c" };
items.Where(o => o.Contains(term.DoTrim()));
Prints out 'called' 3 times.
However, when executing on the database, it's completely up to the ORM to decide how to generate the SQL, which may or may not call TRIM() as part of the query. The only way to know is to test it and see what SQL it generates.
For Linq2Sql, it does not run the trim multiple times. For example:
string term = "a b ";
Warehouses.Where(w => w.AdminComment.Contains(term.DoTrim()));
Produces
-- Region Parameters
DECLARE #p0 NVarChar(1000) = '%a b%'
-- EndRegion
SELECT [t0].[Id], [t0].[Name], [t0].[AdminComment], [t0].[AddressId]
FROM [Warehouse] AS [t0]
WHERE [t0].[AdminComment] LIKE #p0

It's worth noting here a difference between object-based linq that uses Func<…> and expression-based linq that uses Expression<Func<…>>.
In object-based linq, o => o.Contains(term.Trim()) is used as a Func<string, bool>. We can reason about it as such.
Now, considering o => o.Contains(term.Trim()) as such. It has a captured variable term on which the method Trim() is called. And so that's just what happens.
There are three reasons why Trim() must be called every time.
Considering just this delegate, we do not know that Trim() is pure and hence would always return the same value for the same object.
Considering just this delegate we do not know that term is immutable, and so Trim() would return the same value for the same object even if we did know it was pure.
Considering just this delegate, We don't know that the captured variable isn't changed between calls.
In all a delegate can be reasoned about pretty much like a method if we consider the captured variables to be mutable fields. If you looked at:
class SomeClass
{
public string term;
public bool Predicate(string o)
{
return o.Contains(term.Trim());
}
}
You wouldn't expect the call to Trim() to be cached for reasons analogous to the three reasons above.
Now, with expression-based linq, such as Linq2SQL or EF it's a bit different.
In this case o => o.Contains(term.Trim()) is used as a Expression<Func<string, bool>>, and there's a lot more variety in what a provider might do with that.
The provider might have special knowledge to know that term.Trim() could only ever have the one value, and re-write the expression to o => o.Contains("/* result of single call to term.Trim() goes here */")
The provider might have no idea what to do with Trim() and throw an exception.
The provider might not do anything special in how it turns term.Trim() into SQL, but the database processing that SQL in turn might itself realise it needs only to be calculated once, and so do so.
None of the above might happen, with it being essentially the same as the object-based case.
Expressions handling is a lot more flexible in terms of what may or may not happen to optimise the handling.

That is not an L2S IQueryable call. If it were a proper L2S call, then it would be converted to:
field LIKE #p0
where #p0 set to %term%
For example, with Northwind sample database:
var term = "USA";
var items = db.Customers
.Where(c => c.Address.Contains( term.Trim() ))
.ToList();
would produce (thanks to LinqPad):
-- Region Parameters
DECLARE #p0 NVarChar(1000) SET #p0 = '%USA%'
-- EndRegion
SELECT [t0].[CustomerID], [t0].[CompanyName], [t0].[ContactName], [t0].[ContactTitle], [t0].[Address], [t0].[City], [t0].[Region], [t0].[PostalCode], [t0].[Country], [t0].[Phone], [t0].[Fax]
FROM [Customers] AS [t0]
WHERE [t0].[Address] LIKE #p0
Probably you would likely want to write that as this one instead:
var term = "USA".Trim();
var items = Customers
.Where(c => c.Address.Contains( term ))
.ToList();
Generated SQL would be the same.

Why should the result of term.Trim be cached in any way? There is no runtime-magic included that would imply that the result of this operation never changes and therefore the operation is always executed for every single element. To improve this you may cache the result yourself:
var test = term.Trim();
items.Where(o => o.Contains(test).ToList();

What is the best way to write queries in Entity Framework

What is best way to write query performance wise. e.g. i can write query to get all employes on specific last name as follow
ObjectQuery<Employee> queryEmp =
context.CreateQuery<Employee>(
"Select value e from Employees AS e Where e.LastName = #lastName",
new ObjectParameter("lastName", typeof(String)) { Value = "Farooqi" });
ObjectResult<Employee> results = query.Execute(MergeOption.AppendOnly);
OR
var v = from e in context.Employees where e.LastName == "Farooqi" select e;
OR
context.Employees.Where(e => e.LastName == "Farooqi");
OR
context.Employees.Where("LastName = #lastName", new ObjectParameter("lastName", typeof(String)) { Value = "Farooqi" });
My question is this that which method for querying is best.
What i am confused with is that i think (but not sure) that using last three methods other than the first one will fetch all records of employee from database then by enumerating all records those will be returened which fulfill the specified where condition, so i think last three ways will take more time than the first one so i used only first one.
But what is the best one in actual i dont know.
Do all method only fetch records form database which fulfill the were condition or fetch all records from db and then enumrated to return filtered records at applicaion end ?

There is little difference; the 2nd/3rd have the advantage of enforcing type correctness at compile-time, since the expression-tree won't compile if LastName doesn't exist or is (say) an int - avoids typos and the issue of "magic strings".
Re your supposition about bringing back everything; no. The 2nd/3rd compile to expression trees, which can be de-constructed and used to create the correct SQL (performing the filter at the server). Very clever, but often misunderstood. The key point is that this is Queryable.Where (taking an Expression<Func<T, bool>>), not Enumerable.Where (taking a Func<T, bool>). Feel free to perform a trace.
Likewise, there is little difference between the 1st/4th - both will filter at the server.

LINQ: adding where clause only when a value is not null

I know a typical way is like this:
IQueryable query = from staff in dataContext.Staffs;
if(name1 != null)
{
query = from staff in query where (staff.name == name1);
}
However, from a program we took over from other developers, we saw code like this:
IQueryable query = from staff in dataContext.Staffs;
query = from staff in query where (name1 == null || staff.name == name1);
If this is a normal SQL statement, I would definitely say that the 2nd one is a bad practice. Because it adds a meaningless where clause to the query when name1 is null.
But I am new to LINQ, so I am not sure if LINQ is different?

you can write it like
IQueryable query = from staff in dataContext.Staffs;
query = from staff in query where (name1 != null && staff.name == name1);
This way second part of your condition will not be evaluated if your first condition evaluates to false
Update:
if you write
IQueryable query = from staff in dataContext.Staffs;
query = from staff in query where (name1 == null || staff.name == name1);
and name1 is null second part of your condition will not be evaluated since or condition only requires one condition to return true
plz see this link for further detail

Often this sort of thing feels smoother to write using the fluent syntax, rather than the query syntax.
e.g.
IQueryable query = dataContext.Staffs;
if(name1 != null)
{
query = query.Where(x => x.name == name1);
}
So if name1 is null, you just don't do any Where() call. If you have multiple different filters, all of which may or may not be required, and perhaps various different sort orders, I find this becomes a lot more manageable.
Edit for alex: OK, I was answering the question about adding a where clause only when a value is not null. In response to the other part of the question, I tried this out with Entity Framework 4 to see what SQL that LINQ produced. You do this by casting query to an ObjectQuery and calling .ToTraceString(). The results were that the WHERE clause came out as follows:
WHERE #p__linq__0 IS NULL OR [Extent1].[name] = #p__linq__1
So, yes, it's classic bad SQL, if you have an index on the name column, don't expect it to be used.
Edit #2: Tried this again using LINQ to SQL rather than Entity Framework, with rather different results. This time, trying the query with name1 being null results in no WHERE clause at all, as you'd hope; trying it with name1 being "a" resulted in a simple WHERE [t0].[name] = #p0 and #p0 sent as "a". Entity Framework does not seem to optimize thus. That's a bit worrying.

The best way to do this is to create yourself an extension method that will take in a conditional statement and a where expression. If the condition is true then it will use the where expression else it will not use it. This can dramatically clean up your code, eliminating the need for if statements.
public static class LinqExtensions
{
public static IQueryable<T> WhereIf<T>(this IQueryable<T> query, bool condition, Expression<Func<T, bool>> whereClause)
{
if (condition)
{
return query.Where(whereClause);
}
return query;
}
}
Now you can write your code like this:
IQueryable<Staffs> query = dataContext.Staffs.AsQueryable().WhereIf(name1 != null, x => x.Name == name1);

So I tried the .Where(..., x => ...) extension method listed here as an answer but it doesn't work against Entity Framework as Linq To Entities doesn't know how to translate that into TSQL.
So here's my solution getting my Func on:
Expression<Func<SomeEfPoco, bool>> columnBeingFilteredPredicate = x => true; // Default expression to just say yes
if (!string.IsNullOrWhiteSpace(someColumnBeingFilteredValue))
{
columnBeingFilteredPredicate = x => x.someColumnBeingFiltered == someColumnBeingFilteredValue;
}
_context.SomeEfPocos.Where(x => ..... &&
..... &&
..... &&)
.Where(columnBeingFilteredPredicate);
someColumnBeingFilteredValue in my case is a string parameter on the encapsulating method with a default value of NULL.

LINQ is diffrent in some other causes (not in this causes),
LINQ is the way to get data in the "Faster way" with a littel code and clear cod as possible, there a many benefits of LINQ:
Makes it easier to transform data into objects. I'm sure you've heard the term "Impedence Mismatch" being used quite often, meaning that LINQ reduces the amount of work you must do to translate between object-oriented code and data paradigms such as hierarchical, flat-file, messages, relational, and more. It doesn't eliminate the "Impedence Mismatch" because you must still reason about your data in its native form, but the bridge from here to there is (IMO) much shorter.
A common syntax for all data. Once you learn query syntax, you can use it with any LINQ provider. I think this is a much better development paradigm than the Tower of Babel that has grown over the years with data access technologies. Of course, each LINQ provider has unique nuances that are necessary, but the basic approach and query syntax is the same.
Strongly typed code. The C# (or VB.NET) query syntax is part of the language and you code with C# types, which are translated into something a provider understands. This means that you gain the productivity of having your compiler find errors earlier in the development lifecycle than elsewhere. Granted, many errors in stored proc syntax will generate errors when you save, but LINQ is more general than SQL Server. You have to think of all the other types of data sources that generate runtime errors because their queries are formed with strings or some other loosely typed mechanism.
Provider integration. Pulling together data sources is very easy. For example, you can use LINQ to Objects, LINQ to SQL, and LINQ to XML together for some very sophisticated scenarios. I think it's very elegant.
Reduction in work. Before LINQ, I spent a lot of time building DALs, but now my DataContext is the DAL. I've used OPFs too, but now I have LINQ that ships with multiple providers in the box and many other 3rd party providers, giving me the benefits from my previous points. I can set up a LINQ to SQL DataContext in a minute (as fast as my computer and IDE can keep up).
Performance in the general case doesn't become an issue. SQL Server optimizes queries quite well these days, just like stored procs. Of course, there are still cases where stored procs are necessary for performance reasons. For example, I've found it smarter to use a stored proc when I had multiple interactions between tables with additional logic inside of a transaction. The communications overhead of trying to do the same task in code, in addition to getting the DTC involved in a distributed transaction made the choice for a stored proc more compelling. However, for a query that executes in a single statement, LINQ is my preferred choice because even if there was a small performance gain from a stored proc, the benefits in previous points (IMO) carry more weight.
Built-in security. One reason I preferred stored procs before LINQ was that they forced the use of parameters, helping to reduce SQL injection attacks. LINQ to SQL already parameterizes input, which is just as secure.
LINQ is declarative. A lot of attention is paid to working with LINQ to XML or LINQ to SQL, but LINQ to Objects is incredibly powerful. A typical example of LINQ to Objects is reading items from a string[]. However, that's just a small example. If you think about all of the IEnumerable collections (you can also query IEnumerable) that you work with every day, the opportunities are plentiful. i.e. Searching an ASP.NET ListBox control for selected items, performing set operations (such as Union) on two collections, or iterating through a List and running a lambda in a ForEach of each item. Once you begin to think in LINQ, which is declarative in nature, you can find many of your tasks to be simpler and more intuitive than the imperative techniques you use today.
I could probably go on, but I'd better stop there. Hopefully, this will provide a more positive view of how you could be more productive with LINQ and perhaps see it as a useful technology from a broader perspective.

I've seen this pattern in standard SQL, and it seems useful if you have several parameters that may be NULL. For example:
SELECT * FROM People WHERE ( #FirstName IS NULL OR FirstName = #FirstName )
AND ( #LastName IS NULL OR LastName = #LastName )
If you see this in LINQ, it's possible they just blindly translated their old SQL-queries.

I like use the Expression
e.g.
Expression<Func<Persons, bool>> expresionFinal = c => c.Active == true;
if (DateBirth.HasValue)
{
Expression<Func<Persons, bool>> expresionDate = c => (EntityFunctions.TruncateTime(c.DateBirth) == DateBirth);
expresionFinal = PredicateBuilder.And(expresionFinal, expresionDate);
}
IQueryable query = dataContext.Persons;
query = query.Where(expresionFinal);

For EF Core I broke it up like this:
IQueryable<Partners> recs = contextApi.Partners;
if (status != -1)
{
recs = recs.Where(i => i.Status == status);
}
recs = recs.OrderBy(i => i.Status).ThenBy(i => i.CompanyName);
foreach (var rec in recs)
{
}
I had to be explicit with my typing instead of relying on var.

I like the idea with Extension
public static IQueryable<T> WhereIf<T>(this IQueryable<T> query, bool condition, Expression<Func<T, bool>> whereClause)
=> condition ? query.Where(whereClause) : query;

No, I am not strongly agree with you.
here you just gave a simple logic
if(name1 != null)
// do your stuff
but what will happen if you do something different with the name1 that have null value..!!
Ok, now consider this situation.
In this example you shows how to handle possible null values in source collections.
An object collection such as an IEnumerable<T> can contain elements whose value is null.
If a source collection is null or contains an element whose value is null,
and your query does not handle null values, a NullReferenceException will be thrown when you execute the query.
Probably this could be a issue...

I use the extension method below. It's less flexible than the WhereIf extension from the other answers, but it's shorter to use.
public static IQueryable<T1> FilterBy<T1, T2>(this IQueryable<T1> query, T2 expectedValue, Expression<Func<T1, T2>> propertyAccessor)
{
if (propertyAccessor == null) throw new ArgumentNullException(nameof(propertyAccessor));
if (expectedValue == null) return query;
var equalExpr = Expression.Equal(propertyAccessor.Body, Expression.Constant(expectedValue, typeof(T2)));
var lambda = Expression.Lambda<Func<T1, bool>>(equalExpr, propertyAccessor.Parameters);
return query.Where(lambda);
}
It can be used like:
var query = dataContext.Staffs.FilterBy(name, s => s.Name);

All queries combined using a UNION, INTERSECT or EXCEPT operator must have an equal number of expressions in their target lists

I have a linq to sql query where I have to perform union two set of records.
And I do not have equal number of fields, so added the null
eg my psuedo code is
var Values=( from c in containers
some joins
select new PValues
{
regionid=r.regionid,
roomid=r.roomid,
floorid=r.floorid,
maxarea=r1.maxarea,
minarea=r1.minarea,
avgarea=r1.avgarea,
maxheight=r1.maxheight,
minheight=r1.minheight
})
.union
( from c in containers
some joins
select new PValues
{ regionid=j.regionid,
roomid=j.roomid,
floorid=j.floorid,
maxarea=null,
minarea=null,
avgarea=null,
maxheight=j1.maxheight,
minheight=j1.minheight
})
after googling some hours I came to understand that it is bug in 3.5 framework.
Now I want to retrieve the result.
How do I do that
I tried framing into two seperate iqueryable
var a= first query
var b =second query
ilist result =a.union b
This too results in the same error.
How should I form it
Thanks in advance
Regards
Hema

This is most likely the result of a known issue with LINQ to SQL. When a column value is referenced twice, it gets optimized out of the result set (even though you may need it to make unions line up).
The most general-purpose work-around is to use let statements:
var Values=(
from c in containers
some joins
//You'll need one let statement for each COLUMN, even if they share a value.
let maxAreaValue = null
let minAreaValue = null
let avgAreaValue = null
select new PValues
{
regionid=j.regionid,
roomid=j.roomid,
floorid=j.floorid,
maxarea=maxAreaValue,
minarea=minAreaValue,
avgarea=avgAreaValue,
maxheight=j1.maxheight,
minheight=j1.minheight
});
More information:
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=355734
http://slodge.blogspot.com/2009/01/linq-to-sql-and-union-all-queries.html
SqlException about UNION, INTERSECT and EXCEPT

In SQL, this typically means that you need to cast those nulls as the appropriate data type of the first part of the union, so make them the default values for the types you need, like "" or String.Empty for string values, or 0 for ints, etc...

Cast the nulls as mentioned by #GalacticJello, but also cast the first query's parameters that will have nulls in the other query to a nullable version of the type. For instance, if minarea is decimal, cast it to a decimal?; you can do this via: new Nullable(i.minarea) to ensure it infers the correct type.
It's probably inferring a different anonymous signature for both queries. Also, to mitigate the problems, make a class with these properties and change the query from:
select new { .. }
to
select new MyObj { .. }
And that will resolve it too.

Conditional shortcuts in LinqToSql query

Here's a little LinqToSql GOTCHA:
// Returns the number of counties in a state,
// or all counties in the USA if the state is null
public static int CountCounties(State s) {
var q =
from cy in County.GetTable() // my method to get the ITable
where (s == null || s.Code == cy.StateCode) // shortcut OR operator, right...?
select cy;
return q.Count();
}
Guess what - if you pass a null State object to this method, you get a null reference exception! It seems that LinqToSql doesn't use the || shortcut operator as a shortcut!
Answer credit goes to whoever proposes the best explanation & workaround for this.

If it's linq to sql then remeber that Linq is just parsing your query into SQL.
It is therefore sending both of your where clauses to the database, hence the exception. I dont find this surprising really, though it is arguably wrong.
You will just have to do an independant check.
if (!string.isNullOrEmpty(state.statecode)
q = q.where( s => s.code == state.statecode

This is not related to LINQ in general. In this case, the LINQ-to-SQL provider tries to parse the your lambda expression and make it a TSQL query. It cannot make too many assumptions based on your expression since it's trying to delegate most of the work to the database.
Long story short, the provider simply cannot translate it to SQL.