We Keep Coding

Convert LINQ Expression to SQL Text without DB Context

Either LINQ to SQL or LINQ to Entities already have the ability to convert LINQ into a SQL text string. But I want my application to make the conversion without using the db context - which in turn means an active database connection - that both those providers require.
I'd like to convert a LINQ expression into an equivalent SQL string(s) for WHERE and ORDER BY clauses, without a DB context dependency, to make the following repository interface work:
public interface IStore<T> where T : class
{
void Add(T item);
void Remove(T item);
void Update(T item);
T FindByID(Guid id);
//sure could use a LINQ to SQL converter!
IEnumerable<T> Find(Expression<Func<T, bool>> predicate);
IEnumerable<T> FindAll();
}
QUESTION
It is primarily the expression tree traversal and transform I am interested in. Does anyone know of an existing library (nuget?) that I can incorporate to be used in such a custom context?
As it is I've already built my own working "LINQ transformed to SQL text" tool, similar to this expression tree to SQL example which works in my above repository. It allows me to write code like this:
IRepository<Person> repo = new PersonRepository();
var maxWeight = 170;
var results = repo.Find(x => (x.Age > 40 || x.Age < 20) && x.Weight < maxWeight);
But my code and that sample are primitive (and that sample itself relies on a LINQ to SQL db context). For example, neither handle generation of "LIKE" statements.
I don't expect or need a generator-tool that handles every conceivable LINQ query. For example, I'm not worried about handling and generating joins or includes. In fact, with another ~20 hours my own custom code may cover all the cases that I care about (mostly "WHERE" and "ORDER BY" statements).
But at the same time I feel that I should not have to write my own custom code to do this. If I'm stuck writing my own, then I'd still be interested if someone could point me to specific classes I can reflect and imitate (NHibernate, EF, etc.). I'm asking about specific classes to peek at, if you know them, because I don't want to spend hours sifting through the code of a massive tool just to find the part I need.
Not that it matters, but if anyone wants to know why I'm not simply using LINQ to SQL or LINQ to Entities...for my specific application I simply prefer to use a tool such as Dapper.
USE CASES
Whether I finish building the tool myself, or find a 3rd party library, here are reasons why a "LINQ to SQL text string" would be useful:
The predicate I type into the IRepository.Find method has intellisense and basic compile-time checking.
My proposed IStore interface can be implemented for DB access or web service access. To clarify, if I can convert the LINQ "WHERE/ORDER BY" predicate to a SQL "WHERE/ORDER BY" clause then...
The SQL string could be used by Dapper directly.
The SQL string, unlike a LINQ expression, can be sent to a WCF service to be used for direct DB access (which itself might not be using Dapper).
The SQL string could be deserialized, with custom code, back into a LINQ statement by the WCF service. Eric Lippert comments on this.
The UI can use IQueryable mechanics to dynamically generate a predicate to give to the repository
In short, such a tool helps fulfill the "specification" or "query object" notion of repositories according to DDD, and does so without taking a dependency on EF or LINQ to SQL.

Doing this properly is really extremely complicated, especially if right now, you don't seem to know much about expression trees (which is what IQueryable uses to represent queries).
But if you really want to get started (or just get an idea of how much work it would be), have a look at Matt Warren's 17-part series Building an IQueryable provider.

I can confirm as this is a fairly big amount of work that’s suited only for the most experienced .NET developers. Perfect knowledge of C#, experience with multiple languages, including T-SQL is a must. One must be very well versed in both C# (or VB.NET) and T-SQL as they’ll have to write translator using the former into the latter. Additionally, this is in the realm of meta-programming, which is considered a fairly advanced branch of computer science. There is a lot of abstract thinking involved. Layers of abstract concepts stacked on each other.
If all of this isn’t a barrier, then this exercise can actually be quite enjoyable and rewarding, at least the first month or so. One common problem in these providers I noticed is that inflexibility and questionable design choices at the start led to difficulties later on and hacky fixes, etc. Planning as much as possible in advance, clearly understanding the whole process, different stages, components properly identifying layers and concerns would make it much easier to develop this. The biggest mistake I saw in one provider was – failing to break down the output query into its parts – select, from, where and order by. Each part should be represented by its own object throughout and then put together at the end. I explain this approach in my end-to-end tutorial on how to write a provider in the series linked below. There’s a sample project included, with a simpliefied/tutorial variant and the full version made from scratch for a project. Finding the time to write about it was a challenge in itself.
How to write a LINQ to SQL provider in C#:
Introduction
Expression Visitor
Where Clause Visitor
Compiling Expression Trees

This is something I briefly looked into quite a while ago. You may want to have a look at http://iqtoolkit.codeplex.com/ and/or http://expressiontree.codeplex.com/ for ideas. As mentioned by others, Linq query provider building is far from trivial if you do not limit your scope to the minimum set of features you really need.
If your goals relate to "specification" or "query object" notion of repositories according to DDD, this may not be the best direction to take. Instead of CRUD like technology related abstractions, it may be more productive to focus on ways in which the behaviour of the domain can be expressed, with a minimum of direct dependencies on technology related abstractions. As Eric Evans recently discussed, he regrets the focus on the technical building blocks, such as repositories, in his initial descriptions of DDD.

Which Array Type Should I Choose for Easy Conversion From C# to Java?

i am developing a data access layer in c# which will retrive data from multiple datasources. And i want to use the same DAL layer for the java version of my project.
i wonder the point that i should care about to achive this. For the first, i want to know which type i should use for returning a list of objects. Should i consider using List, ArrayList or Array in c# to achive an easy way of converting from c# to java?
Thanks.

(From your comments) It doesn't matter what the most common list type is. Your DAL should return a type that suits the needs of the consuming code. If you are just going to port the code from C# to Java then you don't need to think about 'what is the easiest type to convert'. You just need to use the type which most accurately serves your needs in each language.
Don't let 'ease of porting the code' influence your decision. This may unnecessarily distort your code in both places. IMHO it's more valuable for the code to make sense in each place and spend a little extra effort in porting the code. Having said that, the choice of return type in each case is unlikely to change the effort required to port the code very much at all.
Even if you are going to attempt some kind of communication between C# and Java then the return type from your DAL should still be driven by the above logic and your communication layer should do any necessary translation into interoperable types.

It depends on what you are doing:
for custom code, IList<T> interface is ok
For api exposed, ICollection<T> is more suitable.

You could use a List<T>. ArrayList is deprecated. The return type of your method could also be be IEnumerable<T> so that it won't matter which type of collection, array or list you're returning.

What is System.Linq.Expressions in C# used for?

Is LINQ a new feature in .NET 4.0, unsupported in older versions like .NET 3.5? What is it useful for? It seems to be able to build Expression Trees. What is an Expression Tree, actually? Is LINQ able to extract info like class, method and field from a C# file?
Can someone provide me a working piece of code to demonstrate what LINQ can do?

Linq was added in .Net 3.5 (and added to the c# 3.0 compiler as well as in slightly limited form to the VB.net compiler in the same release)
In is language integrated query, though it covers many complex additions to both the language and the runtime in order to achieve this which are useful in and of themselves.
The Expression functionality is simply put the ability for a program, at runtime, inspect the abstract syntax of certain code constructs passed around. These are called lambdas. And are, in essence a way of writing anonymous functions more easily whilst making runtime introspection of their structure easier.
The 'SQL like' functionality Linq is most closely associated with (though by no means the only one) is called Linq to Sql where by something like this:
from f in Foo where s.Blah == "wibble" select f.Wobble;
is compiled into a representation of this query, rather than simply code to execute the query. The part that makes it linq to sql is the 'backend' which converts it into sql. For this the expression is translated into sql server statements to execute the query against a linked database with mapping from rows to .net objects and conversion of the c# logic into equivalent where clauses. You could apply exactly the same code if Foo was a collection of plain .net objects (at which point it is "Linq to objects") the conversion of the expression would then be to straight .Net code.
The lambda above written in the language integrated way is actually the equivalent of:
Foo.Where(f => f.Blah == "wibble).Select(f => f.Wobble);
Where Foo is a typed collection. For databases classes are synthesized to represent the values in the database to allow this to both compile, and to allow round tripping values from the sql areas to the .net areas and vice versa.
The critical aspect of the Language Integrated part of Linq is that the resulting language constructs are first class parts of the resulting code. Rather than simply resulting in a function they provide the way the function was constructed (as an expression) so that other aspects of the program can manipulate it.
Consumers of this functionality may simply chose to run it (execute the function which the lambda is compiled to) or to ask for the expression which describes it and then do something different with it.
Many aspects of what makes this possible are placed under the "Linq" banner despite not really being Linq themsleves.
For example anonymous types are required for easy use of projection (choosing a subset of the possible properties) but anonymous types can be used outside of Linq as well.
Linq, especially via the lambdas (which make writing anonymous delegates very lightweight in terms of syntax) has lead to an increase in the functional capabilities of c#. this is reinforced by the extension methods on IEnumerable<T> like Select(), corresponding to map in many function languages and Where() corresponding to filter. Like the anonymous types this is not in and of itself "Linq" though is viewed by many as a strongly beneficial effect on c# development (this is not a universal view but is widely held).
For an introduction to Linq from microsoft read this article
For an introduction to how to use Linq-to-Sql in Visual Studio see this series from Scott Guthrie
For a guide to how you can use linq to make plain c# easier when using collections read this article
Expressions are a more advanced topic, and understanding of them is entirely unecessary to use linq, though certain 'tricks' are possible using them.
In general you would care about Expressions only if you were attempting to write linq providers which is code to take an expression rather than just a function and use that to do something other than what the plain function would do, like talk to an external data source.
Here are some Linq Provider examples
A multi part guide to implementing your own provider
The MDSN documentation for the namespace
Other uses would be when you wish to get some meta data about what the internals of the function is doing, perhaps then compiling the expression (resulting in a delegate which will allow you to execute the expression as a function) and doing something with it or just looking at the metadata of the objects to do reflective code which is compile time verified as this answer shows.

One area of this question that hasn't been covered yet is expression trees. There is a really good article on expression trees (and lambda expression) available here.
The other important thing to bring up about expression trees is that by building an expression tree to define what you are going to do, you don't have to actually do anything. I am referring to deferred execution.
//this code will only build the expression tree
var itemsInStock = from item in warehouse.Items
where item.Quantity > 0;
// this code will cause the actual execution
Console.WriteLine("Items in stock: {0}", itemsInStock.Count());

LINQ was introduced with .NET 3.5. This site has a lot of examples.

System.Linq.Expressions is for hand building (or machine generating) expression trees. I have a feeling that given the complexity of building more complicated functionality that this namespace is under used. However it is exceedingly powerful. For instance one of my co workers recently implemented an expression tree that can auto scale any LINQ to SQL object using a cumultive density function. Every column gets its own tree that gets compiled so its fast. I have been building a specialized compiler that uses them extensively to implement basic functionality as well as glue the rest of the generated code together.
Please see this blog post for more information and ideas.

LINQ is a .NET 3.5 feature with built-in language support from C# 3.0 and Visual Basic 2008. There are plenty of examples on MSDN.

What is the Efficiency and Performance of LINQ and Lambda Expression in .Net?

I have used .Net 3.5 and VS 2008 for more than a month. Like most .Net developers, I have evolved from years experience in .Net 1.0 & 2.0 and VS 2005. Just recently, I discovered the simplicity and power of LINQ and Lambda Expressions, as in my recent questions such as Find an item in list by LINQ, Convert or map a class instance to a list of another one by using Lambda or LINQ, and Convert or map a list of class to another list of class by using Lambda or LINQ.
I admit that Lambda and LINQ are much simpler and easy to read and they seem very powerful. Behind the scenes, the .Net compiler must generate lots of code to achieve those functions. Therefore I am little bit hesitant to switch to the new syntax since I already know the "old" way to achieve the same results.
My question is the about the efficiency and performance of Lambda and LINQ. Maybe Lambda expressions are mostly in-line functions, in that case I guess Lambda should be okay. How about LINQ?
Let's limit the discussion to LINQ-to-Objects LINQ-to-SQL (LINQ-to-SQL). Any comments, comparison and experiences?

There's no one single answer that will suffice here.
LINQ has many uses, and many implementations, and thus many implications to the efficiency of your code.
As with every piece of technology at our fingertips, LINQ can and will be abused and misused alike, and the ability to distinguish between that, and proper usage, is only dependent on one thing: knowledge.
So the best advice I can give you is to go and read up on how LINQ is really implemented.
Things you should check into are:
LINQ and how it uses the methods and extension methods on existing collection types
How LINQ Works
How LINQ works internally (Stack Overflow)
How does coding with LINQ work? What happens behind the scenes?
How LINQ-to-objects and LINQ-to-SQL differs
What is the difference between LINQ query expressions and extension methods (Stack Overflow)
Alternatives to the new LINQ syntax, for instance, the usage of the .Where(...) extension method for collections
And as always, when looking at efficiency questions, the only safe approach is just to measure. Create a piece of code using LINQ that does a single, know, thing, and create an alternative, then measure both, and try to improve. Guessing and assuming will only lead to bad results.

Technically the fastest way is to control all the minutia yourself. Here are some performance tests. Notice that the foreach keyword and the ForEach LINQ construct are identically far far slower than just using for and writing procedural code.
However, the compiler can and will be improved and you can always profile your code and optimize any problematic areas. It is generally recommended to use the more expressive features that make code easier to read unless you really need the extra nanoseconds.

For LINQ queries, with the 'new syntax', the IL (code) generated, is fundamentally no different than calling the extension methods provided by Enumerable and Queryable directly.

Dont optimize prematurely. Use Linq and the new extension methods liberally if they improve readability and profile your application afterwards.
Most times the difference between Linq and using plain for loops is not relevant at all. The improved maintainability of your code should be worth a few ms.
Linq can be slower because it works on enumerators which are implemented as state machines. So plain for(...) loops will be faster.
I would recommend following Lasse V. Karlsens advice and append http://www.davesquared.net/2009/07/enumerables-linq-and-speed.html to his link list.

There is no performance difference between LINQ queries and Lambda expressions.
You should completely understand how LINQ feature(both Lambda, LINQ queries) works in .Net before you are looking into performance issues.
Basically you can work with any one of both LINQ queries and Lambda expressions..
LINQ Queries
It is high level readable query.
It is converted into equalent Lambda
expressions and Lambda expressions added as nodes into an expression tree. Expression tree
which makes structure of lambda expressions. This is
done by compiler.
Query provider looks into
expressions(added as nodes in expression tree) and
produces equalent SQL query operators thus equalent sql query formed during
runtime.
Return Type : Result set (IEnumerable).
Lambda Expressions
It is a set of expressions/statements and creates delegate /
expression tree. It can
be passed to a function as an
argument.
It supports all the LINQ methods like LINQ queries.
(Where,Select,Count,Sum,etc)
An expression tree formed which
makes structure of lambda
expressions. This is done by
compiler.
Query provider looks into
expressions(expression tree) and
produces equalent SQL query during
runtime.
Return Type : Delagate /
Expression Tree
Which is Best?
You can understand the LINQ (Queries,Lambda) If you look into the above points.
Advantage of LINQ query - It is readable.
Advantage of Lambda
Lambda will have an advantage since it creates a delegate and by using
the delagte you can just pass the
input paremeters and get the result
for the different input
parameters.You need not write
different queries for different
criteria as well.
You can create dynamic query by
using Lambda expressions and
Expression trees.
You can use Lambda expressions if
you want to pass the result of
statement(s) to a method as an
argument.
expressions are shorter.
So Lambda expression is the best for development over LINQ queries.

In some cases LINQ is just as fast if not faster than other methods, but in other cases it can be slower. We work on a project that we converted to linq and the data lookup is faster but the merging of data between two tables is much slower. There is a little overhead, but in most cases I don't see the speed difference having much effect on your program.

Set operation in .NET C#

I'm working on a something related to roughset right now. The project uses alot of sets operation and manipulation. I've been using string operations as a stop gap measure for set operation. It has worked fine until we need to process some ungodly amount of data ( 500,000 records with about 40+ columns each ) through the algorithm.
I know that there is no set data structure in .net 2.0(2.0 was the latest when I started the project) I want to know if there is any library that offer fast set operation in .net c# or if 3.5 has added native set data structure.
Thanks .

.NET 3.5 already has a native set data type: HashSet. You might also want to look at HashSet and LINQ set operators for the operations.
In .NET 1.0, there was a third party Set data type: Iesi.Collections which was extended with .NET 2.0 generics with Iesi.Collections.Generic.
You might want to try and look at all of them to see which one would benefit you the most. :)

LINQ supports some set operations. See LINQ 101 page for examples.
Also there is a class HashSet (.NET 3.5)
Here is Microsoft guidelines for set operations in .NET:
HashSet and LINQ Set Operations
List of set operations supported by HasSet class:
HashSet Collection Type

Update: This is for .Net 2.0. For .Net 3.5, refer posts by aku, Jon..
This is a good reference for efficiently representing sets in .Net.

It may be worth taking a look at C5, it's a generic collection library for .NET which includes sets.
Note that I haven't looked into it much, but it seems to be a pretty fantastic collection library.

Try HashSet in .NET 3.5.
This page from a member of the .NET BCL team has some good information on the intent of HashSet

I have been abusing the Dictionary class in .NET 2.0 as a set:
private object dummy = "ok";
public void Add(object el) {
dict[el] = dummy;
}
public bool Contains(object el) {
return dict.ContainsKey(el);
}

You can use Linq to Objects in C# 3.0.

You ever think about sing F#? This seems like a job for a functional programming language.

You should take a look at C5 Generic Collection Library. This library is a systematic approach to fix holes in .NET class library by providing missing structures, as well as replacing existing ones with set of well designed interfaces and generic classes.
Among others, there is HashSet<T> - generic Set class based on linear hashing.