We Keep Coding

Building a common set of methods that can operate on any linq table

Problem: We make extensive use of a repository pattern to facilitate read/write operations on our datastore (MS SQL using LINQ) across multiple applications and subsections of functionality. We have series of methods that all do something similar to each other.
For example, we have the ProcessAndSortXXXXX class of methods.
private static IEnumerable<ClassErrorEntry> ProcessAndSortClassErrorLog(IQueryable<ClassErrorDb> queryable, string sortOrder)
{
var dynamic = queryable;
if (!String.IsNullOrEmpty(sortOrder.Trim()))
{
dynamic = dynamic.OrderBy(sortOrder);
}
return dynamic
.Select(l =>
new ClassErrorEntry(l.Id)
{
ClassId = l.ClassId,
Code = l.Code,
Message = l.Message,
Severity = l.Severity,
Target = l.Target
}
);
}
...and...
private static IEnumerable<ClassTimerLogEntry> ProcessAndSortClassTimerLog(IQueryable<ClassTimerDb> queryable, string sortOrder)
{
var dynamic = queryable;
if (!String.IsNullOrEmpty(sortOrder.Trim()))
{
dynamic = dynamic.OrderBy(sortOrder);
}
return dynamic
.Select(l =>
new ClassTimerLogEntry(l.Id)
{
ClassName = l.ClassName,
MethodName = l.MethodName,
StartTime = l.StartTime,
EndTime = l.EndTime,
ParentId = l.ParentId,
ExecutionOrder = l.ExecutionOrder
}
);
}
As you can tell by the code, they're all very similar until you look at the signature and then get to the the return statement where we're building out the instances of the ClassErrorEntry and ClassTimerLogEntry.
I want to build a utility method that I'll add into the base class that all of the repositories inherit from.
I want to be able to pass in arguments that can be used to instantiate the objects and pack them into the returning IEnumerable.
I found this post by ScottGu and that gets me most of what I need. It looks like this (from the sample in the documentation):
var query =
db.Customers.
Where("City = #0 and Orders.Count >= #1", "London", 10).
OrderBy("CompanyName").
Select("new(CompanyName as Name, Phone)");
Here's where I get stuck, though. I need a pointer or suggestion how I can pass in the LINQ tables and DataContext in a generic fashion so I can build out the dynamic query.
If I were to mock up the signature in pseudocode I think it would look something like this:
protected internal IEnumerable ProcessAndSort(IQueryable source, string selectClause, string whereClause, string orderByClause);
I realize that the finished signature may look different as we figure this out.
Thank you!
Update!
I now have code that works to generate an anonymous type but fails when converting to the concrete type.
public static IEnumerable<TResult> ProcessAndSort<T, TResult>(IQueryable<T> queryable,
string selector, Expression<Func<T, bool>> predicate, string sortOrder)
{
var dynamic = queryable.Where(predicate).AsQueryable();
if (!String.IsNullOrEmpty(sortOrder.Trim()))
{
dynamic = dynamic.OrderBy(sortOrder);
}
var result= dynamic.Select(selector).Cast<TResult>();
return result;
}
Here is the code that calls this method:
[TestMethod]
public void TestAnonymousClass()
{
var loggingContext = new LoggingDbDataContext(DatabaseConnectionString);
var repo = new LoggingRepository(loggingContext);
var result = repo.TestGetClassErrorLog(4407, 10, 0,
"new ( ClassId as ClassId, " +
"Code as Code, " +
"Message as Message, " +
"Severity as Severity, " +
"Target as Target )", "Target");
TestContext.WriteLine(result.ToList().Count.ToString());
}
The last line TestContext.WriteLine(result.ToList().Count.ToString()); throws the exception System.InvalidOperationException: No coercion operator is defined between types 'DynamicClass1' and 'Utilities.Logging.ClassErrorEntry'.
This chunk of code, though fails:
[TestMethod]
public void TestNamedClass()
{
var loggingContext = new LoggingDbDataContext(DatabaseConnectionString);
var repo = new LoggingRepository(loggingContext);
var result = repo.TestGetClassErrorLog(4407, 10, 0,
"new ClassErrorEntry(Id) { ClassId = ClassId, " +
"Code = Code, " +
"Message = Message, " +
"Severity = Severity, " +
"Target = Target }", "Target");
TestContext.WriteLine(result.ToList().Count.ToString());
}
This fails on a parsing error. Test method eModal.Repositories.Test.RepositoryBaseTest.TestConcreteClass threw exception:
System.Linq.Dynamic.ParseException: '(' expected, found 'ClassErrorEntry' ('Identifier') at char 19 in 'new ClassErrorEntry(Id) { ChassisAuthId = ChassisAuthId, Code = Code, Message = Message, Severity = Severity, Target = Target }'
I'm not sure that the character position is suspectas the 19th character position is a ( and the type passed into the Validate method indicates a position of 4, or the first 'C'.

I would completely advise you against making weakly typed queries just for the sake of code reuse.
Code reuse is for increasing maintainability, but weak typing can kill it, if used in a wrong way.
By writing your queries in plain text, you're effectively making the classes very hard to refactor and change, and introduce a lot of obscure dependencies.
I suggest you take a look at LinqKit that allows to combine Expressions. For example, we wrote a Paging method that splits query by pages and use it across the project with different types:
var query = CompiledQuery.Compile(
BuildFolderExpr( folder, false )
.Select( msg => selector.Invoke( msg, userId ) ) // re-use selector expression
.OrderBy( mv => mv.DateCreated, SortDirection.Descending )
.Paging() // re-use paging expression
.Expand() // LinqKit method that "injects" referenced expressions
)
public static Expression<Func<T1, T2, PagingParam, IQueryable<TItem>>> Paging<T1, T2, TItem>(
this Expression<Func<T1, T2, IQueryable<TItem>>> expr )
{
return ( T1 v1, T2 v2, PagingParam p ) => expr.Invoke( v1, v2 ).Skip( p.From ).Take( p.Count );
}
In my example, BuildMessageExpr returns a relatively simple select expression (which already depends on folder and another parameter), and different methods reuse this expression by applying filtering, ordering, getting count, further selecting with selector expression being passed as a parameter, et cetera. Once the query is created, it gets cached for future usage when parameters are similar.

It is not a direct answer to your question.
As you have said you have quite a lot code that look similar but return different types. If you will go ahead and look for generic implementation of this approach, the result may have a few hacks, you may still pass some uncomfortable SQL or check the type of the object or do some reflection kung-fu. You may still select this pass and actually someone can have a sensible idea that wouldn't look like a dirty hack.
The other option is to use a proper ORM with generic repository pattern and dependency injection(google link). Your data access layer will look much simpler and easier to maintain.

Using Linq with a DbDataReader

I am trying to make use of the code in this question to implement a query like this:
public void LoadLive(DbConnection pConnection)
{
using (DbDataReader lReader = pConnection.ExecuteReader("..."))
{
mList.AddRange(from t in lReader select new MyObject { Name = t.GetString(0) });
}
}
When I attempt to compile this (with the extension method in place), I receive this error:
error CS1934: Could not find an implementation of the query pattern for source type 'System.Data.Common.DbDataReader'. 'Select' not found. Consider explicitly specifying the type of the range variable 't'.
Am I missing something about how this is supposed to work?

You must call the extension method from the answer in the linked question:
mList.AddRange(from t in lReader.AsEnumerable()
select new MyObject { Name = t.GetString(0) });

Unless you are going to write your own extension method, Select, Where, etc all return an IEnumerable and the typical method signature will resemble Func<DbDataReader, myT>. What you are looking for is something like Jon Skeet's sample here.

Compiler error CS1934 is produced when no standard query operators are implemented for a given datasource.
In your case (DbDataReader), you could specify the type of t as IDataRecord:
mList.AddRange(from IDataRecord t in lReader
select new MyObject { Name = t.GetString(0) });

extending SqlMethods.Like to support property name

I'm trying to extend SqlMethods.Like method to support property name rather than property value, i wrote the following extension method :
public static bool Like(this object obj, string propertyName, string pattern)
{
var properties = obj.GetType().GetProperties().Select(p => p.Name);
if(!properties.Contains(propertyName))
throw new Exception(string.Format("Object does not contain property:{0}", propertyName));
return SqlMethods.Like(obj.GetType().GetProperty(propertyName).GetValue(obj, null).ToString(), pattern);
}
however the method throws the following exception :
Method 'Boolean Like(System.Object, System.String, System.String)' has no supported translation to SQL.
how can i write an extension method with transaction to SQL support ?

I found this answer from RichardD that is exactly the correct answer. Reposting for clarity, but original is linked below.
using System;
using System.Linq;
using System.Linq.Expressions;
public static class Extensions
{
public static IQueryable<T> WhereLike<T>(this IQueryable<T> source, string propertyName, string pattern)
{
if (null == source) throw new ArgumentNullException("source");
if (string.IsNullOrEmpty(propertyName)) throw new ArgumentNullException("propertyName");
var a = Expression.Parameter(typeof(T), "a");
var prop = Expression.Property(a, propertyName);
var body = Expression.Call(typeof(SqlMethods), "Like", null, prop, Expression.Constant(pattern));
var fn = Expression.Lambda<Func<T, bool>>(body, a);
return source.Where(fn);
}
}
...
.WhereLike("Description", "%a%b%c%"));
The solution uses expression trees, but all advanced LinqToSql operations will require familiarity with that.
From: http://forums.asp.net/p/1488418/3503874.aspx

What you want to do does not seem to make sense in the contxt of what SqlMethods.Like actually does. When you pass in a property of a class you are essentially telling it to translate that into the equivelent field in the SQL query. e.g.
var result = from names in db.Names
where SqlMethods.Like(names.FullName, '%Smith%')
select names;
would translate to something like:
SELECT *
FROM Names
WHERE Fullname LIKE '%Smith%'
(in practice it would be different using parameters and sp_executeSQL but coneptually that is what it would do).
If you want to pass in the name of a property what does that mean in terms of SQL, conceptually it makes no sense e.g.
SELECT *
FROM Names
WHERE --what would go here-- LIKE '%Smith%'
As such you are not going to be able to create a Linq To SQL method that creates nonsense SQL.
What are you actually trying to do, the chance is that you are going about it completely the wrong way.
Edit:hmm from your comment i think i understand what you want to do, in essense you want to be able to specify the column you are doing a LIKE comparison with at run time. You cannot do it exactly. You could use a stored procedure that used dynamic SQL and took a string parameter for the column. You could then expose this as a method on your data context class.

Reading a property value with a lambda expression

I'm writing a file generic block for my application and started with using Lambda expressions for managing my rule sets for block generation to avoid the pitfalls of magic strings, configuration hell etc.
Inside my mapping class I have lines similar to:
Map(x => x.Name).Length(20).PadLeft(true).PaddingChar("#");
This works fine and isn't where my question dwells, where I setup saving my information about the expression is in the Map method:
public override IPropertyMap Map(Expression<Func<T, object>> expression)
{
var propertyMap = new FixedLengthPropertyMap
{
//Length = 20,
//PaddingCharacter = " ",
PadLeft = false,
PropertyInfo = ReflectionHelper.GetProperty(expression)
};
_properties.Add(propertyMap);
return propertyMap;
}
_properties is just a List<IPropertyMap> that stores my info where my question from what is the best way to have a real object's data be read from the properties currently I came up with something similar to this:
var map = new AgentMap();
var agent = new Agent {Name = "Bob"};
string output = map.Write(agent);
public override string Write<T>(T agent)
{
var initial = _properties[0];
return initial.PropertyInfo.GetValue(agent, null) as string;
}
Is there a better way than using the GetValue method since earlier on I'm using an expression tree?

I don't see why you really need to use expression trees at all. Just make the Map method take a Func<T, object> and store that:
public override IPropertyMap Map(Func<T, string> fetcher)
{
var propertyMap = new FixedLengthPropertyMap
{
//Length = 20,
//PaddingCharacter = " ",
PadLeft = false,
Delegate = fetcher // Delegate is of type Delegate
};
_properties.Add(propertyMap);
return propertyMap;
}
Then:
public override string Write<T>(T agent)
{
var initial = _properties[0];
Func<T, string> fetcher = (Func<T, string>) initial.Delegate;
return fetcher(agent);
}
Is there any reason you particularly wanted to know the property and use an expression tree?

In part, it depends on what your scenario is. The "simple" answer is to just compile the expression and invoke it, but that has a potential performance impact if you are doing it in a tight loop (passing a delegate would be a lot quicker).
I'm not sure whether if would apply in this particular case (because of the agent), but to avoid doing too much expression compilation, you can look for simple scenarios and read the value directly from the expression tree; a little bit of PropertyInfo/FieldInfo is going to be quicker than compiling it...
For more, look at TryEvaluate here, and how it is used with Compile as a backup strategy (although you have the advantage of a known delegate type).

Retrieving an Expression from a property and adding it to an expression tree

I've tried to simplify this example, as the actual code I'm playing with is more complex. So while this example may seem silly, bear with me. Let's say I'm working with the AdventureWorks database and I decide I want to add a property called Blarg to the Product table that returns an expression that contains code I would like to use in several places:
public partial class Product
{
public Expression<Func<Product, string>> Blarg
{
get { return product => product.ProductModelID.HasValue ? "Blarg?" : "Blarg!"; }
}
}
What I want to do is create an expression expression tree, have it get the Expression from Product.Blarg, and group by the result. Something like this:
var productParameter = Expression.Parameter(typeof(Product), "product");
// The Problem
var groupExpression = Expression.Lambda<Func<Product, string>>(
Expression.Invoke(
Expression.Property(productParameter, "Blarg"),
productParameter),
productParameter);
using (AdventureWorksDataContext db = new AdventureWorksDataContext())
{
var result = db.Products.GroupBy(groupExpression).ToList();
// Throws ArgumentException: "The argument 'value' was the wrong type.
// Expected 'System.Delegate'.
// Actual 'System.Linq.Expressions.Expression`1[System.Func`2[LINQ_Test.Product,System.String]]'."
}
Obviously groupExpression is incorrect (see the code comment for the exception), but I'm not sure how I should be doing it. What I thought I was saying is "get the Expression from the product.Blarg, execute it, and return the string result." I guess that's not what I'm actually saying there, though. I'm still trying to figure out expression trees. Any idea how I could pull this off?

When you call Expression.Invoke, the first argument must be an existing LambdaExpression - it can't be an Expression to a LambdaExpression. Or in other words: it isn't going to evaluate Product.Blarg per row and use a different sub-expression each time.
Instead, you would retrieve this lambda first, perhaps making it static and accessing it via reflection if you only know it by name:
var lambda = (LambdaExpression) typeof(Product)
.GetProperty("Blarg").GetValue(null,null);
And pass lambda in as the argument to Expression.Invoke; here's a fully working LINQ-to-Objects example showing this (via AsQueryable()):
using System;
using System.Linq;
using System.Linq.Expressions;
public partial class Product
{
public static Expression<Func<Product, string>> Blarg
{
get { return product => product.ProductModelID.HasValue ? "Blarg?" : "Blarg!"; }
}
public int? ProductModelID { get; set; }
static void Main()
{
var lambda = (LambdaExpression)typeof(Product)
.GetProperty("Blarg").GetValue(null, null);
var productParameter = Expression.Parameter(typeof(Product), "product");
// The Problem
var groupExpression = Expression.Lambda<Func<Product, string>>(
Expression.Invoke(
lambda,
productParameter),
productParameter);
var data = new[] {
new Product { ProductModelID = 123},
new Product { ProductModelID = null},
new Product { ProductModelID = 456},
};
var qry = data.AsQueryable().GroupBy(groupExpression).ToList();
}
}

var qry = data.AsQueryable().GroupBy(Blarg).ToList();
That works, same as Marc's code.
Note: Blarg is already correct, there is no reason to 're-invoke' it.