We Keep Coding

Using custom method inside Linq Select with Entity Framework

I am trying to use a custom Function inside a Linq Select that is used with EF.
I want to project each item of tblMitarbeiter onto one tblMitarbeiterPersonalkostenstelleHistories that is valid ad the given date.
This should be done with an extension method so that I do not repeat myself ;)
I can only get it to work when used directly on the DbSet, but not inside a Select.
How can I teach EF to recognize my Method (3.) as if I would be writing it out (1.)?
void Main()
{
var date = DateTime.Now;
// 1. works, returns IEnumerable<tblMitarbeiterPersonalkostenstelleHistories>
tblMitarbeiters
.Select(m => m.tblMitarbeiterPersonalkostenstelleHistories.Where(p => p.ZuordnungGültigAb <= date).OrderByDescending(p => p.ZuordnungGültigAb).FirstOrDefault())
.Dump();
// 2. works, returns one tblMitarbeiterPersonalkostenstelleHistories
tblMitarbeiterPersonalkostenstelleHistories
.GetValidItemForDate(p => p.ZuordnungGültigAb, date)
.Dump();
// 3. throws NotSupportedException
tblMitarbeiters
.Select(m => m.tblMitarbeiterPersonalkostenstelleHistories.GetValidItemForDate(p => p.ZuordnungGültigAb, date))
.Dump();
// 4. throws NotSupportedException
tblMitarbeiters
.Select(m => m.tblMitarbeiterPersonalkostenstelleHistories.AsQueryable().GetValidItemForDate(p => p.ZuordnungGültigAb, date))
.Dump();
}
public static class QueryableExtensions
{
public static T GetValidItemForDate<T>(this IQueryable<T> source, Expression<Func<T, DateTime>> selector, DateTime date)
{
var dateAccessor = Expression.Lambda<Func<T, DateTime>>(Expression.Constant(date), selector.Parameters);
var lessThanOrEqual = Expression.LessThanOrEqual(selector.Body, dateAccessor.Body);
var lambda = Expression.Lambda<Func<T, bool>>(lessThanOrEqual, selector.Parameters);
return source.Where(lambda).OrderByDescending(selector).FirstOrDefault();
}
public static T GetValidItemForDate<T>(this IEnumerable<T> source, Func<T, DateTime> selector, DateTime date) =>
source.Where(i => selector(i) <= date).OrderByDescending(selector).FirstOrDefault();
}

You can, to some extent, split up complex LINQ expressions using LINQKit. If you'll excuse me, I'll use an example model that's less germanic:
public class Employee
{
public long Id { get; set; }
public virtual ICollection<EmployeeHistoryRecord> HistoryRecords { get; set; }
}
public class EmployeeHistoryRecord
{
public long Id { get; set; }
public DateTime ValidFrom { get; set; }
public long EmployeeId { get; set; }
public Employee Employee { get; set; }
}
If I understood your question correctly, it should be identical to yours where it matters.
When using LINQKit, and LINQ in general, you must understand that the only tool you have at your disposal when reusing query code, without using stored procedures, is breaking apart and stitching together expressions.
Your utility method would translate to something like this:
private static Expression<Func<IEnumerable<TItem>, TItem>> GetValidItemForDate<TItem>(
Expression<Func<TItem, DateTime>> dateSelector,
DateTime date)
{
return Linq.Expr((IEnumerable<TItem> items) =>
items.Where(it => dateSelector.Invoke(it) <= date)
.OrderByDescending(it => dateSelector.Invoke(it))
.FirstOrDefault())
.Expand();
}
What this method does is dynamically create an expression whose input is an IEnumerable<TItem> that returns a TITem. You can see it's pretty similar to the code you're extracting. A few things to note:
The source collection is not a parameter of the utility method, but of the expression returned.
You have to call the Invoke() extension method from LinqKit on any expressions you're "plugging into" this one.
You should call Expand() on the result if you used any Invoke()s inside it. This will make LINQKit replace the calls to Invoke() in the expression tree with the expression being invoked. (This isn't 100% necessary, but it makes it easier to fix errors when expansion fails for some reason. If you don't Expand() in every helper method, any error that happens during expansion will manifest in the method that does the expansion, and not in the method that actually contains the offending code.)
You then use this similarly, again using Invoke():
var db = new EmployeeHistoryContext();
var getValidItemForDate = GetValidItemForDate((EmployeeHistoryRecord cab) => cab.ValidFrom, DateTime.Now);
var historyRecords = db.Employees.AsExpandable().Select(emp => getValidItemForDate.Invoke(emp.HistoryRecords));
(I've only tested this code against an empty database, insofar that it doesn't make EntityFramework throw a NotSupportedException.)
Here, you should note:
The subexpression you're plugging into the one you're passing into Select() needs to be saved in a local variable, LINQKit doesn't support method calls during expansion.
You need to call AsExpandable() on the first IQueryable in the chain, so LINQKit gets to work its magic.
You're probably not going to be able to use extension method call syntax inside the expression like in your question.
All the subexpressions have to be determined before expansion occurs.
These limitations stem from the fact that what you're doing isn't really calling methods. You're building one ginormous expression from a bunch of smaller ones, but the resulting expression itself still has to be something that LINQ-to-Entities will understand. On the other hand, the input has to be something LINQKit will understand, and it only handles expressions of the form localVariable.Invoke(). Any dynamism has to be in the code outside this expression tree. Basically, it's doing the same as your solution 2, just using syntax more intuitive than building the expression tree programmatically.
Last, but not least: when doing this, do not go overboard. Complex EF queries are already really hard to debug when anything goes wrong, because you're not told where in your code the problem is. If the query was assembled dynamically from bits and pieces all over your codebase, debugging some errors (like the delightful "Unable to cast the type X to type Y") will easily become a nightmare.
(For future questions: I think it's usually a good idea when if you make a code sample from scratch, instead of using bits from your actual codebase. They might be overly domain-specific, and understanding the names might require some context you take for granted. Identifiers should ideally be simple English names everyone can understand. I can maybe speak enough German to interview for a job in it, but "Mitarbeiterpersonalkostenstellehistorie" is just hard to keep in my head and reason about when I haven't actually worked on your project long enough to be familiar with what it's supposed to mean.)

Mock DbFunctions [duplicate]

I'm currently attempting to run some unit tests on a query that is running through the Entity Framework. The query itself runs without any issues on the live version, but the unit tests are always failing.
I've narrowed this down to my usage of DbFunctions.TruncateTime, but I don't know of a way around this to get the unit tests to reflect what is happening on the live server.
Here is the method that I am using:
public System.Data.DataTable GetLinkedUsers(int parentUserId)
{
var today = DateTime.Now.Date;
var query = from up in DB.par_UserPlacement
where up.MentorId == mentorUserId
&& DbFunctions.TruncateTime(today) >= DbFunctions.TruncateTime(up.StartDate)
&& DbFunctions.TruncateTime(today) <= DbFunctions.TruncateTime(up.EndDate)
select new
{
up.UserPlacementId,
up.Users.UserId,
up.Users.FirstName,
up.Users.LastName,
up.Placements.PlacementId,
up.Placements.PlacementName,
up.StartDate,
up.EndDate,
};
query = query.OrderBy(up => up.EndDate);
return this.RunQueryToDataTable(query);
}
If I comment out the lines with DbFunctions in, the tests all pass (except for the ones that are checking that only valid results for a given date are run).
Is there a way I can provide a mocked version of DbFunctions.TruncateTime to use in these tests? Essentially it should just be returning Datetime.Date, but that isn't available in EF queries.
Edit: Here's the test that's failing that uses the date check:
[TestMethod]
public void CanOnlyGetCurrentLinkedUsers()
{
var up = new List<par_UserPlacement>
{
this.UserPlacementFactory(1, 2, 1), // Create a user placement that is current
this.UserPlacementFactory(1, 3, 2, false) // Create a user placement that is not current
}.AsQueryable();
var set = DLTestHelper.GetMockSet<par_UserPlacement>(up);
var context = DLTestHelper.Context;
context.Setup(c => c.par_UserPlacement).Returns(set.Object);
var getter = DLTestHelper.New<LinqUserGetLinkedUsersForParentUser>(context.Object);
var output = getter.GetLinkedUsers(1);
var users = new List<User>();
output.ProcessDataTable((DataRow row) => students.Add(new UserStudent(row)));
Assert.AreEqual(1, users.Count);
Assert.AreEqual(2, users[0].UserId);
}
Edit 2: This is the message and debug trace from the test in question:
Test Result: Failed
Message: Assert.AreEqual failed. Expected:<1>. Actual:<0>
Debug Trace: This function can only be invoked from LINQ to Entities
From what I've read, this is because there isn't a LINQ to Entities implementation of this method that could be used in this place for the Unit Test, although there is on the live version (as it's querying an SQL server).

I know I'm late to the game, but a very simple fix is to write your own method which uses the DbFunction attribute. Then use that function instead of DbFunctions.TruncateTime.
[DbFunction("Edm", "TruncateTime")]
public static DateTime? TruncateTime(DateTime? dateValue)
{
return dateValue?.Date;
}
Using this function will execute the EDM TruncateTime method when used by Linq to Entities and will run the provided code otherwise.

Thanks for all of the help everyone, I managed to track down a solution that worked for me after reading up on shims that qujck mentioned. After adding a fake assembly of EntityFramework, I was able to fix these tests by changing them to the following:
[TestMethod]
public void CanOnlyGetCurrentLinkedUsers()
{
using (ShimsContext.Create())
{
System.Data.Entity.Fakes.ShimDbFunctions.TruncateTimeNullableOfDateTime =
(DateTime? input) =>
{
return input.HasValue ? (DateTime?)input.Value.Date : null;
};
var up = new List<par_UserPlacement>
{
this.UserPlacementFactory(1, 2, 1), // Create a user placement that is current
this.UserPlacementFactory(1, 3, 2, false) // Create a user placement that is not current
}.AsQueryable();
var set = DLTestHelper.GetMockSet<par_UserPlacement>(up);
var context = DLTestHelper.Context;
context.Setup(c => c.par_UserPlacement).Returns(set.Object);
var getter = DLTestHelper.New<LinqUserGetLinkedUsersForParentUser>(context.Object);
var output = getter.GetLinkedUsers(1);
}
var users = new List<User>();
output.ProcessDataTable((DataRow row) => users.Add(new User(row)));
Assert.AreEqual(1, users.Count);
Assert.AreEqual(2, users[0].UserId);
}

There is a way to do it. Since unit testing of business logic is generally encouraged, and since it is perfectly OK for business logic to issue LINQ queries against application data, then it must be perfectly OK to unit test those LINQ queries.
Unfortunately, DbFunctions feature of Entity Framework kills our ability to unit test code that contains LINQ queries. Moreover, it is architecturally wrong to use DbFunctions in business logic, because it couples business logic layer to a specific persistence technology (which is a separate discussion).
Having said that, our goal is the ability to run LINQ query like this:
var orderIdsByDate = (
from o in repo.Orders
group o by o.PlacedAt.Date
// here we used DateTime.Date
// and **NOT** DbFunctions.TruncateTime
into g
orderby g.Key
select new { Date = g.Key, OrderIds = g.Select(x => x.Id) });
In unit test, this will boil down to LINQ-to-Objects running against a plain array of entities arranged in advance (for example). In a real run, it must work against a real ObjectContext of Entity Framework.
Here is a recipe of achieving it - although, it requires a few steps of yours. I'm cutting down a real working example:
Step 1. Wrap ObjectSet<T> inside our own implementation of IQueryable<T> in order to provide our own intercepting wrapper of IQueryProvider.
public class EntityRepository<T> : IQueryable<T> where T : class
{
private readonly ObjectSet<T> _objectSet;
private InterceptingQueryProvider _queryProvider = null;
public EntityRepository<T>(ObjectSet<T> objectSet)
{
_objectSet = objectSet;
}
IEnumerator<T> IEnumerable<T>.GetEnumerator()
{
return _objectSet.AsEnumerable().GetEnumerator();
}
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
return _objectSet.AsEnumerable().GetEnumerator();
}
Type IQueryable.ElementType
{
get { return _objectSet.AsQueryable().ElementType; }
}
System.Linq.Expressions.Expression IQueryable.Expression
{
get { return _objectSet.AsQueryable().Expression; }
}
IQueryProvider IQueryable.Provider
{
get
{
if ( _queryProvider == null )
{
_queryProvider = new InterceptingQueryProvider(_objectSet.AsQueryable().Provider);
}
return _queryProvider;
}
}
// . . . . . you may want to include Insert(), Update(), and Delete() methods
}
Step 2. Implement the intercepting query provider, in my example it is a nested class inside EntityRepository<T>:
private class InterceptingQueryProvider : IQueryProvider
{
private readonly IQueryProvider _actualQueryProvider;
public InterceptingQueryProvider(IQueryProvider actualQueryProvider)
{
_actualQueryProvider = actualQueryProvider;
}
public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
{
var specializedExpression = QueryExpressionSpecializer.Specialize(expression);
return _actualQueryProvider.CreateQuery<TElement>(specializedExpression);
}
public IQueryable CreateQuery(Expression expression)
{
var specializedExpression = QueryExpressionSpecializer.Specialize(expression);
return _actualQueryProvider.CreateQuery(specializedExpression);
}
public TResult Execute<TResult>(Expression expression)
{
return _actualQueryProvider.Execute<TResult>(expression);
}
public object Execute(Expression expression)
{
return _actualQueryProvider.Execute(expression);
}
}
Step 3. Finally, implement a helper class named QueryExpressionSpecializer, which would replace DateTime.Date with DbFunctions.TruncateTime.
public static class QueryExpressionSpecializer
{
private static readonly MethodInfo _s_dbFunctions_TruncateTime_NullableOfDateTime =
GetMethodInfo<Expression<Func<DateTime?, DateTime?>>>(d => DbFunctions.TruncateTime(d));
private static readonly PropertyInfo _s_nullableOfDateTime_Value =
GetPropertyInfo<Expression<Func<DateTime?, DateTime>>>(d => d.Value);
public static Expression Specialize(Expression general)
{
var visitor = new SpecializingVisitor();
return visitor.Visit(general);
}
private static MethodInfo GetMethodInfo<TLambda>(TLambda lambda) where TLambda : LambdaExpression
{
return ((MethodCallExpression)lambda.Body).Method;
}
public static PropertyInfo GetPropertyInfo<TLambda>(TLambda lambda) where TLambda : LambdaExpression
{
return (PropertyInfo)((MemberExpression)lambda.Body).Member;
}
private class SpecializingVisitor : ExpressionVisitor
{
protected override Expression VisitMember(MemberExpression node)
{
if ( node.Expression.Type == typeof(DateTime?) && node.Member.Name == "Date" )
{
return Expression.Call(_s_dbFunctions_TruncateTime_NullableOfDateTime, node.Expression);
}
if ( node.Expression.Type == typeof(DateTime) && node.Member.Name == "Date" )
{
return Expression.Property(
Expression.Call(
_s_dbFunctions_TruncateTime_NullableOfDateTime,
Expression.Convert(
node.Expression,
typeof(DateTime?)
)
),
_s_nullableOfDateTime_Value
);
}
return base.VisitMember(node);
}
}
}
Of course, the above implementation of QueryExpressionSpecializer can be generalized to allow plugging in any number of additional conversions, allowing members of custom types to be used in LINQ queries, even though they are not known to Entity Framework.

Check out this answer: https://stackoverflow.com/a/14975425/1509728
To be honest, upon thinking about it I totally agree with the answer and generally follow the principle that my EF queries are tested against the database and only my application code is tested with Moq.
It looks like there is no elegant solution to using Moq for testing EF queries with your query above, while there are some hacky ideas out there. For example this one and the answer that follows it. Both seem like they could work for you.
Another approach to testing your queries would be one implemented on another project I worked on: Using VS out of box unit tests, each query (again refactored into its own method) test would be wrapped in a transaction scope. Then the project's test framework would take care of manually entering phony data into the db and the query would try to filter this phony data. At the end, the transaction is never completed so it is rolled back. Due to the nature of transaction scopes, this might not be an ideal scenario for a lot of projects. Most probably not on prod environments.
Otherwise if you must continue mocking functionality, you might want to consider other mocking frameworks.

Hmm, not sure but couldn't you do something like this?
context.Setup(s => DbFunctions.TruncateTime(It.IsAny<DateTime>()))
.Returns<DateTime?>(new Func<DateTime?,DateTime?>(
(x) => {
/* whatever modification is required here */
return x; //or return modified;
}));

since i hit the same problem recently, and opted for a simpler solution, wanted to post it here.. this solution requires no Shims, Mocking, nothing expansive etc.
Pass a 'useDbFunctions' boolean flag to your method with default value as true.
When your live code executes, your query will use DbFunctions and everything will work. Due to the default value, callers need not worry about it.
When your unit tests invoke the method to test, they can pass useDbFunctions: false.
In your method, you can make use the flag to compose your IQueryable..
if useDbFunctions is true, use the DbFunctions to add the predicate to the queryable.
if useDbFunctions is false, then skip the DbFunctions method call, and do an explicit C# equivalent solution.
This way, your unit tests will check almost 95% of your method in parity with live code. You still have the delta of "DbFunctions" vs. your equivalent code, but be diligent about it and the 95% will look like a lot of gain.
public System.Data.DataTable GetLinkedUsers(int parentUserId, bool useDbFunctions = true)
{
var today = DateTime.Now.Date;
var queryable = from up in DB.par_UserPlacement
where up.MentorId == mentorUserId;
if (useDbFunctions) // use the DbFunctions
{
queryable = queryable.Where(up =>
DbFunctions.TruncateTime(today) >= DbFunctions.TruncateTime(up.StartDate)
&& DbFunctions.TruncateTime(today) <= DbFunctions.TruncateTime(up.EndDate));
}
else
{
// do db-functions equivalent here using C# logic
// this is what the unit test path will invoke
queryable = queryable.Where(up => up.StartDate < today);
}
var query = from up in queryable
select new
{
up.UserPlacementId,
up.Users.UserId,
up.Users.FirstName,
up.Users.LastName,
up.Placements.PlacementId,
up.Placements.PlacementName,
up.StartDate,
up.EndDate,
};
query = query.OrderBy(up => up.EndDate);
return this.RunQueryToDataTable(query);
}
Unit tests will invoke the mthod as:
GetLinkedUsers(parentUserId: 10, useDbFunctions: false);
Because unit tests would have setup local DbContext entities, the C# logic/DateTime functions would work.

Use of Mocks ended sometime ago. Do not Mock, just connect to real DB. Regenerate/Seed DB on start of test.
If you still want to go ahead with mocks then create your own method as given below. IT changes behaviour runtime. When using real DB it uses DB functions, else this method. Replace DBfunctions method in code with this method
public static class CanTestDbFunctions
{
[System.Data.Entity.DbFunction("Edm", "TruncateTime")]
public static DateTime? TruncateTime(DateTime? dateValue)
{
...
}
}
This is the real function that is called. And remember, time cannot be removed from DateTime object, live with midnight or create a string equivalent.

Best practice for "var" (algorithm help)

I'm working with Simple.data, and the answer is not in the technology aforementioned but helps bring the point across. So ignore the syntax etc.
I am querying a database with a simple query; but based on a set of conditions, the query will change.
So for example: (very simplistic, probably 5-10 conditions)
var result;
if(LoggedAtSelected)
{
// Condition 1 - Calls Logged after a certain date
result = db.Jobs.FindAll(db.Jobs.Logged_At >= startDate);
}
else
{
// Condition 2 - Calls Closed after a certain date
result = db.Jobs.FindAll(db.Jobs.Closed_At >= startDate && dd.Jobs.Closed_At <= endDate);
}
foreach(var JobRecord in result)
{
}
This is the ideal code above, but sadly this is not possible given the dynamic binding and variable nature of var. What is the best practice for this kind of situation? My only idea is to write a "var result = condition..." for every condition, and in the if..else if..else, to assign it to a global variable after converting it to that type; and then using it in the "foreach". Sounds a lot of work. Any ideas? Or is that it!!!?!!!

Instead of:
var result;
Use the actual type returned by db.Jobs.FindAll:
IEnumerable<Job> result;

You can only use var if the compiler can know exactly which type to use (or how to define a new type for you).
In your case you can either define it with the type say
List<Job> result;
or call the constructor to return an instance:
var result = new List<Job>;
(of course your query will return an IEnumarable instance instead of a List, I just used List as an example because you can't instantiate an enumeration.)

Just as a note, as your if statements determine the filters for the query rather than the query itself, you might want to build up a SimpleExpression there and run the query afterwards. For example.
var whereCLause;
if(LoggedAtSelected)
{
// Condition 1 - Calls Logged after a certain date
whereClause = db.Jobs.Logged_At >= startDate;
}
else
{
// Condition 2 - Calls Closed after a certain date
whereClause = db.Jobs.Closed_At >= startDate && dd.Jobs.Closed_At <= endDate;
}
List<Job> results = db.Jobs.All.Where(whereClause);
foreach(Job record in results)
{
...
}

What is the correct way to get a sortable string from a DateTime in a LINQ entity?

// EmployeeService:
[WebMethod]
public List<Employee> GetEmployees()
{
return
(
from p in db.Persons
where p.Type == 'E'
select new Employee
{
Name = p.FullName,
//HireDate = p.CreationDate.ToString(),
// works, but not in the format I need
//HireDate = p.CreationDate.ToString("s"),
// throws a NotSupportedException
//HireDate = Wrapper(p.CreationDate),
// works, but makes me worry about performance
// and feel dead inside
}
).ToList<Employee>();
}
private String Wrapper(DateTime date)
{
return date.ToString("s");
}
// Elsewhere:
public class Employee
{
public String Name;
public String HireDate;
}
I'm using a JavaScript framework that needs dates in ISO 8601 style format, which is exactly what calling .ToString("s") on a DateTime object will return.
Is there a cleaner/more efficient way to do this in LINQ to SQL?

I believe the Wrapper trick is as good as you can get in this situation. Sorry.
Update: Seems this has been asked again here: Linq to Sql - DateTime Format - YYYY-MMM (2009-Mar). The answer was pretty much "sorry" there too; considering who participated in that question, I 'm now really sure that you can't do better.

The problem is that, when using IQueryable, the provider tries to translate all of the LINQ expressions into something it can send down to the database. It has no way of knowing what to do with ToString("s"), so the NotSupported exception is thrown.
If you were to add .AsEnumerable() before the Select call, then it should work. The difference is that the Person object will be brought into memory completely, then the projection (the Select) method will be ran and all of that is done as a .NET compiled method, not as SQL. So essentially anything after AsEnumerable() will be done in memory and not in the database, so it's generally not recommended to do until you've trimmed down the number of rows as much as possible (i.e. after all Where and OrderBys).

How do you declare a by-reference parameter variable for use in a Linq query?

I have a class MyClass with a method:
public bool MyMethod(out DateTime? MyDate) {
...
}
I'd like to call this method in the following way:
var q = from mc in MyClasses.AsEnumerable()
from output in mc.MyMethod(out dt) // how to declare dt?
select new { mc, output, dt };
Obviously this doesn't compile, coz I haven't declared dt. I can declare it outside the query, but that doesn't give me a warm fuzzy feeling: (a) I don't like declaring variables at a level greater than the necessary scope, and (b) it isn't immediately and intuitively obvious that the value would be calculated correctly for each row inside the query.
Is there some syntax that will allow me to declare DateTime? dt inside the query?

Its not just the scope of dt that is the problem here, your MyMethod() would need to return an enumeration for the SelectMany to work (the second from clause)
Not really sure what you are trying to achieve here but I would be inclined to have your function return a struct or class that contains the bool and the date and assign it via a let statement...something like...
var qry = from mc in MyClasses.FunctThatReturnsAnIEnumerable()
let result = mc.MyMethod() //this returns the struct
select new {mc, result.BoolVal, result.DateVal};

My first thought is "ew" (thats my code smell noise). I feel you're really trying to force using that method where it doesn't belong.
My next thought would be to simply create the behavior you want, maybe something such as (although I still don't recommend this as a good solution):
public DateTime? GetMyDate()
{
DateTime? dt;
MyMethod(out dt);
return dt;
}
Then...
var q = from mc in MyClasses.AsEnumerable()
select new { mc, output, Date = GetMyDate() };
I'm making the assumption that the bool returned from MyDate isn't important in this context, which really breaks intent. I only assume that because, given your desired result, you don't seem to care about the bool return either.
In either case, I would recommend rethinking this instead of making due with what you have.