Related
Suppose I have built up, through some conditional logic over many steps, an IQueryable<T> instance we'll call query.
I want to get a count of total records and a page of data, so I want to call query.CountAsync() and query.Skip(0).Take(10).ToListAsync(). I cannot call these in succession, because a race condition occurs where they both try to run a query on the same DbContext at the same time. This is not allowed:
"A second operation started on this context before a previous asynchronous operation completed. Use 'await' to ensure that any asynchronous operations have completed before calling another method on this context. Any instance members are not guaranteed to be thread safe."
I do not want to 'await' the first before even starting the second. I want to fire off both queries as soon as possible. The only way to do this is to run them from separate DbContexts. It seems ridiculous that I might have to build the entire query (or 2, or 3) side-by-side starting with different instances of DbSet. Is there any way to clone or alter an IQueryable<T> (not necessarily that interface, but it's underlying implementation) such that I can have one copy that runs on DbContext "A", and another that will run on DbContext "B", so that both queries can be executing simultaneously? I'm just trying to avoid recomposing the query X times from scratch just to run it on X contexts.
There is no standard way of doing that. The problem is that EF6 query expression trees contain constant nodes holding ObjectQuery instances which are bound to the DbContext (actually the underlying ObjectContext) used when creating the query. Also there is a runtime check before executing the query if there are such expressions bound to a different context than the one executing the query.
The only idea that comes in my mind is to process the query expression tree with ExpressionVisitor and replace these ObjectQuery instances with new ones bound to the new context.
Here is a possible implementation of the aforementioned idea:
using System.Data.Entity.Core.Objects;
using System.Data.Entity.Infrastructure;
using System.Linq;
using System.Linq.Expressions;
namespace System.Data.Entity
{
public static class DbQueryExtensions
{
public static IQueryable<T> BindTo<T>(this IQueryable<T> source, DbContext target)
{
var binder = new DbContextBinder(target);
var expression = binder.Visit(source.Expression);
var provider = binder.TargetProvider;
return provider != null ? provider.CreateQuery<T>(expression) : source;
}
class DbContextBinder : ExpressionVisitor
{
ObjectContext targetObjectContext;
public IQueryProvider TargetProvider { get; private set; }
public DbContextBinder(DbContext target)
{
targetObjectContext = ((IObjectContextAdapter)target).ObjectContext;
}
protected override Expression VisitConstant(ConstantExpression node)
{
if (node.Value is ObjectQuery objectQuery && objectQuery.Context != targetObjectContext)
return Expression.Constant(CreateObjectQuery((dynamic)objectQuery));
return base.VisitConstant(node);
}
ObjectQuery<T> CreateObjectQuery<T>(ObjectQuery<T> source)
{
var parameters = source.Parameters
.Select(p => new ObjectParameter(p.Name, p.ParameterType) { Value = p.Value })
.ToArray();
var query = targetObjectContext.CreateQuery<T>(source.CommandText, parameters);
query.MergeOption = source.MergeOption;
query.Streaming = source.Streaming;
query.EnablePlanCaching = source.EnablePlanCaching;
if (TargetProvider == null)
TargetProvider = ((IQueryable)query).Provider;
return query;
}
}
}
}
One difference with the standard EF6 LINQ queries is that this produces ObjectQuery<T> rather than DbQuery<T>, although except that ToString() does not return the generated SQL, I haven't noticed any difference in the further query building / execution. It seems to work, but use it with care and on your own risk.
You could write a function to build up your query, taking DbContext as a parameter.
public IQueryable<T> MyQuery(DbContext<T> db)
{
return db.Table
.Where(p => p.reallycomplex)
....
...
.OrderBy(p => p.manythings);
}
I've done this many times and it works well.
Now it's easy to make queries with two different contexts:
IQueryable<T> q1 = MyQuery(dbContext1);
IQueryable<T> q2 = MyQuery(dbContext2);
If your concern was the execution time taken to build the IQueryable objects, then my only suggestion is don't worry about it.
So you have an IQueryable<T> that will be performed on DbContext A as soon as the query is executed and you want the same query to run on DbContext B when the query is executed.
For this you'll have to understand the difference between an IEnumerable<T> and an IQueryable<T>.
An IEnumerable<T> holds all code to enumerate over the elements that the enumerable represents. The enumeration starts when GetEnumerator and MoveNext are called. This can be done explicitly. However it is usually done implicitly by functions like foreach, ToList, FirstOrDefault, etc.
An IQueryable does not hold the code to enumerate, it holds an Expression and a Provider. The Provider knows who will execute the query, and it knows how to translate the Expression into the language that is understood by the query executioner.
Due to this separation, it is possible to let the same Expression be executed by different data sources. They don't even have to be of the same type: one data source can be a database management system that understands SQL, the other one could be a comma separated file.
As long as you concatenate Linq statements that return an IQueryable, the query is not executed, only the Expression is changed.
As soon as enumeration starts, either by calling GetEnumerator / MoveNext, or by using foreach or one of the LINQ functions that do not return an IQueryable, the Provider will translate the Expression into the language the the data source understands and communicates with the data source to execute the query. The result of the query is an IEnumerable, which can be enumerated as if all data was in local code.
Some Providers are smart and use some buffering, so that not all data is transferred to local memory, but only part of the data. New data is queried when needed. So if you do a foreach in a database with a zillion elements, only the first few (thousands) elements are queried. More data is queried if your foreach runs out of fetched data.
So you already have one IQueryable<T>, therefore you have an Expression a Provider and an ElementType. You want the same Expression / ElementType to be executed by a differentProvider. You even want to change theExpression` slightly before you execute it.
Therefore you need to be able to create an object that implements IQueryable<T> and you want to be able to set the Expression, ElementType and a Provider
class MyQueryable<T> : IQueryable<T>
{
public type ElementType {get; set;}
public Expression Expression {get; set;}
public Provider Provider {get; set;}
}
IQueryable<T> queryOnDbContextA= dbCotextA ...
IQueryable<T> setInDbContextB = dbContextB.Set<T>();
IQueryable<T> queryOnDbContextB = new MyQueryable<T>()
{
ElementType = queryOnDbContextA.ElementType,
Expression = queryOnDbContextB.Expression,
Provider = setInDbContextB.Provider,
}
If desired you can adjust the query on the other context before executing it:
var getPageOnContextB = queryOnDbContextB
.Skip(...)
.Take(...);
Both queries are still not executed yet. Execute them:
var countA = await queryOnContextA.CountAsync();
var fetchedPageContextB = await getPageOnContextB.ToListAsync();
In my C# code, I have 2 WHERE queries both of which I can call on an IQueryable and have the whole thing compiled down to SQL, and both of which have a great deal of common logic.
I believe this is NOT a duplication of this similar question:
Using Function in Select Clause of Entity Framework Query because in my scenario the function in question CAN be converted into SQL - EF just isn't realising that it can do so.
The queries are approximately:
public static IQueryable<Template> WhereIsOwnedByUser(this IQueryable<Template> set, User user)
{
return set.Where(temp =>
temp.Requests
.Where(req => req.WasSent)
.OrderByDescending(req => req.DueDate)
.Take(2)
.SelectMany(req => req.RequestRecipients.Select(reqRecip => reqRecip.Recipient.Id))
.Contains(user.Id));
}
AND
public static IQueryable<Template> WhereIsOwnedByUser(this IQueryable<DataReturn> set, User user)
{
return set.Where(ret=>
ret.Entity.Id == user.Entity.Id
&&
ret.Request.Template.Requests
.Where(req => req.WasSent)
.OrderByDescending(req => req.DueDate)
.Take(2)
.SelectMany(req => req.RequestRecipients.Select(reqRecip => reqRecip.Recipient.Id))
.Contains(user.Id));
}
So a basic BusinessLogic rule for "owns a template" and then a corollary of that for "owns DataReturn if company matches AND owns template"
As you can see, thinking only about the C#, these could easily be refactored as:
private static bool UserOwnsTemplate(User user, Template temp)
{
return temp.Requests
.Where(req => req.WasSent)
.OrderByDescending(req => req.DueDate)
.Take(2)
.SelectMany(req => req.RequestRecipients.Select(reqRecip => reqRecip.Recipient.Id))
.Contains(user.Id);
}
public static IQueryable<Template> WhereIsOwnedByUser(this IQueryable<Template> set, User user)
{
return set.Where(temp => UserOwnsTemplate(user, temp));
}
public static IQueryable<DataReturn> WhereIsOwnedByUser(this IQueryable<DataReturn> set, User user)
{
return set.Where(
ret =>
ret.Entity.Id == user.Entity.Id
&&
UserOwnsTemplate(user, ret.Request.Template)
);
}
Thus reducing the duplication (Yay!)
But then EF will complain that it doesn't know what to do with UserOwnsTemplate, despite the fact that it can handle the logic in SQL perfectly well.
AFAICT there is no nice way to solve this. I think my options are:
Turn UserOwnsTemplate into a UDF, a SQL function defined in the database.
But I can't create a UDF from a C# lamda, I have to define the SQL, which would be more hassle.
Assign the Expression<Func<Template,bool>> that UserOwnsTemplate defines as a variable, and then build relevant Expression<Func<DataReturn ,bool>> for the DataReturn version by hand using Expression.AndAlso to glue the two "clauses" together.
Meta-programming. Ughhh. I've done this before in another project and it was vile to do, and a nightmare to maintain.
Live with the duplication.
Likely what will happen unless SO can advise otherwise. ;)
Can anyone see any other options available?
Can I do anything to force EF into parsing the function into SQL? (the phrase "inling" comes to mind but I don't 100% know what I think I mean by that?)
Can anyone see a way to convert ret.Request.Template into an IQueryable so that I can just call the other WhereIsOwnedBy extension method on it?
Any other suggestions AT ALL?
You can keep your syntax and make it work but you will need to call an additional method on the outer IQueryable<>.
The trick is to manually replace the IQueryable<>.Expression with a copy in which you replace the function call by the corresponding Expression>.
So the idea is to do something like that:
public static class MyLinqExtensions
{
public static IQueryable<T> InlineFunctions<T>(this IQueryable<T> queryable)
{
var expression = TransformExpression(queryable.Expression);
return (IQueryable<T>)queryable.Provider.CreateQuery(expression);
}
private static Expression TransformExpression(System.Linq.Expressions.Expression expression)
{
var visitor = new InlineFunctionsExpressionVisitor();
return visitor.Visit(expression);
}
private class InlineFunctionsExpressionVisitor : System.Linq.Expressions.ExpressionVisitor
{
protected override System.Linq.Expressions.Expression VisitMethodCall(System.Linq.Expressions.MethodCallExpression methodCallExpression)
{
if (methodCallExpression.Method.IsStatic
&& methodCallExpression.Method.DeclaringType == typeof(MyDeclaringType)
&& methodCallExpression.Method.Name == "WhereIsOwnedByUser")
{
var setArgumentExpression = methodCallExpression.Arguments[0];
var userArgumentExpression = methodCallExpression.Arguments[1];
var methodInfo = ... // Get typeof(IQueryable<Template>).MethodInfo
var whereConditionExpression = ...// Build where condition and use userArgumentExpression
return Expression.MethodCallExpression(methodInfo, setArgumentExpression, whereConditionExpression);
}
return base.VisitMethodCall(methodCallExpression);
// Some ideas to make this more flexible:
// 1. Use an attribute to mark the functions that can be inlined [InlinableAttribute]
// 2. Define an Expression<Func<>> first to be able to get the Expression and substritute the function call with it:
// Expression<Func<IQueryable<Template>, User, IQueryable<Template>>> _whereIsOwnedByUser = (set, user) =>
// {
// return set.Where(temp => UserOwnsTemplate(user, temp));
// };
//
// public static IQueryable<Template> WhereIsOwnedByUser(this IQueryable<Template> set, User user)
// {
// // You should cache the compiled expression
// return _whereIsOwnedByUser.Compile().Invoke(set, user);
// }
//
}
}
}
And then you can do this:
public static IQueryable<DataReturn> WhereIsOwnedByUser(this IQueryable<DataReturn> set, User user)
{
return set.Where(
ret =>
ret.Entity.Id == user.Entity.Id
&&
UserOwnsTemplate(user, ret.Request.Template)
)
.InlineFunctions();
}
The problem is that your method becomes part of the expression tree and that EF cannot evaluate it. In principle, it is possible to evaluate parts of the expression tree before triggering the query. Have a look at Re-Linq: https://relinq.codeplex.com/ It has a class PartialEvaluatingExpressionTreeVisitor which can evaluate all partial expression trees, i.e. it will find your method, evaluate it, and inject the actual expression tree. This will come at a certain performance cost, but it may not be significant and you will have to gauge clean design versus performance.
I'm currently attempting to run some unit tests on a query that is running through the Entity Framework. The query itself runs without any issues on the live version, but the unit tests are always failing.
I've narrowed this down to my usage of DbFunctions.TruncateTime, but I don't know of a way around this to get the unit tests to reflect what is happening on the live server.
Here is the method that I am using:
public System.Data.DataTable GetLinkedUsers(int parentUserId)
{
var today = DateTime.Now.Date;
var query = from up in DB.par_UserPlacement
where up.MentorId == mentorUserId
&& DbFunctions.TruncateTime(today) >= DbFunctions.TruncateTime(up.StartDate)
&& DbFunctions.TruncateTime(today) <= DbFunctions.TruncateTime(up.EndDate)
select new
{
up.UserPlacementId,
up.Users.UserId,
up.Users.FirstName,
up.Users.LastName,
up.Placements.PlacementId,
up.Placements.PlacementName,
up.StartDate,
up.EndDate,
};
query = query.OrderBy(up => up.EndDate);
return this.RunQueryToDataTable(query);
}
If I comment out the lines with DbFunctions in, the tests all pass (except for the ones that are checking that only valid results for a given date are run).
Is there a way I can provide a mocked version of DbFunctions.TruncateTime to use in these tests? Essentially it should just be returning Datetime.Date, but that isn't available in EF queries.
Edit: Here's the test that's failing that uses the date check:
[TestMethod]
public void CanOnlyGetCurrentLinkedUsers()
{
var up = new List<par_UserPlacement>
{
this.UserPlacementFactory(1, 2, 1), // Create a user placement that is current
this.UserPlacementFactory(1, 3, 2, false) // Create a user placement that is not current
}.AsQueryable();
var set = DLTestHelper.GetMockSet<par_UserPlacement>(up);
var context = DLTestHelper.Context;
context.Setup(c => c.par_UserPlacement).Returns(set.Object);
var getter = DLTestHelper.New<LinqUserGetLinkedUsersForParentUser>(context.Object);
var output = getter.GetLinkedUsers(1);
var users = new List<User>();
output.ProcessDataTable((DataRow row) => students.Add(new UserStudent(row)));
Assert.AreEqual(1, users.Count);
Assert.AreEqual(2, users[0].UserId);
}
Edit 2: This is the message and debug trace from the test in question:
Test Result: Failed
Message: Assert.AreEqual failed. Expected:<1>. Actual:<0>
Debug Trace: This function can only be invoked from LINQ to Entities
From what I've read, this is because there isn't a LINQ to Entities implementation of this method that could be used in this place for the Unit Test, although there is on the live version (as it's querying an SQL server).
I know I'm late to the game, but a very simple fix is to write your own method which uses the DbFunction attribute. Then use that function instead of DbFunctions.TruncateTime.
[DbFunction("Edm", "TruncateTime")]
public static DateTime? TruncateTime(DateTime? dateValue)
{
return dateValue?.Date;
}
Using this function will execute the EDM TruncateTime method when used by Linq to Entities and will run the provided code otherwise.
Thanks for all of the help everyone, I managed to track down a solution that worked for me after reading up on shims that qujck mentioned. After adding a fake assembly of EntityFramework, I was able to fix these tests by changing them to the following:
[TestMethod]
public void CanOnlyGetCurrentLinkedUsers()
{
using (ShimsContext.Create())
{
System.Data.Entity.Fakes.ShimDbFunctions.TruncateTimeNullableOfDateTime =
(DateTime? input) =>
{
return input.HasValue ? (DateTime?)input.Value.Date : null;
};
var up = new List<par_UserPlacement>
{
this.UserPlacementFactory(1, 2, 1), // Create a user placement that is current
this.UserPlacementFactory(1, 3, 2, false) // Create a user placement that is not current
}.AsQueryable();
var set = DLTestHelper.GetMockSet<par_UserPlacement>(up);
var context = DLTestHelper.Context;
context.Setup(c => c.par_UserPlacement).Returns(set.Object);
var getter = DLTestHelper.New<LinqUserGetLinkedUsersForParentUser>(context.Object);
var output = getter.GetLinkedUsers(1);
}
var users = new List<User>();
output.ProcessDataTable((DataRow row) => users.Add(new User(row)));
Assert.AreEqual(1, users.Count);
Assert.AreEqual(2, users[0].UserId);
}
There is a way to do it. Since unit testing of business logic is generally encouraged, and since it is perfectly OK for business logic to issue LINQ queries against application data, then it must be perfectly OK to unit test those LINQ queries.
Unfortunately, DbFunctions feature of Entity Framework kills our ability to unit test code that contains LINQ queries. Moreover, it is architecturally wrong to use DbFunctions in business logic, because it couples business logic layer to a specific persistence technology (which is a separate discussion).
Having said that, our goal is the ability to run LINQ query like this:
var orderIdsByDate = (
from o in repo.Orders
group o by o.PlacedAt.Date
// here we used DateTime.Date
// and **NOT** DbFunctions.TruncateTime
into g
orderby g.Key
select new { Date = g.Key, OrderIds = g.Select(x => x.Id) });
In unit test, this will boil down to LINQ-to-Objects running against a plain array of entities arranged in advance (for example). In a real run, it must work against a real ObjectContext of Entity Framework.
Here is a recipe of achieving it - although, it requires a few steps of yours. I'm cutting down a real working example:
Step 1. Wrap ObjectSet<T> inside our own implementation of IQueryable<T> in order to provide our own intercepting wrapper of IQueryProvider.
public class EntityRepository<T> : IQueryable<T> where T : class
{
private readonly ObjectSet<T> _objectSet;
private InterceptingQueryProvider _queryProvider = null;
public EntityRepository<T>(ObjectSet<T> objectSet)
{
_objectSet = objectSet;
}
IEnumerator<T> IEnumerable<T>.GetEnumerator()
{
return _objectSet.AsEnumerable().GetEnumerator();
}
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
return _objectSet.AsEnumerable().GetEnumerator();
}
Type IQueryable.ElementType
{
get { return _objectSet.AsQueryable().ElementType; }
}
System.Linq.Expressions.Expression IQueryable.Expression
{
get { return _objectSet.AsQueryable().Expression; }
}
IQueryProvider IQueryable.Provider
{
get
{
if ( _queryProvider == null )
{
_queryProvider = new InterceptingQueryProvider(_objectSet.AsQueryable().Provider);
}
return _queryProvider;
}
}
// . . . . . you may want to include Insert(), Update(), and Delete() methods
}
Step 2. Implement the intercepting query provider, in my example it is a nested class inside EntityRepository<T>:
private class InterceptingQueryProvider : IQueryProvider
{
private readonly IQueryProvider _actualQueryProvider;
public InterceptingQueryProvider(IQueryProvider actualQueryProvider)
{
_actualQueryProvider = actualQueryProvider;
}
public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
{
var specializedExpression = QueryExpressionSpecializer.Specialize(expression);
return _actualQueryProvider.CreateQuery<TElement>(specializedExpression);
}
public IQueryable CreateQuery(Expression expression)
{
var specializedExpression = QueryExpressionSpecializer.Specialize(expression);
return _actualQueryProvider.CreateQuery(specializedExpression);
}
public TResult Execute<TResult>(Expression expression)
{
return _actualQueryProvider.Execute<TResult>(expression);
}
public object Execute(Expression expression)
{
return _actualQueryProvider.Execute(expression);
}
}
Step 3. Finally, implement a helper class named QueryExpressionSpecializer, which would replace DateTime.Date with DbFunctions.TruncateTime.
public static class QueryExpressionSpecializer
{
private static readonly MethodInfo _s_dbFunctions_TruncateTime_NullableOfDateTime =
GetMethodInfo<Expression<Func<DateTime?, DateTime?>>>(d => DbFunctions.TruncateTime(d));
private static readonly PropertyInfo _s_nullableOfDateTime_Value =
GetPropertyInfo<Expression<Func<DateTime?, DateTime>>>(d => d.Value);
public static Expression Specialize(Expression general)
{
var visitor = new SpecializingVisitor();
return visitor.Visit(general);
}
private static MethodInfo GetMethodInfo<TLambda>(TLambda lambda) where TLambda : LambdaExpression
{
return ((MethodCallExpression)lambda.Body).Method;
}
public static PropertyInfo GetPropertyInfo<TLambda>(TLambda lambda) where TLambda : LambdaExpression
{
return (PropertyInfo)((MemberExpression)lambda.Body).Member;
}
private class SpecializingVisitor : ExpressionVisitor
{
protected override Expression VisitMember(MemberExpression node)
{
if ( node.Expression.Type == typeof(DateTime?) && node.Member.Name == "Date" )
{
return Expression.Call(_s_dbFunctions_TruncateTime_NullableOfDateTime, node.Expression);
}
if ( node.Expression.Type == typeof(DateTime) && node.Member.Name == "Date" )
{
return Expression.Property(
Expression.Call(
_s_dbFunctions_TruncateTime_NullableOfDateTime,
Expression.Convert(
node.Expression,
typeof(DateTime?)
)
),
_s_nullableOfDateTime_Value
);
}
return base.VisitMember(node);
}
}
}
Of course, the above implementation of QueryExpressionSpecializer can be generalized to allow plugging in any number of additional conversions, allowing members of custom types to be used in LINQ queries, even though they are not known to Entity Framework.
Check out this answer: https://stackoverflow.com/a/14975425/1509728
To be honest, upon thinking about it I totally agree with the answer and generally follow the principle that my EF queries are tested against the database and only my application code is tested with Moq.
It looks like there is no elegant solution to using Moq for testing EF queries with your query above, while there are some hacky ideas out there. For example this one and the answer that follows it. Both seem like they could work for you.
Another approach to testing your queries would be one implemented on another project I worked on: Using VS out of box unit tests, each query (again refactored into its own method) test would be wrapped in a transaction scope. Then the project's test framework would take care of manually entering phony data into the db and the query would try to filter this phony data. At the end, the transaction is never completed so it is rolled back. Due to the nature of transaction scopes, this might not be an ideal scenario for a lot of projects. Most probably not on prod environments.
Otherwise if you must continue mocking functionality, you might want to consider other mocking frameworks.
Hmm, not sure but couldn't you do something like this?
context.Setup(s => DbFunctions.TruncateTime(It.IsAny<DateTime>()))
.Returns<DateTime?>(new Func<DateTime?,DateTime?>(
(x) => {
/* whatever modification is required here */
return x; //or return modified;
}));
since i hit the same problem recently, and opted for a simpler solution, wanted to post it here.. this solution requires no Shims, Mocking, nothing expansive etc.
Pass a 'useDbFunctions' boolean flag to your method with default value as true.
When your live code executes, your query will use DbFunctions and everything will work. Due to the default value, callers need not worry about it.
When your unit tests invoke the method to test, they can pass useDbFunctions: false.
In your method, you can make use the flag to compose your IQueryable..
if useDbFunctions is true, use the DbFunctions to add the predicate to the queryable.
if useDbFunctions is false, then skip the DbFunctions method call, and do an explicit C# equivalent solution.
This way, your unit tests will check almost 95% of your method in parity with live code. You still have the delta of "DbFunctions" vs. your equivalent code, but be diligent about it and the 95% will look like a lot of gain.
public System.Data.DataTable GetLinkedUsers(int parentUserId, bool useDbFunctions = true)
{
var today = DateTime.Now.Date;
var queryable = from up in DB.par_UserPlacement
where up.MentorId == mentorUserId;
if (useDbFunctions) // use the DbFunctions
{
queryable = queryable.Where(up =>
DbFunctions.TruncateTime(today) >= DbFunctions.TruncateTime(up.StartDate)
&& DbFunctions.TruncateTime(today) <= DbFunctions.TruncateTime(up.EndDate));
}
else
{
// do db-functions equivalent here using C# logic
// this is what the unit test path will invoke
queryable = queryable.Where(up => up.StartDate < today);
}
var query = from up in queryable
select new
{
up.UserPlacementId,
up.Users.UserId,
up.Users.FirstName,
up.Users.LastName,
up.Placements.PlacementId,
up.Placements.PlacementName,
up.StartDate,
up.EndDate,
};
query = query.OrderBy(up => up.EndDate);
return this.RunQueryToDataTable(query);
}
Unit tests will invoke the mthod as:
GetLinkedUsers(parentUserId: 10, useDbFunctions: false);
Because unit tests would have setup local DbContext entities, the C# logic/DateTime functions would work.
Use of Mocks ended sometime ago. Do not Mock, just connect to real DB. Regenerate/Seed DB on start of test.
If you still want to go ahead with mocks then create your own method as given below. IT changes behaviour runtime. When using real DB it uses DB functions, else this method. Replace DBfunctions method in code with this method
public static class CanTestDbFunctions
{
[System.Data.Entity.DbFunction("Edm", "TruncateTime")]
public static DateTime? TruncateTime(DateTime? dateValue)
{
...
}
}
This is the real function that is called. And remember, time cannot be removed from DateTime object, live with midnight or create a string equivalent.
I'm currently attempting to run some unit tests on a query that is running through the Entity Framework. The query itself runs without any issues on the live version, but the unit tests are always failing.
I've narrowed this down to my usage of DbFunctions.TruncateTime, but I don't know of a way around this to get the unit tests to reflect what is happening on the live server.
Here is the method that I am using:
public System.Data.DataTable GetLinkedUsers(int parentUserId)
{
var today = DateTime.Now.Date;
var query = from up in DB.par_UserPlacement
where up.MentorId == mentorUserId
&& DbFunctions.TruncateTime(today) >= DbFunctions.TruncateTime(up.StartDate)
&& DbFunctions.TruncateTime(today) <= DbFunctions.TruncateTime(up.EndDate)
select new
{
up.UserPlacementId,
up.Users.UserId,
up.Users.FirstName,
up.Users.LastName,
up.Placements.PlacementId,
up.Placements.PlacementName,
up.StartDate,
up.EndDate,
};
query = query.OrderBy(up => up.EndDate);
return this.RunQueryToDataTable(query);
}
If I comment out the lines with DbFunctions in, the tests all pass (except for the ones that are checking that only valid results for a given date are run).
Is there a way I can provide a mocked version of DbFunctions.TruncateTime to use in these tests? Essentially it should just be returning Datetime.Date, but that isn't available in EF queries.
Edit: Here's the test that's failing that uses the date check:
[TestMethod]
public void CanOnlyGetCurrentLinkedUsers()
{
var up = new List<par_UserPlacement>
{
this.UserPlacementFactory(1, 2, 1), // Create a user placement that is current
this.UserPlacementFactory(1, 3, 2, false) // Create a user placement that is not current
}.AsQueryable();
var set = DLTestHelper.GetMockSet<par_UserPlacement>(up);
var context = DLTestHelper.Context;
context.Setup(c => c.par_UserPlacement).Returns(set.Object);
var getter = DLTestHelper.New<LinqUserGetLinkedUsersForParentUser>(context.Object);
var output = getter.GetLinkedUsers(1);
var users = new List<User>();
output.ProcessDataTable((DataRow row) => students.Add(new UserStudent(row)));
Assert.AreEqual(1, users.Count);
Assert.AreEqual(2, users[0].UserId);
}
Edit 2: This is the message and debug trace from the test in question:
Test Result: Failed
Message: Assert.AreEqual failed. Expected:<1>. Actual:<0>
Debug Trace: This function can only be invoked from LINQ to Entities
From what I've read, this is because there isn't a LINQ to Entities implementation of this method that could be used in this place for the Unit Test, although there is on the live version (as it's querying an SQL server).
I know I'm late to the game, but a very simple fix is to write your own method which uses the DbFunction attribute. Then use that function instead of DbFunctions.TruncateTime.
[DbFunction("Edm", "TruncateTime")]
public static DateTime? TruncateTime(DateTime? dateValue)
{
return dateValue?.Date;
}
Using this function will execute the EDM TruncateTime method when used by Linq to Entities and will run the provided code otherwise.
Thanks for all of the help everyone, I managed to track down a solution that worked for me after reading up on shims that qujck mentioned. After adding a fake assembly of EntityFramework, I was able to fix these tests by changing them to the following:
[TestMethod]
public void CanOnlyGetCurrentLinkedUsers()
{
using (ShimsContext.Create())
{
System.Data.Entity.Fakes.ShimDbFunctions.TruncateTimeNullableOfDateTime =
(DateTime? input) =>
{
return input.HasValue ? (DateTime?)input.Value.Date : null;
};
var up = new List<par_UserPlacement>
{
this.UserPlacementFactory(1, 2, 1), // Create a user placement that is current
this.UserPlacementFactory(1, 3, 2, false) // Create a user placement that is not current
}.AsQueryable();
var set = DLTestHelper.GetMockSet<par_UserPlacement>(up);
var context = DLTestHelper.Context;
context.Setup(c => c.par_UserPlacement).Returns(set.Object);
var getter = DLTestHelper.New<LinqUserGetLinkedUsersForParentUser>(context.Object);
var output = getter.GetLinkedUsers(1);
}
var users = new List<User>();
output.ProcessDataTable((DataRow row) => users.Add(new User(row)));
Assert.AreEqual(1, users.Count);
Assert.AreEqual(2, users[0].UserId);
}
There is a way to do it. Since unit testing of business logic is generally encouraged, and since it is perfectly OK for business logic to issue LINQ queries against application data, then it must be perfectly OK to unit test those LINQ queries.
Unfortunately, DbFunctions feature of Entity Framework kills our ability to unit test code that contains LINQ queries. Moreover, it is architecturally wrong to use DbFunctions in business logic, because it couples business logic layer to a specific persistence technology (which is a separate discussion).
Having said that, our goal is the ability to run LINQ query like this:
var orderIdsByDate = (
from o in repo.Orders
group o by o.PlacedAt.Date
// here we used DateTime.Date
// and **NOT** DbFunctions.TruncateTime
into g
orderby g.Key
select new { Date = g.Key, OrderIds = g.Select(x => x.Id) });
In unit test, this will boil down to LINQ-to-Objects running against a plain array of entities arranged in advance (for example). In a real run, it must work against a real ObjectContext of Entity Framework.
Here is a recipe of achieving it - although, it requires a few steps of yours. I'm cutting down a real working example:
Step 1. Wrap ObjectSet<T> inside our own implementation of IQueryable<T> in order to provide our own intercepting wrapper of IQueryProvider.
public class EntityRepository<T> : IQueryable<T> where T : class
{
private readonly ObjectSet<T> _objectSet;
private InterceptingQueryProvider _queryProvider = null;
public EntityRepository<T>(ObjectSet<T> objectSet)
{
_objectSet = objectSet;
}
IEnumerator<T> IEnumerable<T>.GetEnumerator()
{
return _objectSet.AsEnumerable().GetEnumerator();
}
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
return _objectSet.AsEnumerable().GetEnumerator();
}
Type IQueryable.ElementType
{
get { return _objectSet.AsQueryable().ElementType; }
}
System.Linq.Expressions.Expression IQueryable.Expression
{
get { return _objectSet.AsQueryable().Expression; }
}
IQueryProvider IQueryable.Provider
{
get
{
if ( _queryProvider == null )
{
_queryProvider = new InterceptingQueryProvider(_objectSet.AsQueryable().Provider);
}
return _queryProvider;
}
}
// . . . . . you may want to include Insert(), Update(), and Delete() methods
}
Step 2. Implement the intercepting query provider, in my example it is a nested class inside EntityRepository<T>:
private class InterceptingQueryProvider : IQueryProvider
{
private readonly IQueryProvider _actualQueryProvider;
public InterceptingQueryProvider(IQueryProvider actualQueryProvider)
{
_actualQueryProvider = actualQueryProvider;
}
public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
{
var specializedExpression = QueryExpressionSpecializer.Specialize(expression);
return _actualQueryProvider.CreateQuery<TElement>(specializedExpression);
}
public IQueryable CreateQuery(Expression expression)
{
var specializedExpression = QueryExpressionSpecializer.Specialize(expression);
return _actualQueryProvider.CreateQuery(specializedExpression);
}
public TResult Execute<TResult>(Expression expression)
{
return _actualQueryProvider.Execute<TResult>(expression);
}
public object Execute(Expression expression)
{
return _actualQueryProvider.Execute(expression);
}
}
Step 3. Finally, implement a helper class named QueryExpressionSpecializer, which would replace DateTime.Date with DbFunctions.TruncateTime.
public static class QueryExpressionSpecializer
{
private static readonly MethodInfo _s_dbFunctions_TruncateTime_NullableOfDateTime =
GetMethodInfo<Expression<Func<DateTime?, DateTime?>>>(d => DbFunctions.TruncateTime(d));
private static readonly PropertyInfo _s_nullableOfDateTime_Value =
GetPropertyInfo<Expression<Func<DateTime?, DateTime>>>(d => d.Value);
public static Expression Specialize(Expression general)
{
var visitor = new SpecializingVisitor();
return visitor.Visit(general);
}
private static MethodInfo GetMethodInfo<TLambda>(TLambda lambda) where TLambda : LambdaExpression
{
return ((MethodCallExpression)lambda.Body).Method;
}
public static PropertyInfo GetPropertyInfo<TLambda>(TLambda lambda) where TLambda : LambdaExpression
{
return (PropertyInfo)((MemberExpression)lambda.Body).Member;
}
private class SpecializingVisitor : ExpressionVisitor
{
protected override Expression VisitMember(MemberExpression node)
{
if ( node.Expression.Type == typeof(DateTime?) && node.Member.Name == "Date" )
{
return Expression.Call(_s_dbFunctions_TruncateTime_NullableOfDateTime, node.Expression);
}
if ( node.Expression.Type == typeof(DateTime) && node.Member.Name == "Date" )
{
return Expression.Property(
Expression.Call(
_s_dbFunctions_TruncateTime_NullableOfDateTime,
Expression.Convert(
node.Expression,
typeof(DateTime?)
)
),
_s_nullableOfDateTime_Value
);
}
return base.VisitMember(node);
}
}
}
Of course, the above implementation of QueryExpressionSpecializer can be generalized to allow plugging in any number of additional conversions, allowing members of custom types to be used in LINQ queries, even though they are not known to Entity Framework.
Check out this answer: https://stackoverflow.com/a/14975425/1509728
To be honest, upon thinking about it I totally agree with the answer and generally follow the principle that my EF queries are tested against the database and only my application code is tested with Moq.
It looks like there is no elegant solution to using Moq for testing EF queries with your query above, while there are some hacky ideas out there. For example this one and the answer that follows it. Both seem like they could work for you.
Another approach to testing your queries would be one implemented on another project I worked on: Using VS out of box unit tests, each query (again refactored into its own method) test would be wrapped in a transaction scope. Then the project's test framework would take care of manually entering phony data into the db and the query would try to filter this phony data. At the end, the transaction is never completed so it is rolled back. Due to the nature of transaction scopes, this might not be an ideal scenario for a lot of projects. Most probably not on prod environments.
Otherwise if you must continue mocking functionality, you might want to consider other mocking frameworks.
Hmm, not sure but couldn't you do something like this?
context.Setup(s => DbFunctions.TruncateTime(It.IsAny<DateTime>()))
.Returns<DateTime?>(new Func<DateTime?,DateTime?>(
(x) => {
/* whatever modification is required here */
return x; //or return modified;
}));
since i hit the same problem recently, and opted for a simpler solution, wanted to post it here.. this solution requires no Shims, Mocking, nothing expansive etc.
Pass a 'useDbFunctions' boolean flag to your method with default value as true.
When your live code executes, your query will use DbFunctions and everything will work. Due to the default value, callers need not worry about it.
When your unit tests invoke the method to test, they can pass useDbFunctions: false.
In your method, you can make use the flag to compose your IQueryable..
if useDbFunctions is true, use the DbFunctions to add the predicate to the queryable.
if useDbFunctions is false, then skip the DbFunctions method call, and do an explicit C# equivalent solution.
This way, your unit tests will check almost 95% of your method in parity with live code. You still have the delta of "DbFunctions" vs. your equivalent code, but be diligent about it and the 95% will look like a lot of gain.
public System.Data.DataTable GetLinkedUsers(int parentUserId, bool useDbFunctions = true)
{
var today = DateTime.Now.Date;
var queryable = from up in DB.par_UserPlacement
where up.MentorId == mentorUserId;
if (useDbFunctions) // use the DbFunctions
{
queryable = queryable.Where(up =>
DbFunctions.TruncateTime(today) >= DbFunctions.TruncateTime(up.StartDate)
&& DbFunctions.TruncateTime(today) <= DbFunctions.TruncateTime(up.EndDate));
}
else
{
// do db-functions equivalent here using C# logic
// this is what the unit test path will invoke
queryable = queryable.Where(up => up.StartDate < today);
}
var query = from up in queryable
select new
{
up.UserPlacementId,
up.Users.UserId,
up.Users.FirstName,
up.Users.LastName,
up.Placements.PlacementId,
up.Placements.PlacementName,
up.StartDate,
up.EndDate,
};
query = query.OrderBy(up => up.EndDate);
return this.RunQueryToDataTable(query);
}
Unit tests will invoke the mthod as:
GetLinkedUsers(parentUserId: 10, useDbFunctions: false);
Because unit tests would have setup local DbContext entities, the C# logic/DateTime functions would work.
Use of Mocks ended sometime ago. Do not Mock, just connect to real DB. Regenerate/Seed DB on start of test.
If you still want to go ahead with mocks then create your own method as given below. IT changes behaviour runtime. When using real DB it uses DB functions, else this method. Replace DBfunctions method in code with this method
public static class CanTestDbFunctions
{
[System.Data.Entity.DbFunction("Edm", "TruncateTime")]
public static DateTime? TruncateTime(DateTime? dateValue)
{
...
}
}
This is the real function that is called. And remember, time cannot be removed from DateTime object, live with midnight or create a string equivalent.
I have Fluent NHibernate Linq queries where I check values based on run time arrays. A basic example would be something like:
var array = [1,2,3,4,5,6];
using (var session = SessionProvider.SessionFactory.OpenSession())
{
return session.Query<MyObject>().Where(x => array.Contains(x.CompareVal)).ToList();
}
I would expect the generated SQL statement to look something like this:
SELECT CompareVal, Column1, Column2
FROM MyObject
WHERE CompareVal IN (1,2,3,4,5,6)
However, what I'm finding instead is that the generated SQL statement simply emits the WHERE clause (proven by watching in Profiler) and selects the entire table, and then seems to run the filter in memory once it gets all the data back.
Something to note - I have a Generic Repository class that all of these calls are funneled through. The Query method is as follows:
public IList<T> Query(Func<T, bool> criteria)
{
using (var session = SessionProvider.SessionFactory.OpenSession())
{
return session.Query<T>().Where(criteria).ToList();
}
}
Obviously this (lack of a where clause) is not acceptable in a table with a large amount of data. What can I do to force NHibernate to generate the query correctly with the WHERE clause and still keep a generic pattern for repositories?
Does it make a difference if you change your Query method to the following ?
public IList<T> Query(Expression<Func<T, bool>> criteria)
{
using (var session = SessionProvider.SessionFactory.OpenSession())
{
return session.Query<T>().Where(criteria).ToList();
}
}
This is how I usually proceed with a generic Query :
public List<TOut> GetEntitiesLinq<TIn,TOut>(Expression<Func<IQueryable<TIn>,IQueryable<TOut>>> myFunc)
{
var t = (myFunc.Compile())(_session.Query<TIn>()) ;
return t.ToList();
}
Then how I would use it in your case :
var myObjList = myQueryManager.GetEntitiesLinq<MyObject,MyObject>(x=>x.Where(myObj => array.Contains(myObj.CompareVal)));
Hope this will help
Use Any:
return session.Query<MyObject>().Where(x => array.Any(y => y == x.CompareVal)).ToList();
Your repository pattern (using plain Func) automatically materializes your query to list, if you want something to be deferredly executed, use IQueryable, don't use Func only
Something to note - I have a Generic Repository class that all of
these calls are funneled through. The Query method is as follows:
public IList<T> Query(Func<T, bool> criteria)
{
using (var session = SessionProvider.SessionFactory.OpenSession())
{
return session.Query<T>().Where(criteria).ToList();
}
}
Your repository just mimic what is already provided out of the box by NHibernate
Can you use QueryOver and WhereRestrictionOn instead?
session.QueryOver<MyObject>().WhereRestrictionOn(o => o.CompareVal).IsIn(array).List();