How to use DbFunction translation in EF Core? - c#

I'm looking for something like EF.Functions.FreeText that was implemented in SQL Server but using the MATCH...AGAINST syntax of MySQL.
This is my current workflow:
AspNetCore 2.1.1
EntityFrameworkCore 2.1.4
Pomelo.EntityFrameworkCore.MySql 2.1.4
The problem is that MySQL uses two functions and I don't know how to interpret that with DbFunction and separate the arguments for each one. Does anyone know how to implement this?
This should be the Linq syntax:
query.Where(x => DbContext.FullText(new[] { x.Col1, x.Col2, x.Col3 }, "keywords"));
And this should be the result generated in SQL:
SELECT * FROM t WHERE MATCH(`Col1`, `Col2`, `Col3`) AGAINST('keywords');
I'm trying to follow the following examples using the HasTranslation function:
https://github.com/aspnet/EntityFrameworkCore/issues/11295#issuecomment-511440395
https://github.com/aspnet/EntityFrameworkCore/issues/10241#issuecomment-342989770
Note: I know it can be solved with FromSql, but it's not what I'm looking for.

Your use case is very similar to mine when I needed ROW_NUMBER support in EF Core.
Example:
// gets translated to
// ROW_NUMBER() OVER(PARTITION BY ProductId ORDER BY OrderId, Count)
DbContext.OrderItems.Select(o => new {
RowNumber = EF.Functions.RowNumber(o.ProductId, new {
o.OrderId,
o.Count
})
})
Use anonymous classes instead of arrays
The first thing you have to do is to switch from using an array to an anonymous class, i.e. you change the call from
DbContext.FullText(new[] { x.Col1, x.Col2, x.Col3 }, "keywords")
to
DbContext.FullText(new { x.Col1, x.Col2, x.Col3 }, "keywords")
The sort order of the parameters will stay as it is defined by in the query,
i.e new { x.Col1, x.Col2 } will be translated to Col1, Col2
and new { x.Col2, x.Col1 } to Col2, Col1.
You can even to the following: new { x.Col1, _ = x.Col1, Foo = "bar" } that is going to be translated to Col1, Col1, 'bar'.
Implement custom IMethodCallTranslator
If you need some hints then you can look through my code on Azure DevOps: RowNumber Support or if you can wait a few days then I will provide a blog post about the implementation of custom functions.
Updated (31 july, 2019)
Blog posts:
Entity Framework Core: Custom Functions (using IMethodCallTranslator)
Entity Framework Core: Custom Functions (using HasDbFunction)
Updated (july 27, 2019)
Thanks to the comments below I see that some clarification is required.
1) As pointed out in the comment below there is another approach. With HasDbFunction I could save me some typing like the code for registration of the translator with EF but I would still need the RowNumberExpression because the function has 2 sets of parameters (for PARTITION BY and ORDER BY) and the existing SqlFunctionExpression doesn't support that. (or did I missed something?) The reason I've chosen the approach with IMethodCallTranslator is because I want the configuration of this feature to be done during setting up of the DbContextOptionsBuilder and not in OnModelCreating. That is, it’s a personal preference of mine.
In the end the thread creator can use HasDbFunction to implement the desired feature as well. In my case the code would look something like the following:
// OnModelCreating
var methodInfo = typeof(DemoDbContext).GetMethod(nameof(DemoRowNumber));
modelBuilder.HasDbFunction(methodInfo)
.HasTranslation(expressions => {
var partitionBy = (Expression[])((ConstantExpression)expressions.First()).Value;
var orderBy = (Expression[])((ConstantExpression)expressions.Skip(1).First()).Value;
return new RowNumberExpression(partitionBy, orderBy);
});
// the usage with this approach is identical to my current approach
.Select(c => new {
RowNumber = DemoDbContext.DemoRowNumber(
new { c.Id },
new { c.RowVersion })
})
2) An anonymous type can’t enforce the type(s) of its members, so you can get a runtime exception if the function is called with, say, integer instead of string. Still, it can be valid solution. Depending on the customer you are working for the solution may be more or less viable, in the end the decision lies with the customer. Not providing any alternatives is a possible solution as well but not a satisfying one.
Especially, if the usage of SQL is not desired (because you get even less support from compiler) so the runtime exception may be a good compromise after all.
But, if the compromise is still not acceptable then we can make a research on how to add support for arrays.
First approach could be the implementation of a custom IExpressionFragmentTranslator to “redirect” the handling of arrays to us.
Please note, it is just a prototype and needs more investigation/testing :-)
// to get into EF pipeline
public class DemoArrayTranslator : IExpressionFragmentTranslator
{
public Expression Translate(Expression expression)
{
if (expression?.NodeType == ExpressionType.NewArrayInit)
{
var arrayInit = (NewArrayExpression)expression;
return new DemoArrayInitExpression(arrayInit.Type, arrayInit.Expressions);
}
return null;
}
}
// lets visitors visit the array-elements
public class DemoArrayInitExpression : Expression
{
private readonly ReadOnlyCollection<Expression> _expressions;
public override Type Type { get; }
public override ExpressionType NodeType => ExpressionType.Extension;
public DemoArrayInitExpression(Type type,
ReadOnlyCollection<Expression> expressions)
{
Type = type ?? throw new ArgumentNullException(nameof(type));
_expressions = expressions ?? throw new ArgumentNullException(nameof(expressions));
}
protected override Expression Accept(ExpressionVisitor visitor)
{
var visitedExpression = visitor.Visit(_expressions);
return NewArrayInit(Type.GetElementType(), visitedExpression);
}
}
// adds our DemoArrayTranslator to the others
public class DemoRelationalCompositeExpressionFragmentTranslator
: RelationalCompositeExpressionFragmentTranslator
{
public DemoRelationalCompositeExpressionFragmentTranslator(
RelationalCompositeExpressionFragmentTranslatorDependencies dependencies)
: base(dependencies)
{
AddTranslators(new[] { new DemoArrayTranslator() });
}
}
// Register the translator
services
.AddDbContext<DemoDbContext>(builder => builder
.ReplaceService<IExpressionFragmentTranslator,
DemoRelationalCompositeExpressionFragmentTranslator>());
For testing I introduced another overload containing Guid[] as parameter.
Although, this method doesn't make sense in my use case at all :)
public static long RowNumber(this DbFunctions _, Guid[] orderBy)
And adjusted the usage of the method
// Translates to ROW_NUMBER() OVER(ORDER BY Id)
.Select(c => new {
RowNumber = EF.Functions.RowNumber(new Guid[] { c.Id })
})

Related

Using Function logic in LINQ Query net core 3

I have the following enum:
public enum WorkType
{
Type1,
Type2,
Type3,
Type4,
Type5,
Type6
}
and a class
public class Work {
public WorkType Type {get; set;}
....
}
and an extension method:
public static partial class WorkTypeExtensions
{
public static bool IsHighValueWork(this WorkType value)
{
switch (value)
{
case WorkType.Type1:
case WorkType.Type2:
return true;
default:
return false;
}
}
}
and SQL Linq query
public List<Work> GetHighValueWork()
{
var query = Context.Work.Where( w => w.IsHighValueWork());
return query.ToList();
}
This is a simplified version of my problem. This query used to work, but it is not working any more after the code was converted from net core 2.1 to 3.1. The error msg is
The query could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(). I don't want to change it to
public List<Work> GetHighValueWork()
{
var query = Context.Work.Where( w => w.Type == WorkType.Type1 || w.Type == WorkType.Type2);
return query.ToList();
}
Because actual function is very complex. I searched it seems LINQ Expression Func can be used, but I haven't figured that yet. What is the best way to do this?
IsHighValueWork is just a simple C# method. There is no way to convert that function to SQL by EF.
It is really explained well in that link, why it was working in .net core 2.1. It seems that, in previous versions when EF Core couldn't convert an expression that was part of a query to either SQL or a parameter, it automatically evaluated the expression on the client.
And it is really bad. Because, as noted:
For example, a condition in a Where() call which can't be translated can cause all rows from the table to be transferred from
the database server, and the filter to be applied on the client.
So, it seems previously you were just loading all data to the client and then applying filter on the client side.
So, the problem with your code is, that Func cant be translated into Sql.
Either fetch all data into app explicitly and filter then or use second version of you code.
Context.Work.ToList()
.Where( w => w.Type.IsHighValueWork());
But, I don't recommend to use that version. It is better to use second version like so:
Func<Work, bool> IsHighValueWork = (work) =>
work.Type == WorkType.Type1 || work.Type == WorkType.Type2;
And then:
var query = Context.Work.Where(IsHighValueWork);

GroupBy returns different results

IEnumerable<IGrouping<StatusType, Request>> group = requests.GroupBy(r=> r.StatusType );
The grouping function above works with when requests (List<Requests>) is from EntityFramework/db.
When changing the assignment of requests from db direct, to a web service,
the grouping isn't working as intended.
Digging a bit, I found that the hash or equality of the StatusType's is different when coming from db vs web (found out thru this post).
From the accepted answer of the post, I can bypass/(resolve?) the problem by overriding..
public class StatusType : IEquatable<int>
{ // omitted other crucial equality comparison components.
// but for brevity..
public override int GetHashCode()
{
return Id;
}
}
Although overriding StatusType somewhat resolves the issue,
I feel its quite risky as
I am not the author of the code base.
There are multiple references to StatusType increasing the potential
of impending failure.
My question,
Is there a way to group by the StatusTypeId (int)
requests.groupBy(r=> r.StatusTypeId) // returns IEnumerable<IGrouping<int,Rquest>>
but get the StatusType?
IEnumerable<IGrouping<StatusType,Rquest>>
Define comparer for StatusType:
public class StatusTypeComparer : IEqualityComparer<StatusType>
{
public bool Equals(StatusType x, StatusType y)
{
return x.Id == y.Id;
}
public int GetHashCode(StatusType obj)
{
return obj.Id.GetHashCode();
}
}
Pass it to GroupBy method:
IEnumerable<IGrouping<StatusType, Request>> group =
requests.GroupBy(r => r.StatusType,
new StatusTypeComparer());
Disclaimer: Backs has a much better answer than mine but I thought I'd post it anyway in the interests of diversity.
You might be able to get something like the functionality you're looking for by using multiple Linq queries. I don't know if there is an accessible implementation of IGrouping I can use, so I've gone with Tuple<StatusType, List<Request>> instead. It should have a similar effect. So, from your original query:
IEnumerable<IGrouping<int, Request>> group = requests.GroupBy(r=> r.StatusTypeId );
You can add the following line:
IEnumerable<Tuple<StatusType, List<Request>>> groupByStatusType =
group.Select(x => new Tuple<StatusType, List<Request>>(x.First().StatusType,
x.ToList()));
Or, you can do it all on one line:
IEnumerable<Tuple<StatusType, List<Request>>> group =
requests.GroupBy(r => r.StatusTypeId)
.Select(x => new Tuple<StatusType, List<Request>>(x.First().StatusType,
x.ToList()));
You can of course tweak the queries depending on what kind of output you're expecting, but this should at least get you started. Alternately, you could get a similar result by implementing a function that iterates through everything and "manually" creates an output.

Mock DbFunctions [duplicate]

I'm currently attempting to run some unit tests on a query that is running through the Entity Framework. The query itself runs without any issues on the live version, but the unit tests are always failing.
I've narrowed this down to my usage of DbFunctions.TruncateTime, but I don't know of a way around this to get the unit tests to reflect what is happening on the live server.
Here is the method that I am using:
public System.Data.DataTable GetLinkedUsers(int parentUserId)
{
var today = DateTime.Now.Date;
var query = from up in DB.par_UserPlacement
where up.MentorId == mentorUserId
&& DbFunctions.TruncateTime(today) >= DbFunctions.TruncateTime(up.StartDate)
&& DbFunctions.TruncateTime(today) <= DbFunctions.TruncateTime(up.EndDate)
select new
{
up.UserPlacementId,
up.Users.UserId,
up.Users.FirstName,
up.Users.LastName,
up.Placements.PlacementId,
up.Placements.PlacementName,
up.StartDate,
up.EndDate,
};
query = query.OrderBy(up => up.EndDate);
return this.RunQueryToDataTable(query);
}
If I comment out the lines with DbFunctions in, the tests all pass (except for the ones that are checking that only valid results for a given date are run).
Is there a way I can provide a mocked version of DbFunctions.TruncateTime to use in these tests? Essentially it should just be returning Datetime.Date, but that isn't available in EF queries.
Edit: Here's the test that's failing that uses the date check:
[TestMethod]
public void CanOnlyGetCurrentLinkedUsers()
{
var up = new List<par_UserPlacement>
{
this.UserPlacementFactory(1, 2, 1), // Create a user placement that is current
this.UserPlacementFactory(1, 3, 2, false) // Create a user placement that is not current
}.AsQueryable();
var set = DLTestHelper.GetMockSet<par_UserPlacement>(up);
var context = DLTestHelper.Context;
context.Setup(c => c.par_UserPlacement).Returns(set.Object);
var getter = DLTestHelper.New<LinqUserGetLinkedUsersForParentUser>(context.Object);
var output = getter.GetLinkedUsers(1);
var users = new List<User>();
output.ProcessDataTable((DataRow row) => students.Add(new UserStudent(row)));
Assert.AreEqual(1, users.Count);
Assert.AreEqual(2, users[0].UserId);
}
Edit 2: This is the message and debug trace from the test in question:
Test Result: Failed
Message: Assert.AreEqual failed. Expected:<1>. Actual:<0>
Debug Trace: This function can only be invoked from LINQ to Entities
From what I've read, this is because there isn't a LINQ to Entities implementation of this method that could be used in this place for the Unit Test, although there is on the live version (as it's querying an SQL server).
I know I'm late to the game, but a very simple fix is to write your own method which uses the DbFunction attribute. Then use that function instead of DbFunctions.TruncateTime.
[DbFunction("Edm", "TruncateTime")]
public static DateTime? TruncateTime(DateTime? dateValue)
{
return dateValue?.Date;
}
Using this function will execute the EDM TruncateTime method when used by Linq to Entities and will run the provided code otherwise.
Thanks for all of the help everyone, I managed to track down a solution that worked for me after reading up on shims that qujck mentioned. After adding a fake assembly of EntityFramework, I was able to fix these tests by changing them to the following:
[TestMethod]
public void CanOnlyGetCurrentLinkedUsers()
{
using (ShimsContext.Create())
{
System.Data.Entity.Fakes.ShimDbFunctions.TruncateTimeNullableOfDateTime =
(DateTime? input) =>
{
return input.HasValue ? (DateTime?)input.Value.Date : null;
};
var up = new List<par_UserPlacement>
{
this.UserPlacementFactory(1, 2, 1), // Create a user placement that is current
this.UserPlacementFactory(1, 3, 2, false) // Create a user placement that is not current
}.AsQueryable();
var set = DLTestHelper.GetMockSet<par_UserPlacement>(up);
var context = DLTestHelper.Context;
context.Setup(c => c.par_UserPlacement).Returns(set.Object);
var getter = DLTestHelper.New<LinqUserGetLinkedUsersForParentUser>(context.Object);
var output = getter.GetLinkedUsers(1);
}
var users = new List<User>();
output.ProcessDataTable((DataRow row) => users.Add(new User(row)));
Assert.AreEqual(1, users.Count);
Assert.AreEqual(2, users[0].UserId);
}
There is a way to do it. Since unit testing of business logic is generally encouraged, and since it is perfectly OK for business logic to issue LINQ queries against application data, then it must be perfectly OK to unit test those LINQ queries.
Unfortunately, DbFunctions feature of Entity Framework kills our ability to unit test code that contains LINQ queries. Moreover, it is architecturally wrong to use DbFunctions in business logic, because it couples business logic layer to a specific persistence technology (which is a separate discussion).
Having said that, our goal is the ability to run LINQ query like this:
var orderIdsByDate = (
from o in repo.Orders
group o by o.PlacedAt.Date
// here we used DateTime.Date
// and **NOT** DbFunctions.TruncateTime
into g
orderby g.Key
select new { Date = g.Key, OrderIds = g.Select(x => x.Id) });
In unit test, this will boil down to LINQ-to-Objects running against a plain array of entities arranged in advance (for example). In a real run, it must work against a real ObjectContext of Entity Framework.
Here is a recipe of achieving it - although, it requires a few steps of yours. I'm cutting down a real working example:
Step 1. Wrap ObjectSet<T> inside our own implementation of IQueryable<T> in order to provide our own intercepting wrapper of IQueryProvider.
public class EntityRepository<T> : IQueryable<T> where T : class
{
private readonly ObjectSet<T> _objectSet;
private InterceptingQueryProvider _queryProvider = null;
public EntityRepository<T>(ObjectSet<T> objectSet)
{
_objectSet = objectSet;
}
IEnumerator<T> IEnumerable<T>.GetEnumerator()
{
return _objectSet.AsEnumerable().GetEnumerator();
}
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
return _objectSet.AsEnumerable().GetEnumerator();
}
Type IQueryable.ElementType
{
get { return _objectSet.AsQueryable().ElementType; }
}
System.Linq.Expressions.Expression IQueryable.Expression
{
get { return _objectSet.AsQueryable().Expression; }
}
IQueryProvider IQueryable.Provider
{
get
{
if ( _queryProvider == null )
{
_queryProvider = new InterceptingQueryProvider(_objectSet.AsQueryable().Provider);
}
return _queryProvider;
}
}
// . . . . . you may want to include Insert(), Update(), and Delete() methods
}
Step 2. Implement the intercepting query provider, in my example it is a nested class inside EntityRepository<T>:
private class InterceptingQueryProvider : IQueryProvider
{
private readonly IQueryProvider _actualQueryProvider;
public InterceptingQueryProvider(IQueryProvider actualQueryProvider)
{
_actualQueryProvider = actualQueryProvider;
}
public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
{
var specializedExpression = QueryExpressionSpecializer.Specialize(expression);
return _actualQueryProvider.CreateQuery<TElement>(specializedExpression);
}
public IQueryable CreateQuery(Expression expression)
{
var specializedExpression = QueryExpressionSpecializer.Specialize(expression);
return _actualQueryProvider.CreateQuery(specializedExpression);
}
public TResult Execute<TResult>(Expression expression)
{
return _actualQueryProvider.Execute<TResult>(expression);
}
public object Execute(Expression expression)
{
return _actualQueryProvider.Execute(expression);
}
}
Step 3. Finally, implement a helper class named QueryExpressionSpecializer, which would replace DateTime.Date with DbFunctions.TruncateTime.
public static class QueryExpressionSpecializer
{
private static readonly MethodInfo _s_dbFunctions_TruncateTime_NullableOfDateTime =
GetMethodInfo<Expression<Func<DateTime?, DateTime?>>>(d => DbFunctions.TruncateTime(d));
private static readonly PropertyInfo _s_nullableOfDateTime_Value =
GetPropertyInfo<Expression<Func<DateTime?, DateTime>>>(d => d.Value);
public static Expression Specialize(Expression general)
{
var visitor = new SpecializingVisitor();
return visitor.Visit(general);
}
private static MethodInfo GetMethodInfo<TLambda>(TLambda lambda) where TLambda : LambdaExpression
{
return ((MethodCallExpression)lambda.Body).Method;
}
public static PropertyInfo GetPropertyInfo<TLambda>(TLambda lambda) where TLambda : LambdaExpression
{
return (PropertyInfo)((MemberExpression)lambda.Body).Member;
}
private class SpecializingVisitor : ExpressionVisitor
{
protected override Expression VisitMember(MemberExpression node)
{
if ( node.Expression.Type == typeof(DateTime?) && node.Member.Name == "Date" )
{
return Expression.Call(_s_dbFunctions_TruncateTime_NullableOfDateTime, node.Expression);
}
if ( node.Expression.Type == typeof(DateTime) && node.Member.Name == "Date" )
{
return Expression.Property(
Expression.Call(
_s_dbFunctions_TruncateTime_NullableOfDateTime,
Expression.Convert(
node.Expression,
typeof(DateTime?)
)
),
_s_nullableOfDateTime_Value
);
}
return base.VisitMember(node);
}
}
}
Of course, the above implementation of QueryExpressionSpecializer can be generalized to allow plugging in any number of additional conversions, allowing members of custom types to be used in LINQ queries, even though they are not known to Entity Framework.
Check out this answer: https://stackoverflow.com/a/14975425/1509728
To be honest, upon thinking about it I totally agree with the answer and generally follow the principle that my EF queries are tested against the database and only my application code is tested with Moq.
It looks like there is no elegant solution to using Moq for testing EF queries with your query above, while there are some hacky ideas out there. For example this one and the answer that follows it. Both seem like they could work for you.
Another approach to testing your queries would be one implemented on another project I worked on: Using VS out of box unit tests, each query (again refactored into its own method) test would be wrapped in a transaction scope. Then the project's test framework would take care of manually entering phony data into the db and the query would try to filter this phony data. At the end, the transaction is never completed so it is rolled back. Due to the nature of transaction scopes, this might not be an ideal scenario for a lot of projects. Most probably not on prod environments.
Otherwise if you must continue mocking functionality, you might want to consider other mocking frameworks.
Hmm, not sure but couldn't you do something like this?
context.Setup(s => DbFunctions.TruncateTime(It.IsAny<DateTime>()))
.Returns<DateTime?>(new Func<DateTime?,DateTime?>(
(x) => {
/* whatever modification is required here */
return x; //or return modified;
}));
since i hit the same problem recently, and opted for a simpler solution, wanted to post it here.. this solution requires no Shims, Mocking, nothing expansive etc.
Pass a 'useDbFunctions' boolean flag to your method with default value as true.
When your live code executes, your query will use DbFunctions and everything will work. Due to the default value, callers need not worry about it.
When your unit tests invoke the method to test, they can pass useDbFunctions: false.
In your method, you can make use the flag to compose your IQueryable..
if useDbFunctions is true, use the DbFunctions to add the predicate to the queryable.
if useDbFunctions is false, then skip the DbFunctions method call, and do an explicit C# equivalent solution.
This way, your unit tests will check almost 95% of your method in parity with live code. You still have the delta of "DbFunctions" vs. your equivalent code, but be diligent about it and the 95% will look like a lot of gain.
public System.Data.DataTable GetLinkedUsers(int parentUserId, bool useDbFunctions = true)
{
var today = DateTime.Now.Date;
var queryable = from up in DB.par_UserPlacement
where up.MentorId == mentorUserId;
if (useDbFunctions) // use the DbFunctions
{
queryable = queryable.Where(up =>
DbFunctions.TruncateTime(today) >= DbFunctions.TruncateTime(up.StartDate)
&& DbFunctions.TruncateTime(today) <= DbFunctions.TruncateTime(up.EndDate));
}
else
{
// do db-functions equivalent here using C# logic
// this is what the unit test path will invoke
queryable = queryable.Where(up => up.StartDate < today);
}
var query = from up in queryable
select new
{
up.UserPlacementId,
up.Users.UserId,
up.Users.FirstName,
up.Users.LastName,
up.Placements.PlacementId,
up.Placements.PlacementName,
up.StartDate,
up.EndDate,
};
query = query.OrderBy(up => up.EndDate);
return this.RunQueryToDataTable(query);
}
Unit tests will invoke the mthod as:
GetLinkedUsers(parentUserId: 10, useDbFunctions: false);
Because unit tests would have setup local DbContext entities, the C# logic/DateTime functions would work.
Use of Mocks ended sometime ago. Do not Mock, just connect to real DB. Regenerate/Seed DB on start of test.
If you still want to go ahead with mocks then create your own method as given below. IT changes behaviour runtime. When using real DB it uses DB functions, else this method. Replace DBfunctions method in code with this method
public static class CanTestDbFunctions
{
[System.Data.Entity.DbFunction("Edm", "TruncateTime")]
public static DateTime? TruncateTime(DateTime? dateValue)
{
...
}
}
This is the real function that is called. And remember, time cannot be removed from DateTime object, live with midnight or create a string equivalent.

A "Composable" Full Text Search with a Code First Model

UPDATE 18 Sep 2013
It looks like there isn't an easy way to do this. I'm holding out for a solution that involves some extension to Entity Framework.
If you'd like to see these features in Entity Framework, vote for them on the user voice site, perhaps here and here
There are several similar questions on SO but I can't find a question new and similar enough to have the answer I'm looking for.
If this looks like information overload, jump down to In Summary.
Background
I'm writing a WebApi REST service to expose some pre-existing data through an OData end point. I'm using the EntitySetContoller<TEntity, TKey> to do all the grunt work for me. As well as the standard OData parameters, that are routed and translated by the base class, I've added some custom parameters, to allow specific functionality for my controller.
My database server is MS SQL Server with a full text index on the [BigText] NVarChar[4000] column of the [SomeEntity] table.
I have one limitation, I must use a Code First model.
// Model POCO
public class SomeEntity
{
public int Id { get; set; }
public string BigText { get; set; }
}
// Simple Controller
public class SomeEntityController : EntitySetController<SomeEntity, int>
{
private readonly SomeDbContext context = new SomeDbContext();
public override IQueryable<SomeEntity> Get()
{
var parameters = Request.GetQueryNameValuePairs()
.ToDictionary(p => p.Key, p => p.Value);
if (parameters.ContainsKey("BigTextContains")
(
var searchTerms = parameters["BigTextContains"];
// return something special ...
)
return this.context.SomeEntities;
}
// ... The rest is omitted for brevity.
}
The Problem
How to implement the // return something special ... part of my example?
Obviously, the niave
return this.context.SomeEntities.Where(e =>
e.BigText.Contains(searchTerm));
is completely wrong, it composes to a WHERE clause like
[BigText] LIKE '%' + #searchTerm + '%'
This doesn't use Full Text Searching so, doesn't support complex search terms and otherwise, performs terribley.
This approach,
return this.context.SomeEntities.SqlQuery(
"SELECT E.* FROM [dbo].[SomeEntity] E " +
"JOIN CONTAINSTABLE([SomeEntity], [BigText], #searchTerm) FTS " +
" ON FTS.[Key] = E.[Id]",
new object[] { new SqlParameter("#searchTerm", searchTerm) })
.AsQueryable();
Looks promising, it actually uses Full Text Searching, and is quite functional. However, you'll note that DbSqlQuery, the type returned from the SqlQuery function does not implement IQueryable. Here, it is coerced to the right return type with the AsQueryable() extension but, this breaks the "chain of composition". The only statement that will be performed on the server is the one specified in the code above. Any additional clauses, specified on the OData URL will be serviced on the API hosting web server, without benefitting from the indices and specialised set based functionality of the database engine.
In Summary
What is the most expedient way of accessing MS SQL Server's Full Text Search CONTAINSTABLE function with an Entity Framework 5 Code First model and acquiring a "composable" result?
Do I need to write my own IQueryProvider? Can I extend EF in some way?
I don't want to use Lucene.Net, I don't want to use a Database Generated Model. Perhaps I could add extra packages or wait for EF6, would that help?
It is not perfect, but you can accomplish what you are after with 2 calls to the database.
The first call would retrieve a list of matching key's from CONTAINSTABLE and then the second call would be your composable query utilizing the IDs that you returned from the first call.
//Get the Keys from the FTS
var ids = context.Database.SqlQuery<int>(
"Select [KEY] from CONTAINSTABLE([SomeEntity], [BigText], #searchTerm)",
new object[] { new SqlParameter("#searchTerm", searchTerm) });
//Use the IDs as an initial filter on the query
var composablequery = context.SomeEntities.Where(d => ids.Contains(d.Id));
//add on whatever other parameters were captured to the 'composablequery' variable
composablequery = composablequery.Where(.....)
I had this same issue recently:
EF 5 Code First FTS Queriable
Let me extend that post.
Your first option was mine first as well - using SqlQuery
I also needed to do more filtering, so instead of always writing full sql I used QueryBuilder, to which I made some changes and added more functions to fit my needs(I could upload it somewhere if needed):
QueryBuilder
After I have found another idea which I implemented.
Someone already mention it here, and that is to use SqlQuery that will return HashSet of Ids and that you can use it in EF queries with Contains.
This is better but not most optimal since you need 2 queries and Id list in memory.
Example:
public IQueryable<Company> FullTextSearchCompaniesByName(int limit, int offset, string input, Guid accountingBureauId, string orderByColumn)
{
FtsQueryBuilder ftsQueryBuilder = new FtsQueryBuilder();
ftsQueryBuilder.Input = FtsQueryBuilder.FormatQuery(input);
ftsQueryBuilder.TableName = FtsQueryBuilder.GetTableName<Company>();
ftsQueryBuilder.OrderByTable = ftsQueryBuilder.TableName;
ftsQueryBuilder.OrderByColumn = orderByColumn;
ftsQueryBuilder.Columns.Add("CompanyId");
if (accountingBureauId != null && accountingBureauId != Guid.Empty)
ftsQueryBuilder.AddConditionQuery<Guid>(Condition.And, "" , #"dbo.""Company"".""AccountingBureauId""", Operator.Equals, accountingBureauId, "AccountingBureauId", "");
ftsQueryBuilder.AddConditionQuery<bool>(Condition.And, "", #"dbo.""Company"".""Deleted""", Operator.Equals, false, "Deleted", "");
var companiesQuery = ftsQueryBuilder.BuildAndExecuteFtsQuery<Guid>(Context, limit, offset, "Name");
TotalCountQuery = ftsQueryBuilder.Total;
HashSet<Guid> companiesIdSet = new HashSet<Guid>(companiesQuery);
var q = Query().Where(a => companiesIdSet.Contains(a.CompanyId));
return q;
}
However EF 6 now has something called Interceptors that can be used to implement queriable FTS, and it is pretty simple and generic(last post):
EF 6 Interceptors for FTS.
I have tested this and it works fine.
!! REMARK: EF Code First, even with version 6, does not support Custom Stored Procedures.
There are only some for predefined CUD operations if I understood it well:
Code First Insert/Update/Delete Stored Procedure Mapping, so it can't be done with it.
Conclusion: if you can use EF 6 go for third options, is gives all you need.
If you are stucked with EF 5 or less, second option is better then first but not most optimal.

SQL "not in" syntax for Entity Framework 4.1

I have a simple issue with Entity Framework syntax for the "not in" SQL equivalent. Essentially, I want to convert the following SQL syntax into Entity Framework syntax:
select ID
from dbo.List
where ID not in (list of IDs)
Here is a method that I use for looking up a single record:
public static List GetLists(int id)
{
using (dbInstance db = new dbInstance())
{
return db.Lists.Where(m => m.ID == id);
}
}
Here is a pseudo-method that I want to use for this:
public static List<List> GetLists(List<int> listIDs)
{
using (dbInstance db = new dbInstance())
{
return db.Lists.Where(**** What Goes Here ****).ToList();
}
}
Can anyone give me pointers as to what goes in the Where clause area? I read some forums about this and saw mention of using .Contains() or .Any(), but none of the examples were a close enough fit.
Give this a go...
public static List<List> GetLists(List<int> listIDs)
{
using (dbInstance db = new dbInstance())
{
// Use this one to return List where IS NOT IN the provided listIDs
return db.Lists.Where(x => !listIDs.Contains(x.ID)).ToList();
// Or use this one to return List where IS IN the provided listIDs
return db.Lists.Where(x => listIDs.Contains(x.ID)).ToList();
}
}
These will turn into approximately the following database queries:
SELECT [Extent1].*
FROM [dbo].[List] AS [Extent1]
WHERE NOT ([Extent1].[ID] IN (<your,list,of,ids>))
or
SELECT [Extent1].*
FROM [dbo].[List] AS [Extent1]
WHERE [Extent1].[ID] IN (<your,list,of,ids>)
respectively.
This one requires you to think backwards a little bit. Instead of asking if the value is not in some list of ids, you have to ask of some list of id's does not contain the value. Like this
int[] list = new int[] {1,2,3}
Result = (from x in dbo.List where list.Contains(x.id) == false select x);
Try this for starters ...
m => !listIDs.Contains(m.ID)
This might be a way to do what you want:
// From the method you provided, with changes...
public static List GetLists(int[] ids) // Could be List<int> or other =)
{
using (dbInstance db = new dbInstance())
{
return db.Lists.Where(m => !ids.Contains(m.ID));
}
}
However I've found that doing so might raise error on some scenarios, specially when the list is too big and connection is somewhat slow.
Remember to check everything else BEFORE so this filter might have less values to check.
Also remember that Linq does not populate the variable when you build your filter/query (at least not by default). If you're going to iterate for each record, remember to call a ToList() or ToArray() method before, unless each record has 500MB or more...

Categories