I have a code-first EF database where objects have "statuses" to track history. They're implemented something like this:
public class Example
{
public Example()
{
this.Statuses = new HashSet<Status>();
}
public Guid Id { get; set; }
public virtual ICollection<Status> Statuses { get; set; }
}
public class Status
{
public Guid Id { get; set; }
public DateTimeOffset SetOn { get; set; }
public string SetBy { get; set; }
}
We have a few instances in code where we need to get either the oldest or newest status. Currently we've been using chained linq expressions like the following:
var setBy = example.Statuses.OrderByDescending(s => s.SetOn).FirstOrDefault().SetBy;
I think it would be more readable if we could do some of that with extensions, since getting the newest or oldest status is just a difference of whether it's sorted by ascending or descending.
A simple extension method like this works with linq-to-objects, if we've already gotten results from the database:
public static Status Newest(this IQueryable<Status> items)
{
return items.OrderByDescending(s => s.SetOn).FirstOrDefault();
}
However, this doesn't work if I'm running it on an IQueryable representing our database, since EF is unable to translate it to a store expression. For instance, if "repository" below is an IQueryable<Example> representing Examples in a SQL backend, the following will fail:
var date = DateTimeOffset.Parse("4/1/2018");
var query = repository.Where(ex => ex.Statuses.Newest().SetOn > date).FirstOrDefault();
Is there a way I can refactor this into an extension method or expression that can be translated to a store expression?
This can be done with LINQKit by defining an expression that returns a Status from an Example, and wrapping the IQueryable with LINQKit's Expandable. Using the above classes, I could do something like
private Expression<Func<Example, Status>> Newest =
e => e.Statuses.OrderByDescending(s => s.SetOn).FirstOrDefault();
And invoke it like
var results = from example in repository.AsExpandable()
select new
{
Example = example,
LatestStatus = Newest.Invoke(example)
};
Related
I have several Stored Procedures which I use to search the database. Each returns several fields to do with the amount of data found/returned. The models look like this:
// simplified
public abstract class SearchResult {
public int RowCount { get; set; }
public int FilteredRowCount { get; set; }
// other properties
}
public class FooSearch {
public string Id { get; set; }
public string Foo { get; set; }
public SearchResult Result { get; set; }
}
Then in the DBContext I'm using OwnsOne() to link the classes, as per this MS guide
public DbSet<FooSearch> FooSearch { get; set; }
// OnModelCreating
builder.Entity<FooSearch>().OwnsOne(t => t.Result);
Finally I make the SP call:
var searchResult = db.FooSearch.FromSqlRaw("EXECUTE [Search].[Foo] {0}, {1}", foo, bar).ToList();
However this last step is giving me the following error:
System.InvalidOperationException: 'FromSqlRaw or FromSqlInterpolated was called with non-composable SQL and with a query composing over it. Consider calling AsEnumerable after the FromSqlRaw or FromSqlInterpolated method to perform the composition on the client side.'
Changing ToList() to AsEnumerable() makes no difference to the result.
If I remove the OwnsOne() and the Result property from the FooSearch class then the SP works and I get results. What is the cause of this issue, and how can I fix it?
I faced the same issue and fixed it after reading the issue #18232.
You need to add .IgnoreQueryFilters() after FromSqlRaw. Like that:
var searchResult = db.FooSearch
.FromSqlRaw("EXECUTE [Search].[Foo] {0}, {1}", foo, bar)
.IgnoreQueryFilters()
.ToList();
First Of all, you need to inform the model builder that your entity is a Keyless
entity
modelBuilder.Entity<FooSearch>().HasNoKey().ToView(null)
when using a keyless entity or executing RawSQL, EF will map all of the
properties mentioned in the query to your "FooSearch", how would ef map to
SearchResult object?
remove the SearchResult and replace it with fields you need to select.
I have a model with a linked list of foreign keys i.e.
[Table("a"]
public class A {
[Key]
[Column("a_id")]
public int Id { get; set; }
public List<B> Bs { get; set; } = new List<B>();
}
[Table("b"]
public class B {
[Key]
[Column("b_id")]
public int Id { get; set; }
[NotMapped]
public string MyFunctionValue { get; set; }
[ForeignKey("a_id")]
public A A { get; set; }
}
I've then defined a function which links to a scalar sql function like so...
public static class MySqlFunctions {
[DbFunction("MyFunction", "dbo")]
public static string MyFunction(int bId) {
throw new NotImplementedException();
}
}
and registered in my context like so...
modelBuilder.HasDbFunction(() => MySqlFunctions.MyFunction(default));
What I want to be able to do in my repository class is to grab the A records with the linked B records in a List with their MyFunctionValue value set to the return value of the function when ran against the id of B. Something like...
myContext.A
.Include(a => a.Bs.Select(b => new B {
Id = b.Id,
MyFunctionValue = MySqlFunctions.MyFunction(b.Id)
});
However with all the options I've tried so far I'm getting either a InvalidOperationException or NotImplementedException I guess because it can't properly convert it to SQL?
Is there any way I can write a query like this or is it too complex for EF to generate SQL for? I know there's a possibility I could use .FromSql but I'd rather avoid it if possible as it's a bit messy.
EDIT:
So I've managed to get it working with the following code but it's obviously a bit messy, if anyone has a better solution I'd be grateful.
myContext.A
.Include(a => a.Bs)
.Select(a => new {
A = a,
MyFunctionValues = a.Bs.Select(b => MySqlFunctions.MyFunction(b.Id))
})
.AsEnumerable()
.Select(aWithMfvs => {
for (int i = 0; i < aWithMfvs.MyFunctionValues.Count(); i++) {
aWithMfvs.A.Bs[i].MyFunctionValue = aWithMfvs.MyFunctionValues[i];
}
return aWithMfvs.A;
})
.AsQueryable();
There are several things you should consider with db functions:
When you declare a DbFunction as static method, you don't have to register it with the modelBuilder
Registering is only needed, when you would use Fluent API (which IMHO I recommend anyway in order to have you entities free of any dependencies)
The return value, the method name and the count, type and order of the method parameters must match your code in the user defined function (UDF)
You named the method parameter as bId. Is it exactly the same in your UDF or rather as in the table like b_id?
I have the following method which is meant to build me up a single object instance, where its properties are built via recursively calling the same method:
public ChannelObjectModel GetChannelObject(Guid id, Guid crmId)
{
var result = (from channelObject in _channelObjectRepository.Get(x => x.Id == id)
select new ChannelObjectModel
{
Id = channelObject.Id,
Name = channelObject.Name,
ChannelId = channelObject.ChannelId,
ParentObjectId = channelObject.ParentObjectId,
TypeId = channelObject.TypeId,
ChannelObjectType = channelObject.ChannelObjectTypeId.HasValue ? GetChannelObject(channelObject.ChannelObjectTypeId.Value, crmId) : null,
ChannelObjectSearchType = channelObject.ChannelObjectSearchTypeId.HasValue ? GetChannelObject(channelObject.ChannelObjectSearchTypeId.Value, crmId) : null,
ChannelObjectSupportingObject = channelObject.ChannelObjectSupportingObjectId.HasValue ? GetChannelObject(channelObject.ChannelObjectSupportingObjectId.Value, crmId) : null,
Mapping = _channelObjectMappingRepository.Get().Where(mapping => mapping.ChannelObjectId == channelObject.Id && mapping.CrmId == crmId).Select(mapping => new ChannelObjectMappingModel
{
CrmObjectId = mapping.CrmObjectId
}).ToList(),
Fields = _channelObjectRepository.Get().Where(x => x.ParentObjectId == id).Select(field => GetChannelObject(field.Id, crmId)).ToList()
}
);
return result.First();
}
public class ChannelObjectModel
{
public ChannelObjectModel()
{
Mapping = new List<ChannelObjectMappingModel>();
Fields = new List<ChannelObjectModel>();
}
public Guid Id { get; set; }
public Guid ChannelId { get; set; }
public string Name { get; set; }
public List<ChannelObjectMappingModel> Mapping { get; set; }
public int TypeId { get; set; }
public Guid? ParentObjectId { get; set; }
public ChannelObjectModel ParentObject { get; set; }
public List<ChannelObjectModel> Fields { get; set; }
public Guid? ChannelObjectTypeId { get; set; }
public ChannelObjectModel ChannelObjectType { get; set; }
public Guid? ChannelObjectSearchTypeId { get; set; }
public ChannelObjectModel ChannelObjectSearchType { get; set; }
public Guid? ChannelObjectSupportingObjectId { get; set; }
public ChannelObjectModel ChannelObjectSupportingObject { get; set; }
}
this is connecting to a SQL database using Entity Framework Core 2.1.1
Whilst it technically works, it causes loads of database queries to be made - I realise its because of the ToList() and First() etc. calls.
However because of the nature of the object, I can make one huge IQueryable<anonymous> object with a from.... select new {...} and call First on it, but the code was over 300 lines long going just 5 tiers deep in the hierarchy, so I am trying to replace it with something like the code above, which is much cleaner, albeit much slower..
ChannelObjectType, ChannelObjectSearchType, ChannelObjectSupportingObject
Are all ChannelObjectModel instances and Fields is a list of ChannelObjectModel instances.
The query takes about 30 seconds to execute currently, which is far too slow and it is on a small localhost database too, so it will only get worse with a larger number of db records, and generates a lot of database calls when I run it.
The 300+ lines code generates a lot less queries and is reasonably quick, but is obviously horrible, horrible code (which I didn't write!)
Can anyone suggest a way I can recursively build up an object in a similar way to the above method, but drastically cut the number of database calls so it's quicker?
I work with EF6, not Core, but as far as I know, same things apply here.
First of all, move this function to your repository, so that all calls share the DbContext instance.
Secondly, use Include on your DbSet on properties to eager load them:
ctx.DbSet<ChannelObjectModel>()
.Include(x => x.Fields)
.Include(x => x.Mapping)
.Include(x => x.ParentObject)
...
Good practice is to make this a function of context (or extension method) called for example BuildChannelObject() and it should return the IQueryable - just the includes.
Then you can start the recursive part:
public ChannelObjectModel GetChannelObjectModel(Guid id)
{
var set = ctx.BuildChannelObject(); // ctx is this
var channelModel = set.FirstOrDefault(x => x.Id == id); // this loads the first level
LoadRecursive(channelModel, set);
return channelModel;
}
private void LoadRecursive(ChannelObjectModel c, IQueryable<ChannelObjectModel> set)
{
if(c == null)
return; // recursion end condition
c.ParentObject = set.FirstOrDefault(x => x.Id == c?.ParentObject.Id);
// all other properties
LoadRecursive(c.ParentObject, set);
// all other properties
}
If all this code uses the same instance of DbContext, it should be quite fast. If not, you can use another trick:
ctx.DbSet<ChannelObjectModel>().BuildChannelObjectModel().Load();
This loads all objects to memory cache of your DbContext. Unfortunately, it dies with context instance, but it makes those recursive calls much faster, since no database trip is made.
If this is still to slow, you can add AsNoTracking() as last instruction of BuildChannelObjectModel().
If this is still to slow, just implement application wide memory cache of those objects and use that instead of querying database everytime - this works great if your app is a service that can have long startup, but then work fast.
Whole another approach is to enable lazy loading by marking navigation properties as virtual - but remember that returned type will be derived type anonymous proxy, not your original ChannelObjectModel! Also, properties will load only as long you don't dispose the context - after that you get an exception. To load all properties with the context and then return complete object is also a little bit tricky - easiest (but not the best!) way to do it to serialize the object to JSON (remember about circural references) before returning it.
If that does not satisfy you, switch to nHibernate which I hear has application wide cache by default.
I'm attempting to build an adapter to make using JQuery Datables with Entity Framework considerably simpler. The adapter is very similar to the one built by Telerik in their Kendo UI Extensions. I've managed to get most of the logic working, but the final piece that's giving me some trouble is getting the dynamically generated Linq to work.
I've looked into both LinqKit and Dynamic Expressions, and I'm a bit torn on how to approach this. I'm currently using the Dynamic Linq extension with limited success. It appears to work fine with Varchar and Int fields, but it stumbles with dates.
I'm using it like so:
public class Search
{
[DataMember(Name = "value")]
public string Value { get; set; }
[DataMember(Name = "regex")]
public string Regex { get; set; }
public string ToExpression(IList<Column> columns)
{
var list = columns.Select(column => $"{column.Data}.Contains.(#{columns.IndexOf(column)})").ToList();
return string.Join("or", list);
}
}
public class FilterExpression
{
public string Filter { get; }
public IEnumerable<object> Values { get; }
public FilterExpression(IEnumerable<Column> columns, Search search)
{
if (search.Value == null) return;
var list = columns.Where(n => n.Searchable).ToList();
Filter = ToExpression(list);
Values = list.Select(n => search.Value);
}
private static string ToExpression(IEnumerable<Column> columns)
{
var colList = columns.Where(n => n.Searchable).ToList();
var list = colList.Select(column => $"{column.Data}.ToString().Contains(#{colList.IndexOf(column)})").ToList();
return string.Join(" or ", list);
}
}
public static IQueryable Where(this IQueryable source, FilterExpression filterExpression)
{
return filterExpression?.Values == null ? source : source.Where(filterExpression.Filter, filterExpression.Values.ToArray());
}
The above works for most cases, but again Dates are a bit of a problem. The goal is to prevent developers from having to manually write their Linq Where statements and instead allow the adapter to simply generate it.
Again, the above works for both INT fields and VARCHAR, but if I include a DATE field in the model, the error I get is: System.NotSupportedException: 'LINQ to Entities does not recognize the method 'System.String ToString()' method, and this method cannot be translated into a store expression.'
https://www.c-sharpcorner.com/blogs/exception-linq-to-entities-does-not-recognize-the-method-tostring
Check this one out.
Bascially when you work with linq to entites, it creats a query which is fired on sql server. ToString() works with in memory objects.
If the object was in memory ,it would have worked just fine. Please go through the link.
I am using Entity Framework CodeFirst where I have used Parent Child relations using ICollection as
public class Person
{
public string UserName { get;set}
public ICollection<Blog> Blogs { get; set;}
}
public class Blog
{
public int id { get; set; }
public string Subject { get; set; }
public string Body { get; set; }
}
Ok, so far everything is working ok, but my concern is, whenever I want to get the Blogs of a person, I get it as
var thePerson = _context.Persons.Where(x => x.UserName = 'xxx').SingleOrDefault();
var theBlogs = thePerson.Blogs.OrderBy(id).Take(5);
Now, I understand that, when the line is executed, all Blogs for that person is loaded into the memory and then sorting and selecting is done from memory. That is not ideal for a record of Person who has large number of blogs. I want to make the Blog Child as IQueryable so that the Sorting and Selecting is done in SQL database before pulling to Memory.
I know I could declare the Blogs as IQueryable in my context so that I could directly query as
var theBlogs = _context.Blogs.Where(.....)
but that is not feasible for me due to design choice, I want to avoid any circular reference as much as possible due to serialization problem. So, I did not make any reference of the parent entity in my child.
I found that, i can call AsQueryable() method on the blogs as
var theBlogs = thePerson.Blogs.AsQueryable().OrderBy(id).Take(5);
That looks like a magic for me and seems too good to be true. So my question. Does this AsQueryable really make the ICollection as IQueryable in reality and makes all Query process in SQL Server (Lazy loading) OR it is just a casting where Blogs are loaded into memory as like before, but change the interface from ICollection to IQueryable ?
So actually it appears that writing your navigation property as IQueryable<T> is not possible.
What you could do is adding a navigation property to Blog:
public class Blog
{
public int id { get; set; }
public string Subject { get; set; }
public string Body { get; set; }
public virtual Person Owner { get; set; }
}
From that, you can query as follows so it won't load everything into memory:
var thePerson = _context.Persons.Where(x => x.UserName = 'xxx').SingleOrDefault();
var results = _context.Blogs.Where(z => z.Person.Name = thePerson.Name).OrderBy(id).Take(5)
I suggest you to try LINQPad to see how LINQ is translated into SQL, and what is actually requested from the DB.
A better approach is described in Ladislav's answer. In your case:
var theBlogs = _context.Entry(thePerson)
.Collection(x => x.Blogs)
.Query()
.OrderBy(x => x.id)
.Take(5);