We Keep Coding

How could I make this EF Core query better?

I need to fetch from the database this:
rack
it's type
single shelf with all its boxes and their box types
single shelf above the previous shelf without boxes and with shelf type
Shelves have VerticalPosition which is in centimeters from the ground - when I am querying for e.g. second shelf in rack, I need to order them and select shelf on index 1.
I have this ugly EF query now:
var targetShelf = await _warehouseContext.Shelves
.Include(s => s.Rack)
.ThenInclude(r => r.Shelves)
.ThenInclude(s => s.Type)
.Include(s => s.Rack)
.ThenInclude(r => r.Type)
.Include(s => s.Rack)
.ThenInclude(r => r.Shelves)
.Include(s => s.Boxes)
.ThenInclude(b => b.BoxType)
.Where(s => s.Rack.Aisle.Room.Number == targetPosition.Room)
.Where(s => s.Rack.Aisle.Letter == targetPosition.Aisle)
.Where(s => s.Rack.Position == targetPosition.Rack)
.OrderBy(s => s.VerticalPosition)
.Skip(targetPosition.ShelfNumber - 1)
.FirstOrDefaultAsync();
but this gets all boxes from all shelves and it also shows warning
Compiling a query which loads related collections for more than one collection navigation, either via 'Include' or through projection, but no 'QuerySplittingBehavior' has been configured. By default, Entity Framework will use 'QuerySplittingBehavior.SingleQuery', which can potentially result in slow query performance.
Also I would like to use AsNoTracking(), because I don't need change tracker for these data.
First thing: for AsNoTracking() I would need to query Racks, because it complains about circular include.
Second thing: I tried conditional include like this:
.Include(r => r.Shelves)
.ThenInclude(s => s.Boxes.Where(b => b.ShelfId == b.Shelf.Rack.Shelves.OrderBy(sh => sh.VerticalPosition).Skip(shelfNumberFromGround - 1).First().Id))
but this won't even translate to SQL.
I have also thought of two queries - one will retrieve rack with shelves and second only boxes, but I still wonder if there is some single call command for this.
Entities:
public class Rack
{
public Guid Id { get; set; }
public Guid RackTypeId { get; set; }
public RackType Type { get; set; }
public ICollection<Shelf> Shelves { get; set; }
}
public class RackType
{
public Guid Id { get; set; }
public ICollection<Rack> Racks { get; set; }
}
public class Shelf
{
public Guid Id { get; set; }
public Guid ShelfTypeId { get; set; }
public Guid RackId { get; set; }
public int VerticalPosition { get; set; }
public ShelfType Type { get; set; }
public Rack Rack { get; set; }
public ICollection<Box> Boxes { get; set; }
}
public class ShelfType
{
public Guid Id { get; set; }
public ICollection<Shelf> Shelves { get; set; }
}
public class Box
{
public Guid Id { get; set; }
public Guid ShelfId { get; set; }
public Guid BoxTypeId { get; set; }
public BoxType BoxType { get; set; }
public Shelf Shelf { get; set; }
}
public class BoxType
{
public Guid Id { get; set; }
public ICollection<Box> Boxes { get; set; }
}
I hope I explained it good enough.

Query Splitting
First, I'd recommend benchmarking the query as-is before deciding whether to attempt any optimization.
It can be faster to perform multiple queries than one large query with many joins. While you avoid a single complex query, you have additional network round-trips if your DB isn't on the same machine, and some databases (e.g. SQL Server without MARS enabled) only support one active query at a time. Your mileage may vary in terms of actual performance.
Databases do not generally guarantee consistency between separate queries (SQL Server allows you to mitigate that with the performance-expensive options of serializable or snapshot transactions). You should be cautious using a multiple-query strategy if intervening data modifications are possible.
To split a specific query, use the AsSplitQuery() extension method.
To use split queries for all queries against a given DB context,
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder
.UseSqlServer(
#"Server=(localdb)\mssqllocaldb;Database=EFQuerying;Trusted_Connection=True;ConnectRetryCount=0",
o => o.UseQuerySplittingBehavior(QuerySplittingBehavior.SplitQuery));
}
Reference.
Query that won't translate
.Include(r => r.Shelves)
.ThenInclude(s => s.Boxes.Where(b => b.ShelfId == b.Shelf.Rack.Shelves.OrderBy(sh => sh.VerticalPosition).Skip(shelfNumberFromGround - 1).First().Id))
Your expression
s.Boxes.Where(b => b.ShelfId == b.Shelf.Rack.Shelves.OrderBy(sh => sh.VerticalPosition).Skip(shelfNumberFromGround - 1).First().Id
resolves to an Id. ThenInclude() expects an expression that ultimately specifies a collection navigation (in other words, a table).

Ok, from your question I'm assuming you have a method where you need these bits of information:
single shelf with all its boxes and their box types
single shelf above the previous shelf without boxes and with shelf type
rack and it's type
Whether EF breaks up the queries or you do doesn't really make much of a difference performance-wise. What matters is how well the code is later understood and can adapt if/when requirements change.
The first step I would recommend is to identify the scope of detail you actually need. You mention that you don't need tracking, so I would expect you intend to deliver these results or otherwise consume the information without persisting changes. Project this down to just the details from the various tables that you need to be served by a DTO or ViewModel, or an anonymous type if the data doesn't really need to travel. For instance you will have a shelf & shelf type which is effectively a many-to-one so the shelf type details can probably be part of the shelf results. Same with the Box and BoxType details. A shelf would then have an optional set of applicable box details. The Rack & Racktype details can come back with one of the shelf queries.
[Serializable]
public class RackDTO
{
public int RackId { get; set; }
public int RackTypeId { get; set; }
public string RackTypeName { get; set; }
}
[Serializable]
public class ShelfDTO
{
public int ShelfId { get; set; }
public int VerticalPosition { get; set; }
public int ShelfTypeId { get; set; }
public string ShelfTypeName { get; set; }
public ICollection<BoxDTO> Boxes { get; set; } = new List<BoxDTO>();
public RackDTO Rack { get; set; }
}
[Serializable]
public class BoxDTO
{
public int BoxId { get; set; }
public int BoxTypeId { get; set; }
public string BoxTypeName { get; set; }
}
Then when reading the information, I'd probably split it into two queries. One to get the "main" shelf, then a second optional one to get the "previous" one if applicable.
ShelfDTO shelf = await _warehouseContext.Shelves
.Where(s => s.Rack.Aisle.Room.Number == targetPosition.Room
&& s.Rack.Aisle.Letter == targetPosition.Aisle
&& s.Rack.Position == targetPosition.Rack)
.Select(s => new ShelfDTO
{
ShelfId = s.ShelfId,
VerticalPosition = s.VerticalPosition,
ShelfTypeId = s.ShelfType.ShelfTypeId,
ShelfTypeName = s.ShelfType.Name,
Rack = s.Rack.Select(r => new RackDTO
{
RackId = r.RackId,
RackTypeId = r.RackType.RackTypeId,
RackTypeName = r.RackType.Name
}).Single(),
Boxes = s.Boxes.Select(b => new BoxDTO
{
BoxId = b.BoxId,
BoxTypeId = b.BoxType.BoxTypeId,
BoxTypeName = b.BoxType.Name
}).ToList()
}).OrderBy(s => s.VerticalPosition)
.Skip(targetPosition.ShelfNumber - 1)
.FirstOrDefaultAsync();
ShelfDTO previousShelf = null;
if (targetPosition.ShelfNumber > 1 && shelf != null)
{
previousShelf = await _warehouseContext.Shelves
.Where(s => s.Rack.RackId == shelf.RackId
&& s.VerticalPosition < shelf.VerticalPosition)
.Select(s => new ShelfDTO
{
ShelfId = s.ShelfId,
VerticalPosition = s.VerticalPosition,
ShelfTypeId = s.ShelfType.ShelfTypeId,
ShelfTypeName = s.ShelfType.Name,
Rack = s.Rack.Select(r => new RackDTO
{
RackId = r.RackId,
RackTypeId = r.RackType.RackTypeId,
RackTypeName = r.RackType.Name
}).Single()
}).OrderByDescending(s => s.VerticalPosition)
.FirstOrDefaultAsync();
}
Two fairly simple to read queries that should return what you need without much problem. Because we project down to a DTO we don't need to worry about eager loading and potential cyclical references if we wanted to load an entire detached graph. Obviously this would need to be fleshed out to include the details from the shelf, box, and rack that are relevant to the consuming code/view. This can be trimmed down even more by leveraging Automapper and it's ProjectTo method to take the place of that whole Select projection as a one-liner.

In SQL raw it could look like
WITH x AS(
SELECT
r.*, s.Id as ShelfId, s.Type as ShelfType
ROW_NUMBER() OVER(ORDER BY s.verticalposition) as shelfnum
FROM
rooms
JOIN aisles on aisles.RoomId = rooms.Id
JOIN racks r on r.AisleId = aisles.Id
JOIN shelves s ON s.RackId = r.Id
WHERE
rooms.Number = #roomnum AND
aisles.Letter = #let AND
r.Position = #pos
)
SELECT *
FROM
x
LEFT JOIN boxes b
ON
b.ShelfId = x.ShelfId AND x.ShelfNum = #shelfnum
WHERE
x.ShelfNum BETWEEN #shelfnum AND #shelfnum+1
The WITH uses room/aisle/rack joins to locate the rack; you seem to have these identifiers. Shelves are numbered in increasing height off ground. Outside the WITH, boxes are left joined only if they are on the shelf you want, but two shelves are returned; the shelf you want with all it's boxes and the shelf above but box data will be null because the left join fails

As an opinion, if your query is getting this level of depth, you might want to consider either using views as a shortcut in your database or use No-SQL as a read store.
Having to do lots of joins, and doing taxing operations like order by during runtime with LINQ is something I'd try my best to avoid.
So I'd approach this as a design problem, rather than a code/query problem.

In EF, All related entities loaded with Include, ThenInclude etc. produce joins on the database end. This means that when we load related master tables, the list values will get duplicated across all records, thus causing what is called "cartesian explosion". Due to this, there was a need to split huge queries into multiple calls, and eventually .AsSplitQuery() was introduced.
Eg:
var query = Context.DataSet<Transactions>()
.Include(x => x.Master1)
.Include(x => x.Master2)
.Include(x => x.Master3)
.ThenInclude(x => x.Master3.Masterx)
.Where(expression).ToListAsync();
Here we can introduce splitquery
var query = Context.DataSet<Transactions>()
.Include(x => x.Master1)
.Include(x => x.Master2)
.Include(x => x.Master3)
.ThenInclude(x => x.Master3.Masterx)
.Where(expression).AsSplitQuery.ToListAsync();
As an alternate to include this to all existing queries, which could be time consuming, we could specify this globally like
services.AddDbContextPool<EntityDataLayer.ApplicationDbContext>(options =>
{
options.EnableSensitiveDataLogging(true);
options.UseMySql(mySqlConnectionStr,
ServerVersion.AutoDetect(mySqlConnectionStr), x =>
x.UseQuerySplittingBehavior(QuerySplittingBehavior.SplitQuery)
x.EnableRetryOnFailure(
maxRetryCount: 10,
maxRetryDelay: TimeSpan.FromSeconds(30),
errorNumbersToAdd: null));
});
This will ensure that all queries are called as split queries.
Now in case we need single query, we can just override this by stating single query explicitly in individual queries. This may be done vice-versa though.
var data = await query.AsSingleQuery().ToListAsync();

How to Use Include More Than Once Ef Core? [duplicate]

Suppose I have a entity class with one to one relationship as below:
public class Transaction
{
public int TransactionID { get; set; }
public Double Amount { get; set; }
public int TransactionDetailID { get; set; }
public virtual TransactionDetail TransactionDetailFk { get; set; }
}
public class TransactionDetail
{
public int TransactionDetailID { get; set; }
public DateTime PostedDate { get; set; }
public int TransactionTypeID { get; set; }
public int TransactionCategoryID { get; set; }
public int PaymentMethodID { get; set; }
public int PaymentPayorID { get; set; }
public virtual TransactionType TransactionTypeFk { get; set; }
public virtual TransactionCategory TransactionCategoryFk { get; set; }
public virtual PaymentMethod PaymentMethodFk { get; set; }
public virtual PaymentPayor PaymentPayorFk { get; set; }
public virtual Transaction TransactionFk { get; set; }
}
Now I would like to get a Transaction Object based on TransactionID, in addition I would like to get all my related object from Transaction to TransactionDetail
to (TransactionType/TransactionCategory/PaymentMethod/PaymentPayor), which is a two-level data mapping and my function would be like:
public async Task<Transaction> GetSingleFullTransactionByIDAsync(int transactionID)
=> await GetSingleOrDefaultAsync(
predicate: tr => (tr.TransactionID == transactionID),
include: (obj => (
obj
.Include(entity => entity.TransactionDetailFk)
.ThenInclude(td => td.PaymentPayorFk)
.Include(entity => entity.TransactionDetailFk)
.ThenInclude(td => td.PaymentMethodFk)
.Include(entity => entity.TransactionDetailFk)
.ThenInclude(td => td.TransactionTypeFk)
.Include(entity => entity.TransactionDetailFk)
.ThenInclude(td => td.TransactionCategoryFk)
))
);
I feel my code is not that clean and tidy, since for each related entity of Transaction Detail, I am actually including multiple instance of transaction detail...I would like to do something like below which only include one instance of Transaction Detail but entity framework does not allow me to do that:
public async Task<Transaction> GetSingleFullTransactionByIDAsync(int transactionID)
=> await GetSingleOrDefaultAsync(
predicate: tr => (tr.TransactionID == transactionID),
include: (obj => (
obj
.Include(entity => entity.TransactionDetailFk)
.ThenInclude(td => td.PaymentPayorFk)
.ThenInclude(td => td.PaymentMethodFk)
.ThenInclude(td => td.TransactionTypeFk)
.ThenInclude(td => td.TransactionCategoryFk)
))
);
So what would be the efficient way for doing that using EF core? Note here I am using repository pattern so I can't use sql-to-linq expression, but only have to use "Include"/"ThenInclude" operation...

I feel my code is not that clean and tidy, since for each related entity of Transaction Detail, I am actually including multiple instance of transaction detail...
This is exactly the intended ("by design")way of including multiple related entities in EF Core. It is explained (with example) in the Loading Related Data - Including multiple levels section of the EF Core documentation:
You may want to include multiple related entities for one of the entities that is being included. For example, when querying Blogs, you include Posts and then want to include both the Author and Tags of the Posts. To do this, you need to specify each include path starting at the root. For example, Blog -> Posts -> Author and Blog -> Posts -> Tags. This does not mean you will get redundant joins, in most cases EF will consolidate the joins when generating SQL.
Note the last paragraph. To recap, every Include / ThenInclude chain represents an entity path to be loaded. Each entity contained in the include paths is included just once.

Entity Framework Core many-to-many navigation issues

Entity Framework Core has yet to implement many-to-many relationships, as tracked in GitHub issue #1368; however, when I follow the navigation examples in that issue or similar answers here at Stack Overflow, my enumeration fails to yield results.
I have a many-to-many relationship between Photos and Tags.
After implementing the join table, examples show I should be able to:
var tags = photo.PhotoTags.Select(p => p.Tag);
While that yields no results, I am able to to load via:
var tags = _context.Photos
.Where(p => p.Id == 1)
.SelectMany(p => p.PhotoTags)
.Select(j => j.Tag)
.ToList();
Relevant code:
public class Photo
{
public int Id { get; set; }
public virtual ICollection<PhotoTag> PhotoTags { get; set; }
}
public class Tag
{
public int Id { get; set; }
public virtual ICollection<PhotoTag> PhotoTags { get; set; }
}
public class PhotoTag
{
public int PhotoId { get; set; }
public Photo Photo { get; set; }
public int TagId { get; set; }
public Tag Tag { get; set; }
}
protected override void OnModelCreating(ModelBuilder builder)
{
base.OnModelCreating(builder);
builder.Entity<PhotoTag>().HasKey(x => new { x.PhotoId, x.TagId });
}
What am I missing from other examples?

In fact this is not a specific for many-to-many relationship, but in general to the lack of lazy loading support in EF Core. So in order to have Tag property populated, it has to be eager (or explicitly) loaded. All this is (sort of) explained in the Loading Related Data section of the EF Core documentation. If you take a look at Including multiple levels section, you'll see the following explanation
You can drill down thru relationships to include multiple levels of related data using the ThenInclude method. The following example loads all blogs, their related posts, and the author of each post.
and example for loading the Post.Author which is pretty much the same as yours:
using (var context = new BloggingContext())
{
var blogs = context.Blogs
.Include(blog => blog.Posts)
.ThenInclude(post => post.Author)
.ToList();
}
So to make this working
var tags = photo.PhotoTags.Select(p => p.Tag);
the photo variable should have been be retrieved using something like this:
var photo = _context.Photos
.Include(e => e.PhotoTags)
.ThenInclude(e => e.Tag)
.FirstOrDefault(e => e.Id == 1);

query multi-level entity with filter at the lowest level

So I have 3 entity classes:
public partial class Event
{
public Event()
{
Recurrences = new HashSet<Recurrence>();
}
public int Id { get; set; }
public ICollection<Recurrence> Recurrences { get; set; }
}
public partial class Recurrence
{
public Recurrence()
{
AspNetUsers = new HashSet<AspNetUser>();
}
public int Id { get; set; }
public int EventId { get; set; }
public ICollection<AspNetUser> AspNetUsers { get; set; }
}
public partial class AspNetUser
{
public AspNetUser()
{
Recurrences = new HashSet<Recurrence>();
}
public string Id { get; set; }
public string UserName { get; set; }
public ICollection<Recurrence> Recurrences { get; set; }
}
I would like to get the event given the aspnetuser.id using line to entity. so far this is what I have but it's returning an error:
// GET: api/Events?userId={userId}
public IQueryable<Event> GetEvents(string userId)
{
return db.Events
.Include(e => e.Recurrences
.Select(u => u.AspNetUsers.Where(i => i.Id == userId)));
}
When I exclude the where clause it works fine. Please help.
Thanks in advance!

I don't think Include() means what you think it means. (https://msdn.microsoft.com/en-us/library/bb738708%28v=vs.110%29.aspx) What it does is tell the db set to be sure to bring in relationships for that object. By default (last I checked), the db context will auto pull in all relationships, so this isn't necessary. However, if you've turned off the lazy-loading (http://www.entityframeworktutorial.net/EntityFramework4.3/lazy-loading-with-dbcontext.aspx) then you'll need to .Include() all the relationships you want to have in the query.
This should solve your problem. I don't guarantee the SQL generated won't be silly, though.
If you have lazy-loading turned on:
db.Events.Include("Recurrences").Include("Recurrences.AspNetUsers")
.Where(e => e.Recurrences
.Any(r => r.AspNetUsers
.Any(u => u.Id ==userId)));
If you have lazy-loading turned off:
db.Events
.Where(e => e.Recurrences
.Any(r => r.AspNetUsers
.Any(u => u.Id ==userId)));
Also, if you have trouble seeing errors, you can .ToList() the query before returning so that it fails in your code and not deep inside the Web API stack. Personally, I like to do this so that I can try/catch the query and handle it properly.

One-to-One Relationship Not Loading All Entities

I have a (contrived and over-simplified, due to contractual constraints) data model that should look like this:
public class ProvisionalData
{
public int ProvisionalDataID { get; set; }
public string Data { get; set; }
public Lot Lot { get; set; }
}
public class Destination
{
public int DestinationID { get; set; }
public string Name { get; set; }
}
public class LotDestination
{
public int LotDestinationID { get; set; }
public int DestinationID { get; set; }
public DateTime Month { get; set; }
public Destination Destination { get; set; }
}
public class Lot
{
public int LotID { get; set; }
public int ProvisionalDataID { get; set; }
public int LotDestinationID { get; set; }
public ProvisionalData ProvisionalData { get; set; }
public LotDestination LotDestination { get; set; }
}
The relationship from Lot to ProvisionalData is a required one-to-one on both sides. Please note, this is not the whole model, nor is the domain the concern. The concern is with configuring the one-to-one relationship.
My relevant to the one-to-one mapping fluent configurations:
public class LotConfig : EntityTypeConfiguration<Lot>
{
public LotConfig()
{
ToTable("Lot");
HasKey(x => x.LotID);
HasRequired(x => x.ProvisionalData)
.WithRequiredDependent(x => x.Lot)
.WillCascadeOnDelete(true);
}
}
public class ProvisionalDataConfig : EntityTypeConfiguration<ProvisionalData>
{
public ProvisionalDataConfig()
{
ToTable("ProvisionalData");
HasKey(x => x.ProvisionalDataID);
}
}
The other relationships shown are in fact set up - I have verified that they are being configured in my context, and all IDbSets exist and function correctly. In fact, everything "works" with this setup, except that some ProvisionalData entities on navigation properties are not populated by the following query:
var lotDestination = db.lotDestinations
.Where(x => x.DestinationId == destinationId && x.Month == month)
.Include(x => x.Lots)
.Include("Lots.ProvisionalData")
.Include(x => x.Destination)
.SingleOrDefault();
In my real dataset, this query will return a destination with 30 Lots. 16 of those lots have their ProvisionalData navigation property loaded. 14 do not. This problem persists when I manually loop through each Lot and db.Entry(lot).Reference(ProvisionalData).Load(). When I examine those entries, all 30 return true for .IsLoaded. The query and .Includes appears to be doing what they are supposed to, but some of the entities aren't coming back for reasons I don't understand. I'm hoping it's something simple that I can't see because I've been staring at it for too long.
However, when I change the relationship (ignoring existing database constraints) to a one-to-many with ProvisionalData entities looking like this:
public class ProvisionalData
{
public int ProvisionalDataID { get; set; }
public string Data { get; set; }
public IList<Lot> Lots { get; set; }
}
and a new Lot configuration like this:
public class LotConfig : EntityTypeConfiguration<Lot>
{
public LotConfig()
{
ToTable("Lot");
HasKey(x => x.LotID);
HasRequired(x => x.ProvisionalData)
.WithMany(x => x.Lots)
.HasForeignKey(x => x.ProvisionalDataID);
}
}
everything works flawlessly. The only drawback here is that this doesn't reflect the true constraints in the database, so you could technically try to add multiple Lots to the same piece of ProvisionalData, which would break when trying to save. I can build the logic in to prevent this myself, but why can't I express it here in Entity Framework? Is my configuration incorrect?
Also interestingly, when I switch the above-mentioned query around to this dumb version to test things (with the one-to-one mapping still in place in EF):
var quota = db.Lots
.Where(l => l.LotDestination.DestinationID == destinationId && l.LotDestination.Month == m)
.Include(x => x.ProvisionalData)
.Include(x => x.LotDestination)
.Include(x => x.LotDestination.Destination)
.Select(x => x.LotDestination)
.FirstOrDefault();
all of the provisional data comes back, but some of the Destinations do not. This hints to me that it has something to do with including navigation properties multiple levels deep across a one-to-one. Has anyone else experienced this behavior before?

EF does not officially support 1:1 associations other than in shared primary key 1:1 association situations.
What you are doing is create 1:many's and trying to tell EF that it's really a 1:1. The problem is, the db schema is really a 1:many schema and EF will have problems here.
If your requirement is 1:1, then you need to use a shared primary key (both entities have the same primary key), and one also treats it as a foreign key).

I am facing the problems before, Mine solution was using 1:many because if POCOA include POCOB
.Include(x => x.POCOA.POCOB)
really return the POCOB as a list, since we know it 100% return one record, then during getting the data, we can say a.SingleOrDefault();