We Keep Coding

Is there a way to include certain fields of a class in linq?

My class structure is as follows. I'm trying to include the User field for the Order result. But I don't want to get the Orders property of the User class.
public class Order{
public int OrderId { get; set; }
public virtual User User { get; set; }
}
public class User{
public int UserId {get; set;}
public string Name {get; set;}
public string SurName {get; set;}
public virtual ICollection<Order> Orders { get; set; }
}
I wrote this code.
var orders = context.Set<Order>()
.Include(t => new { Name = t.User.Name, Surname = t.User.SurName })
.ToList();
But I get an error that
The expression 'new <>f__AnonymousType20`2(Name = (t As Order).User.Name, Surname = (t As Order).User.SurName)' is invalid inside an 'Include' operation, since it does not represent a property access: 't => t.MyProperty'. To target navigations declared on derived types, use casting ('t => ((Derived)t).MyProperty') or the 'as' operator ('t => (t as Derived).MyProperty'). Collection navigation access can be filtered by composing Where, OrderBy(Descending), ThenBy(Descending), Skip or Take operations. For more information on including related data, see http://go.microsoft.com/fwlink/?LinkID=746393."

I'm trying to include the User field for the Order result.
So you want to query a sequence of Orders, every Order with several properties of the User of this Order.
var orders = dbContext.Orders
.Where(order => ...) // if you don't want all Orders
.Select(order => new
{
// Select only the Order properties that you plan to use
Id = order.Id,
Total = order.Total,
...
// The User of this Order
User = new
{
// Select only the User properties that you plan to use
Id = order.User.Id,
Name = order.User.Name,
...
},
});
I use anonymous type here. If you need to return the fetched data from a procedure you need to put the data in a predefined class.
Some versions of entity framework won't support using the virtual properties. In that case you'll have to do the (Group-)Join yourself. If you start at the "one" side in a one-to-many relation, do a GroupJoin; if you start at the "many" side, a standard join will suffice. Use the overload that has a parameter resultSelector to precisely request what you want.
var orders = dbContext.Orders
.Where(order => ...)
.Join(dbContext.Users,
order => order.UserId, // from every Order take the foreign key to the user
user => user.UserId, // from every User take the primary key
// Parameter resultSelector: get every Order with its one and only user
// to make one new
(order, userOfThisOrder) => new
{
Id = order.Id,
TotalPrice = order.TotalPrice,
User = new
{
Id = userOfThisOrder.UserId,
Name = userOfThisOrder.Name,
...
},
});
Why not use include?
Include will fetch the complete row of the table, inclusive the properties that you won't use, or properties of which you already know the value.
Suppose you have a School database with Schools and Students, and a one-to-many relation with a foreign key: every School has zero or more Students, every Student attends exactly one School, namely the School that the foreign key SchoolId refers to.
Now if you fetch School [10], and use Include to fetch all its 2000 Students, then you'll fetch the foreign key SchoolId once for every fetched Student. You already know that this foreign key will have a value 10. You will be transferring this value over 2000 times. What a waste of processing power.
Another reason not to fetch complete rows, and not use Include is the following. The DbContext holds a ChangeTracker. When you fetch a complete row, the data is stored in the ChangeTracker, together with a Clone. You get the reference to the Original. Whenever you change values of properties of the retrieved data, you change the values in the Original. If you update the changed data using SaveChanges, then the original is compared by value with the clone in the ChangeTracker. Only the changed properties will be sent to the database.
So if you fetch School [10] with all its 2000 Students, and you don't plan to update the fetched data, then fetching the complete rows will fetch one School and 2000 Students. Every fetched item will be copied. And if you later fetch Student [42] to change his Address, and call SaveChanges, all fetched 2000 Students will be compared with their Clones, value by value. What a waste of processing power, if you didn't plan to update any of these 2000 Students.
When using entity framework always use Select and select only the properties that you actually plan to use. Only fetch complete rows, only use Include, if you plan to update the fetched data.

Select() decline in performance

I'm working on small app which is written in c# .net core and I'm populating one prop in a code because that information is not available in database, code looks like this:
public async Task<IEnumerable<ProductDTO>> GetData(Request request)
{
IQueryable<Product> query = _context.Products;
var products = await query.ToListAsync();
// WARNING - THIS SOLUTION LOOKS EXPENCIVE TO ME!
return MapDataAsDTO(products).Select(c =>
{
c.HasBrandStock = products.Any(cc => cc.ParentProductId == c.Id);
return c;
});
}
}
private IEnumerable<ProductDTO> MapDataAsDTO(IEnumerable<Product> products)
{
return products.Select(p => MapData(p)).ToList();
}
What is bothering me here is this code:
return MapDataAsDTO(products).Select(c =>
{
c.HasBrandStock = data.Any(cc => cc.ParentProductId == c.Id);
return c;
});
}
I've tested it on like 300k rows and it seems slow, I'm wondering is there a better solutions in this situations?
Thanks guys!
Cheers

First up, this method is loading all products, and generally that is a bad idea unless you are guaranteeing that the total number of records will remain reasonable, and the total size of those records will be reasonable. If the system can grow, add support for server-side pagination now. (Page # and Page size, leveraging Skip & Take) 300k products is not a reasonable number to be loading all data in one hit. Any way you skin this cat it will be slow, expensive, and error prone due to server load without paging. One user making a request on the server will need to have the DB server allocate for and load up 300k rows, transmit that data over the wire to the app server, which will allocate memory for those 300k rows, then transmit that data over the wire to the client who literally does not need those 300k rows at once. What do you think happens when 10 users hit this page? 100? And what happens when it's "to slow" and they start hammering the F5 key a few times. >:)
Second, async is not a silver bullet. It doesn't make queries faster, it actually makes them a bit slower. What it does do is allow your web server to be more responsive to other requests while those slower queries are running. Default to synchronous queries, get them running as efficiently as possible, then for the larger ones that are justified, switch them to asynchronous. MS made async extremely easy to implement, perhaps too easy to treat as a default. Keep it simple and synchronous to start, then re-factor methods to async as needed.
From what I can see you want to load all products into DTOs, and for products that are recognized as being a "parent" of at least one other product, you want to set their DTO's HasBrandStock to True. So given product IDs 1 and 2, where 2's parent ID is 1, the DTO for Product ID 1 would have a HasBrandStock True while Product ID 2 would have HasBrandStock = False.
One option would be to tackle this operation in 2 queries:
var parentProductIds = _context.Products
.Where(x => x.ParentProductId != null)
.Select(x => x.ParentProductId)
.Distinct()
.ToList();
var dtos = _context.Products
.Select(x => new ProductDTO
{
ProductId = x.ProductId,
ProductName = x.ProductName,
// ...
HasBrandStock = parentProductIds.Contains(x.ProductId)
}).ToList();
I'm using a manual Select here because I don't know what your MapAsDto method is actually doing. I'd highly recommend using Automapper and it's ProjectTo<T> method if you want to simplify the mapping code. Custom mapping functions can too easily hide expensive bugs like ToList calls when someone hits a scenario that EF cannot translate.
The first query gets a distinct list of just the Product IDs that are the parent ID of at least one other product. The second query maps out all products into DTOs, setting the HasBrandStock based on whether each product appears in the parentProductIds list or not.
This option will work if a relatively limited number of products are recognized as "parents". That first list can only get so big before it risks crapping out being too many items to translate into an IN clause.
The better option would be to look at your mapping. You have a ParentProductId, does a product entity have an associated ChildProducts collection?
public class Product
{
public int ProductId { get; set; }
public string ProductName { get; set; }
// ...
public virtual Product ParentProduct { get; set; }
public virtual ICollection<Product> ChildProducts { get; set; } = new List<Product>();
}
public class ProductConfiguration : EntityTypeConfiguration<Product>
{
public ProductConfiguration()
{
HasKey(x => x.ProductId);
HasOptional(x => x.ParentProduct)
.WithMany(x => x.ChildProducts)
.Map(x => x.MapKey("ParentProductId"));
}
}
This example maps the ParentProductId without exposing a field in the entity (recommended). Otherwise, if you do expose a ParentProductId, substitute the .Map(...) call with .HasForeignKey(x => x.ParentProductId).
This assumes EF6 as per your tags, if you're using EF Core then you use HasForeignKey("ParentProductId") in place of Map(...) to establish a shadow property for the FK without exposing a property. The entity configuration is a bit different with Core.
This allows your queries to leverage the relationship between parent products and any related children products. Populating the DTOs can be accomplished with one query:
var dtos = _context.Products
.Select(x => new ProductDTO
{
ProductId = x.ProductId,
ProductName = x.ProductName,
// ...
HasBrandStock = x.ChildProducts.Any()
}).ToList();
This leverages the relationship to populate your DTO and it's flag in one pass. The caveat here is that there is now a cyclical relationship between product and itself represented in the entity. This means don't feed entities to something like a serializer. That includes avoiding adding entities as members of DTOs/ViewModels.

What is the best way to optimize a nested where clause that has multiple conditions?

I'm trying to find a list of ReturnItems where the quantity of a single item being returned is more than the original ordered quantity for that item. So there are 2 different lists of objects in play here - IEnumerable<ReturnItem> and IEnumerable<OrderItem>. The problem is that depending on the source for making the return (there are multiple places in our workflow where a return can be made) the ItemNumber on a given ReturnItem may be null. In this case we would need to rely on the ReturnItem.OrderItemId to match it to an OrderItem.
I have solved the problem using LINQ, but it requires a nested for loop (under the hood) so I'm trying to avoid that if possible while also maintain readability.In other words, I want to avoid a run time of O(N^2) and look for O(N) or better but again, while maintaining readability (I know I'm asking for a lot here but I figured I'd see if anyone has a creative solution). I created a solution where I have two dictionaries for the order items. One of them, the key is the item number and the other's key is the order Item Id. This works and solves the performance problem, but I completely loose the readability.
Here is the original LINQ statement I had:
// ItemsForReturn = IEnumerable<ReturnItem>
// OrderItems = IEnumerable<OrderItem>
var invalidQuantityItems = message.ItemsForReturn.Where(returnItem =>
{
var matchingOrderItemQuantity = message.OrderItems
.Where(orderItem => orderItem.ItemNumber.Equals(returnItem.ItemNumber) || orderItem.OrderItemId == returnItem.OrderItemId)
.Sum(orderItem => orderItem.Quantity);
return matchingOrderItemQuantity < returnItem.Quantity;
});
and the corresponding types of the variables used above:
public class ReturnItem
{
public int OrderItemId {get; set;}
public string ItemNumber {get; set;}
public int Quantity {get; set;}
// There's more properties but these are the ones that matter
{
public class OrderItem
{
public int OrderItemId {get; set;}
public string ItemNumber {get; set;}
public int Quantity {get; set;}
// There's more properties but these are the ones that matter
{
I expect that var invalidQuantityItems will be an IEnumerable<ReturnItems> whose quantity for an individual item is greater than the amount ordered (i.e. they're trying to return more than they ordered in the first place).
Cheers!

Small correction - the time complexity of the current implementation is O(N*M) and the best you can get is O(N+M).
The problem is how to efficiently correlate the two sets. In LINQ this is achieved with joins, and for this one-to-many type of correlation - group join. The equivalent of the || criteria will be Union of the results of two group joins (matching sets).
Speaking about readability, LINQ and joins, the best would be using the LINQ query syntax (there is a reason some people also call it comprehension syntax).
So the query in question can efficiently (and hopefully readable) be rewritten as follows:
var invalidQuantityItems =
from returnItem in message.ItemsForReturn
join orderItem in message.OrderItems on returnItem.ItemNumber equals orderItem.ItemNumber
into matchingOrderItems1
join orderItem in message.OrderItems on returnItem.OrderItemId equals orderItem.OrderItemId
into matchingOrderItems2
let matchingOrderItemQuantity = matchingOrderItems1.Union(matchingOrderItems2)
.Sum(orderItem => orderItem.Quantity)
where matchingOrderItemQuantity < returnItem.Quantity
select returnItem;

I think the dictionary approach is the best way to go.
About the readability, I think this should be not too bad:
var quantityByItemNumber = message.OrderItems.
Where(i => i.ItemNumber != null).
ToDictionary(
i => i.ItemNumber,
i => i.Quantity);
var quantityByOrderItemId = message.OrderItems.ToDictionary(
i => i.OrderItemId,
i => i.Quantity);
var invalidQuantityItems = message.ItemsForReturn.Where(returnItem =>
{
int matchingOrderItemQuantity;
var isNumberMatch = returnItem.ItemNumber != null) &&
quantityByItemNumber.TryGetValue(returnItem.ItemNumber, out matchingOrderItemQuantity);
if (!isNumberMatch)
quantityByOrderItemId.TryGetValue(returnItem.OrderItemId, out matchingOrderItemQuantity)
return matchingOrderItemQuantity < returnItem.Quantity;
});
In fact I think this is even more readable, because it does not wrongly pretend that there is more than one matching OrderItem, which quantities has to be summed up.

As far as optimizing multiple conditions goes:
Always put the most likely conditions that will end evaluation first (you would have to determine this based on existing data or your knowledge of the system).
If there is not a large likelihood of one condition occurring more frequently than the other, then we can consider the evaluation itself. For example, if an int comparison is faster than a string comparison, then put the int comparison first.
Also, your code doesn't need a separate line to get the Sum; you can do it in the same expression:
var invalidQuantityItems = message.ItemsForReturn.Where(returnItem =>
message.OrderItems
.Where(orderItem =>
orderItem.OrderItemId == returnItem.OrderItemId ||
orderItem.ItemNumber.Equals(returnItem.ItemNumber))
.Sum(orderItem => orderItem.Quantity) < returnItem.Quantity);

Using Include with Intersect/Union/Exclude in Linq

What seemed that it should be a relatively straight-forward task has turned into something of a surprisingly complex issue. To the point that I'm starting to think that my methodology perhaps is simply out of scope with the capabilities of Linq.
What I'm trying to do is piece-together a Linq query and then invoke .Include() in order to pull-in values from a number of child entities. For example, let's say I have these entities:
public class Parent
{
public int Id { get; set; }
public string Name { get; set; }
public string Location { get; set; }
public ISet<Child> Children { get; set; }
}
public class Child
{
public int Id { get; set; }
public int ParentId { get; set; }
public Parent Parent { get; set; }
public string Name { get; set; }
}
And let's say I want to perform a query to retrieve records from Parent, where Name is some value and Location is some other value, and then include Child records, too. But for whatever reason I don't know the query values for Name and Location at the same time, so I have to take two separate queryables and join them, such:
MyDbContext C = new MyDbContext();
var queryOne = C.Parent.Where(p => p.Name == myName);
var queryTwo = C.Parent.Where(p => p.Location == myLocation);
var finalQuery = queryOne.Intersect(queryTwo);
That works fine, producing results exactly as if I had just done:
var query = C.Parent.Where(p => p.Name == myName && p.Location = myLocation);
And similarly, I can:
var finalQuery = queryOne.Union(queryTwo);
To give me results just as if I had:
var query = C.Parent.Where(p => p.Name == myName || p.Location = myLocation);
What I cannot do, however, once the Intersect() or Union() is applied, however, is then go about mapping the Child using Include(), as in:
finalQuery.Include(p => p.Children);
This code will compile, but produces results as follows:
In the case of a Union(), a result set will be produced, but no Child entities will be enumerated.
In the case of an Intersect(), a run-time error is generated upon attempt to apply Include(), as follows:
Expression of type
'System.Collections.Generic.IEnumerable`1[Microsoft.EntityFrameworkCore.Query.Internal.AnonymousObject]'
cannot be used for parameter of type
'System.Collections.Generic.IEnumerable`1[System.Object]' of method
'System.Collections.Generic.IEnumerable`1[System.Object]
Intersect[Object](System.Collections.Generic.IEnumerable`1[System.Object],
System.Collections.Generic.IEnumerable`1[System.Object])'
The thing that baffles me is that this code will work exactly as expected:
var query = C.Parent.Where(p => p.Name == myName).Where(p => p.Location == myLocation);
query.Include(p => p.Children);
I.e., with the results as desired, including the Child entities enumerated.

my methodology perhaps is simply out of scope with the capabilities of Linq
The problem is not LINQ, but EF Core query translation, and specifically the lack of Intersect / Union / Concat / Except method SQL translation, tracked by #6812 Query: Translate IQueryable.Concat/Union/Intersect/Except/etc. to server.
Shortly, such queries currently use client evaluation, which with combination of how the EF Core handles Include leads to many unexpected runtime exceptions (like your case #2) or wrong behaviors (like Ignored Includes in your case #1).
So while your approach technically perfectly makes sense, according to the EF Core team leader response
Changing this to producing a single SQL query on the server isn't currently a top priority
so this currently is not even planned for 3.0 release, although there are plans to change (rewrite) the whole query translation pipeline, which might allow implementing that as well.
For now, you have no options. You may try processing the query expression trees yourself, but that's a complicated task and you'll probably find why it is not implemented yet :) If you can convert your queries to the equivalent single query with combined Where condition, then applying Include will be fine.
P.S. Note that even now your approach technically "works" w/o Include, prefomance wise the way it is evaluated client side makes it absolutely non equivalent of the corresponding single query.

A long time has gone by, but this .Include problem still exists in EF 6. However, there is a workaround: Append every child request with .Include before intersecting/Unionizing.
MyDbContext C = new MyDbContext();
var queryOne = db.Parents.Where(p => p.Name == parent.Name).Include("Children");
var queryTwo = db.Parents.Where(p => p.Location == parent.Location).Include("Children");
var finalQuery = queryOne.Intersect(queryTwo);
As stated by #Ivan Stoev, Intersection/Union is done with after-fetched data, while .Include is ok at request time.
So, as of now, you have this one option available.

LINQ Query - Only get Order and MAX Date from Child Collection

I'm trying to get a list that displays 2 values in a label from a parent and child (1-*) entity collection model.
I have 3 entities:
[Customer]: CustomerId, Name, Address, ...
[Order]: OrderId, OrderDate, EmployeeId, Total, ...
[OrderStatus]: OrderStatusId, StatusLevel, StatusDate, ...
A Customer can have MANY Order, which in turn an Order can have MANY OrderStatus, i.e.
[Customer] 1--* [Order] 1--* [OrderStatus]
Given a CustomerId, I want to get all of the Orders (just OrderId) and the LATEST (MAX?) OrderStatus.StatusDate for that Order.
I've tried a couple of attempts, but can seem to get the results I want.
private IQueryable<Customer> GetOrderData(string customerId)
{
var ordersWithLatestStatusDate = Context.Customers
// Note: I am not sure if I should add the .Expand() extension methods here for the other two entity collections since I want these queries to be as performant as possible and since I am projecting below (only need to display 2 fields for each record in the IQueryable<T>, but thinking I should now after some contemplation.
.Where(x => x.CustomerId == SelectedCustomer.CustomerId)
.Select(x => new Custom
{
CustomerId = x.CustomerId,
...
// I would like to project my Child and GrandChild Collections, i.e. Orders and OrderStatuses here but don't know how to do that. I learned that by projecting, one does not need to "Include/Expand" these extension methods.
});
return ordersWithLatestStatusDate ;
}
---- UPDATE 1 ----
After the great solution from User: lazyberezovsky, I tried the following:
var query = Context.Customers
.Where(c => c.CustomerId == SelectedCustomer.CustomerId)
.Select(o => new Customer
{
Name = c.Name,
LatestOrderDate = o.OrderStatus.Max(s => s.StatusDate)
});
In my hastiness from my initial posting, I didn't paste everything in correctly since it was mostly from memory and didn't have the exact code for reference at the time. My method is a strongly-typed IQueryabled where I need it to return a collection of items of type T due to a constraint within a rigid API that I have to go through that has an IQueryable query as one of its parameters. I am aware I can add other entities/attributes by either using the extension methods .Expand() and/or .Select(). One will notice that my latest UPDATED query above has an added "new Customer" within the .Select() where it was once anonymous. I'm positive that is why the query failed b/c it couldn't be turn into a valid Uri due to LatestOrderDate not being a property of Customer at the Server level. FYI, upon seeing the first answer below, I had added that property to my client-side Customer class with simple { get; set; }. So given this, can I somehow still have a Customer collection with the only bringing back those 2 fields from 2 different entities? The solution below looked so promising and ingenious!
---- END UPDATE 1 ----
FYI, the technologies I'm using are OData (WCF), Silverlight, C#.
Any tips/links will be appreciated.

This will give you list of { OrderId, LatestDate } objects
var query = Context.Customers
.Where(c => c.CustomerId == SelectedCustomer.CustomerId)
.SelectMany(c => c.Orders)
.Select(o => new {
OrderId = o.OrderId,
LatestDate = o.Statuses.Max(s => s.StatusDate) });
.
UPDATE construct objects in-memory
var query = Context.Customers
.Where(c => c.CustomerId == SelectedCustomer.CustomerId)
.SelectMany(c => c.Orders)
.AsEnumerable() // goes in-memory
.Select(o => new {
OrderId = o.OrderId,
LatestDate = o.Statuses.Max(s => s.StatusDate) });
Also grouping could help here.

If I read this correctly you want a Customer entity and then a single value computed from its Orders property. Currently this is not supported in OData. OData doesn't support computed values in the queries. So no expressions in the projections, no aggregates and so on.
Unfortunately even with two queries this is currently not possible since OData doesn't support any way of expressing the MAX functionality.
If you have control over the service, you could write a server side function/service operation to execute this kind of query.