We Keep Coding

Replacing Include() calls to Select()

Im trying to eliminate the use of the Include() calls in this IQueryable definition:
return ctx.timeDomainDataPoints.AsNoTracking()
.Include(dp => dp.timeData)
.Include(dp => dp.RecordValues.Select(rv => rv.RecordKind).Select(rk => rk.RecordAlias).Select(fma => fma.RecordAliasGroup))
.Include(dp => dp.RecordValues.Select(rv => rv.RecordKind).Select(rk => rk.RecordAlias).Select(fma => fma.RecordAliasUnit))
.Where(dp => dp.RecordValues.Any(rv => rv.RecordKind.RecordAlias != null))
.Where(dp => dp.Source == 235235)
.Where(dp => dp.timeData.time >= start && cd.timeData.time <= end)
.OrderByDescending(cd => cd.timeData.time);
I have been having issues with the database where the run times are far too long and the primary cause of this is the Include() calls are pulling everything.
This is evident in viewing the table that is returned from the resultant SQL query generated from this showing lots of unnecessary information being returned.
One of the things that you learn I guess.
The Database has a large collection of data points which there are many Recorded values.
Each Recorded value is mapped to a Record Kind which may have a Record Alias.
I have tried creating a Select() as an alternative but I just cant figure out how to construct the right Select and also keep the entity hierarchy correctly loaded. I.e. the related entities are loaded with unnecessary calls to the DB.
Does anyone has alternate solutions that may jump start me to solve this problem.
Ill add more detail if needed.

You are right. One of the slower parts of a database query is the transport of the selected data from the DBMS to your local process. Hence it is wise to limit this.
Every TimeDomainDataPoint has a primary key. All RecordValues of this TimeDomainDataPoint have a foreign key TimeDomainDataPointId with a value equal to this primary key.
So If TimeDomainDataPoint with Id 4 has a thousand RecordValues, then every RecordValue will have a foreign key with a value 4. It would be a waste to transfer this value 4 a 1001 times, while you only need it once.
When querying data, always use Select and select only the properties you actually plan to use. Only use Include if you plan to update the fetched included items.
The following will be much faster:
var result = dbContext.timeDomainDataPoints
// first limit the datapoints you want to select
.Where(datapoint => d.RecordValues.Any(rv => rv.RecordKind.RecordAlias != null))
.Where(datapoint => datapoint.Source == 235235)
.Where(datapoint => datapoint.timeData.time >= start
&& datapoint.timeData.time <= end)
.OrderByDescending(datapoint => datapoint.timeData.time)
// then select only the properties you actually plan to use
Select(dataPoint => new
{
Id = dataPoint.Id,
RecordValues = dataPoint.RecordValues
.Where(recordValues => ...) // if you don't want all RecordValues
.Select(recordValue => new
{
// again: select only the properties you actually plan to use:
Id = recordValue.Id,
// not needed, you know the value: DataPointId = recordValue.DataPointId,
RecordKinds = recordValues.RecordKinds
.Where(recordKind => ...) // if you don't want all recordKinds
.Select(recordKind => new
{
... // only the properties you really need!
})
.ToList(),
...
})
.ToList(),
TimeData = dataPoint.TimeData.Select(...),
...
});
Possible imporvement
The part:
.Where(datapoint => d.RecordValues.Any(rv => rv.RecordKind.RecordAlias != null))
is used to fetch only datapoints that have recordValues with a non-null RecordAlias. If you are selecting the RecordAlias anyway, consider doing this Where after your select:
.Select(...)
.Where(dataPoint => dataPoint
.Where(dataPoint.RecordValues.RecordKind.RecordAlias != null)
.Any());
I'm not really sure whether this is faster. If your database management system internally first creates a complete table with all columns of all joined tables and then throws away the columns that are not selected, then it won't make a difference. However, if it only creates a table with the columns it actually uses, then the internal table will be smaller. This could be faster.

your problem is hierarchy joins in your query.In order to decrease this problem create other query for get result from relation table as follows:
var items= ctx.timeDomainDataPoints.AsNoTracking().Include(dp =>dp.timeData).Include(dp => dp.RecordValues);
var ids=items.selectMany(item=>item.RecordValues).Select(i=>i.Id);
and on other request to db:
var otherItems= ctx.RecordAlias.AsNoTracking().select(dp =>dp.RecordAlias).where(s=>ids.Contains(s.RecordKindId)).selectMany(s=>s.RecordAliasGroup)
to this approach your query do not have internal joins.

When Include() executing in Entity Framework?

I have a simple question but didn't find an answers.
If I do so
var result = _db.Table.Include(t => t.Child).Where(t => t.Id == id).Single();
when join is calling?
After it found my entity or it includes every child during SQL looking for the row?
Lets see at the example based on simple db model:
public class Head
{
//... columns
public virtual Child {get; set;}
public Guid? ChildId {get; set;}
}
void main()
{
//The first version of code
var child = _db.Head.Include(h => h.Child)
.FirstOrDefault(//boring staff but we don't need child here)
?.Child;
if (child != null)
foo(child);
//The second one
var head = _db.Head.FirstOrDefault(//boring staff);
if (head != null && head.ChildId.HasValue)
foo(head.Child); // I know here we make a new request to our db
}
Which of two options are more productive?
I'm worry about "extra childs loading by SQL" when I need only one object based on filters query of parent table.
Thanks in advance!

It will evaluate the where condition first. Not in C# but in SQL which gets generated.
This will generate a SQL something like
SELECT top 1 .... FROM Table t
JOIN Child c ....
WHERE t.Id = id
Your database server will create a execution plan which will look for the item in the index and get corresponding child.

Without Include the loading of Child objects is deferred until you need them. Hence, if you were to iterate parent/child groups like this
foreach (var parent in _db.Table.Include(t => t.Child).Where(p => p.Name.StartsWith("Q")))
foreach (var child in parent.Child)
Console.WriteLine($"{child}, child of {parent}");
the number of round-trips would be equal to the number of parents plus one.
If you use Include, all Child objects are loaded along with the parent object, without making a separate round-trip for each parent. Hence, the number of database round-trips for the above code would be equal to 1.
In a case with Single, which could be rewritten as follows
var result = _db.Table.Include(t => t.Child).Single(t => t.Id == id);
the number of round-trips would be 1 with Include and 2 without Include.

Entity framework query on just added but not saved values

I'm using Entity Framework from a couple of years and I have a little problem now.
I add an entity to my table, with
Entities.dbContext.MyTable.Add(obj1);
and here ok.
Then, I'd like to make a query on MyTable, like
Entities.dbContext.MyTable.Where(.....)
The code above will query on my MyTable in the db.
Is there a way to query also on the just added value, before the saveChanges? (obj1) How?
UPDATE
Why do I need this? Because, for each new element I add, I need to edit some values in the previous and the next record (there is a datetime field in this table)
UPDATE2
Let's say I have to add a lot of objects, but I call the saveChanges only after the last item is added. Every time I add the new item, I read its datetime field and I search in the database the previous and the next record. Here, I edit a field of the previous and of the next record. Now, here is problem: if I insert another item, and, for example, the next item is "Obj1", I have to find and edit it, but I can't find it since I haven't saved my changes. Is it clearer now?

You should be able to get your added entities out of the dbContext via the change tracker like this:
var addedEntities = dbContext.ChangeTracker.Entries()
.Where(x => x.State == EntityState.Added && x.Entity is Mytable)
.Select(x => x.Entity as MyTable)
.Where(t => --criteria--);
Or using the type testing with pattern matching in c# 7.0:
var addedEntities = dbContext.ChangeTracker.Entries()
.Where(x => x.State == EntityState.Added && x.Entity is Mytable t && --test t for criteria--)
.Select(x => x.Entity as MyTable);
because you are only querying added entities, you can combine this with
dbContext.MyTable.Where(t => --criteria--).ToList().AddRange(addedEntities);
to get all of the relevant objects

I think this is a good situation for Transactions. I am going to assume you are using EF 6 since you did not provide a version. =)
UPDATE2 changes
public void BulkInsertObj(List<TEntity> objList)
{
using (var context = new dbContext())
{
using (var dbContextTransaction = context.Database.BeginTransaction())
{
try
{
foreach(var obj1 in objList)
{
dbContext.MyTable.Add(obj1);
//obj1 should be on the context now
var previousEntity = dbContext.MyTable.Where(.....) //However you determine this
previousEntity.field = something
var nextEntity = dbContext.MyTable.Where(.....) //However you determine this
nextEntity.field = somethingElse
}
context.SaveChanges();
dbContextTransaction.Commit();
}
catch (Exception)
{
dbContextTransaction.Rollback();
}
}
}
}
MSDN EF6 Transactions

Entity Framework Query: What's Happening Under the Covers Here?

I have the following code.
var category = Office.Categories.FirstOrDefault(c => c.Description == name);
Office is a Framework Entity object that was read earlier.
Even though my database uses case-insensitive string comparisons, and one of the items matches exactly except for the case, this code is returning null.
My understanding of what is happening here is that Office.Categories is returning all related rows, and then those rows are being searched by regular LINQ, which is case-sensitive.
If so, that is horribly inefficient as I only want to return the row that matches my condition.
Can anyone confirm my understanding of this? And is it possible to force the filtering to take place in SQL so that I don't need to return the rows I'm not interested in? (And the text comparison will be case-insensitive?)
Thanks for any help.

If you already have the Office object loaded (also depends on lazy loading, but assuming you're outside the context) then you're basically doing a linq-to-object query which is case sensitive. If you want to do linq-to-sql then you should do it differently, something like:
context.Offices.FirstOrDefault(o=>o.OfficeId == someId)
.Categories.FirstOrDefault(c => c.Description == name);
or
context.Categories.FirstOrDefault(c => c.OfficeId == Office.OfficeId
&& c.Description == name);

Correct, so you can't:
Call the Categories property getter if using lazy loading because this causes EF to retrieve the full list of categories from the database
Use eager loading, because this also causes the full list of categories to be loaded from the database
To avoid this, you can use the following code (from this answer here: Using CreateSourceQuery in CTP4 Code First)
public ObjectQuery<T> CreateNavigationSourceQuery<T>(object entity, string navigationProperty)
{
var ose = this.ObjectContext.ObjectStateManager.GetObjectStateEntry(entity);
var rm = this.ObjectContext.ObjectStateManager.GetRelationshipManager(entity);
var entityType = (EntityType)ose.EntitySet.ElementType;
var navigation = entityType.NavigationProperties[navigationProperty];
var relatedEnd = rm.GetRelatedEnd(navigation.RelationshipType.FullName, navigation.ToEndMember.Name);
return ((dynamic)relatedEnd).CreateSourceQuery();
}
Add it to your context and call it to create an ObjectQuery.. you can call FirstOrDefault on this and it won't retrieve all entities from the database:
var category = context.CreateNavigationSourceQuery<Category>(Office, "Categories").FirstOrDefault(c => c.Description == name);

How to convert this foreach loop into Linq code?

I am new one with Linq and I would like to modify my old c# code to use Linq.
The idea of this code to select all tables where it's not set and reference’s field PrimaryTable equal "myTable"
foreach (Table table in dbServer.Tables)
{
if (!table.IsSet)
{
foreach (Reference refer in table.References)
{
if (refer.PrimaryTable == "myTable")
{
tables.Add(table);
}
}
}
}
After digging in internet I have got this code
var q = from table in dbServer.Tables
let refers = from refer in table.References
where refer.PrimaryTable == "myTable"
select refer.ForeignTable
where refers.Contains(table.Name)
select table;
But it does not work at all and I need your help to make it works.
Thanks in advance.

var tables = dbServer.Tables
.Where(t => !t.IsSet)
.SelectMany(t => t.References)
.Where(r => r.PrimaryTable == "myTable")
.ToList();
Assuming tables is a List<T>
EDIT: As the comment points out, this isn't the same as the original - it looks like what you actually want is this:
var tables = dbServer.Tables
.Where(t => !t.IsSet && t.References.Any(r => r.PrimaryTable == "myTable"))
.ToList();
This will give you all the tables which have a reference whose PrimaryTable is 'myTable' which assumes that there will only be one matching reference table. Otherwise you could have the same table added multiple times.

Just need to use two from's
var q = from table in dbServer.Tables
where !table.IsSet
from refer in table.References
where refer.PrimaryTable == "myTable"
select table;

EDIT
Actually, I'm a bit confused by this code. Are you sure it's doing what it's meant to do? In particular, what's throwing me off is the fact that you're enumerating over table.References, but then, when a certain condition holds for a particular Reference (i.e., refer.PrimaryTable == "myTable"), you're adding the Table (table) instead of the Reference (refer).
What this means is that if a Table has multiple Reference objects with PrimaryTable == "myTable", your tables collection might contain multiple copies of this Table. Is this correct?
I'm going to go out on a limb and guess that what you really want to check is simply that a Table has, in its References collection, any Reference object with PrimaryTable == "myTable". If that's the case, in your original code after tables.Add(table) I would have simply added break to avoid duplicates. (It may be that only one Reference in each collection would ever have the same PrimaryTable, which case you'd be fine; but you could still stop enumerating at this point. Unless of course you want the duplicates.)
In any case, Lee's code (and what I have below) is not duplicating this behavior. Rather, it's adding the Reference objects to a list (because the final ToList call is on an IEnumerable<Reference>).
It seems like, if what I've described above is the behavior you're after, you might want to do this:
var tables = dbServer.Tables
.Where(table => !table.IsSet)
.Where(
table => table.References.Any(refer => refer.PrimaryTable == "myTable")
).ToList();
ORIGINAL ANSWER
I'm going to expand on Lee's answer. Let's analyze it line by line.
// 1. enumerating over a collection
foreach (Table table in dbServer.Tables)
{
// 2. checking a condition
if (!table.IsSet)
{
// 3. enumerating over another collection
foreach (Reference refer in table.References)
{
// 4. checking a condition
if (refer.PrimaryTable == "myTable")
{
// 5. adding to a collection
tables.Add(table);
}
}
}
}
OK. So we've got:
Enumeration - simple -- that's where we start
Condition checking - we'll need a Where
Enumeration over another collection - SelectMany
Condition checking - Where again
Adding - most likely ToList (depends on what type of collection you want)
Here's what it comes out to:
var tables = dbServer.Tables // step 1
.Where(table => !table.IsSet) // step 2
.SelectMany(table => table.References) // step 3
.Where(refer => refer.PrimaryTable == "myTable") // step 4
.ToList(); // step 5
Make sense?

tables.AddRange(dbServer.Tables
.Where(t => !t.IsSet)
.SelectMany(t => table.References)
.Where(r => r.PrimaryTable == "myTable"));