We Keep Coding

Accessing info from CommonPart is extremely slow?

I'm new to Orchard and this must be something involving how the underlying data is stored.
The joining with CommonPart seems fast enough, like this:
var items = _contentManager.Query<MyUserPart, MyUserPartRecord>("someTypeName")
.ForVersion(VersionOptions.Published)
.Join<CommonPartRecord>().List().ToList();
That runs fairly fast. But whenever I try accessing some field in CommonPart, it runs extremely slow like this:
var items = _contentManager.Query<MyUserPart, MyUserPartRecord>("someTypeName")
.ForVersion(VersionOptions.Published)
.Join<CommonPartRecord>().List()
//access some field from commonpart
.Select(e => new {
User = e.As<CommonPart>().Owner.UserName
}).ToList();
The total data is just about 1200 items, and the time it needs is about 5 seconds, it cannot be slow like that. For a simple SQL query run in background, it should take a time of about 0.5 second or even less than.
I've tried investigating the Orchard's source code but found nothing that could be the issue. Everything seems to go into a blackbox at the accessing point of IContent. I hope someone here could give me some suggestion to diagnose and solve this hard issue. Thanks!
Update:
I've tried debugging a bit and seen that the following method is hit inside the DefaultContentManager:
ContentItem New(string contentType) { ... }
Well that's really interesting, the query is just asking for data without modifying, inserting and updating anything. But that method being hit shows that something's wrong here.
Update:
With #Bertrand Le Roy's comment, I've tried the following codes with QueryHint but looks like it does not change anything:
var items = _contentManager.Query<MyUserPart, MyUserPartRecord>("someTypeName")
.ForVersion(VersionOptions.Published)
.Join<CommonPartRecord>()
.WithQueryHints(new QueryHints().ExpandParts<CommonPart>())
.List()
//access some field from commonpart
.Select(e => new {
User = e.As<CommonPart>().Owner.UserName
}).ToList();
and this (without .Join)
var items = _contentManager.Query<MyUserPart, MyUserPartRecord>("someTypeName")
.ForVersion(VersionOptions.Published)
.WithQueryHints(new QueryHints().ExpandParts<CommonPart>())
.List()
//access some field from commonpart
.Select(e => new {
User = e.As<CommonPart>().Owner.UserName
}).ToList();

Accessing the Owner property from your Select causes the lazy loader in CommonPartHandler to ask the content manager to load the user content item: _contentManager.Get<IUser>(part.Record.OwnerId). This happens once per content item result from your query, so results in a select n+1 where n = 1200 according to your question.
There are at least two ways of avoiding that:
You can use HQL and craft a query that gives you everything you need up front in 1 operation.
You can make a 1st content manager query to get the set of owner ids, and then
make a second content manager query for those Ids and get everything you need with a total of 2 queries instead of 1201.

Least expensive operation to check for new records in table with EF

I have PostgreSQL running together with EF7.
My table structure:
Id (bigint)
Content (text)
CreatedAt (timestamp without time zone NOT NULL)
I have infinite scroll in my frontend app and query results with .Skip(n) & .Take(m).
I.e.
.Skip(0), .Take(10);
.Skip(10), .Take(10);
.Skip(20), .Take(10);
<...>
Now, while scrolling, if there're newer records, I have to know how many and add them to .Skip(n) function. I don't need to display them, just need to add them to consideration while skipping.
Currently I'm checking for them like so but this seemingly should be quite expensive after table will exceed 50k-100k records:
_myRepository.GetAll().Where(x => x.CreatedAt > newestActivityDate).Count();
GetAll():
public IQueryable<Activity> GetAll()
{
return _context.Activities.OrderByDescending(x => x.CreatedAt);
}
What would be the best (most performant) operation to check whether there're new records & how many? Checking by date and only then doing count if there're any new records?
EDIT:
Added GetAll() description for more clarity.

I don't know what your .GetAll() method does, so that really is the crux of your problem.
LINQ has what is called deferred execution. What that means is that the query will not get executed until the results are acted upon. So in the case of when you're building a SQL query, nothing will get sent to the database until you finish with it.
So if your .GetAll() method queries with a .ToList() in there, then it will pull everything from the database immediately before adding the rest of your filtering. In that case, yes, it will be very expensive.
However, you can get around that by just asking for the count. You already have most of that set up.
If you add a new method in your repository like this:
public int GetNewRecordsCount(DateTime newestActivityDate)
{
_context.Widgets.Count(x => x.CreatedAt > newestActivityDate);
}
Then this will create a different SQL query that is just something similar to this:
SELECT COUNT(*)
FROM Widgets
WHERE CreatedAt > newestActivityDate;
That is a very quick operation where the database will do all of the filtering for you. The only thing returned to your code is just a single row, single column result of the total count.

EF LINQ ToList is very slow

I am using ASP NET MVC 4.5 and EF6, code first migrations.
I have this code, which takes about 6 seconds.
var filtered = _repository.Requests.Where(r => some conditions); // this is fast, conditions match only 8 items
var list = filtered.ToList(); // this takes 6 seconds, has 8 items inside
I thought that this is because of relations, it must build them inside memory, but that is not the case, because even when I return 0 fields, it is still as slow.
var filtered = _repository.Requests.Where(r => some conditions).Select(e => new {}); // this is fast, conditions match only 8 items
var list = filtered.ToList(); // this takes still around 5-6 seconds, has 8 items inside
Now the Requests table is quite complex, lots of relations and has ~16k items. On the other hand, the filtered list should only contain proxies to 8 items.
Why is ToList() method so slow? I actually think the problem is not in ToList() method, but probably EF issue, or bad design problem.
Anyone has had experience with anything like this?
EDIT:
These are the conditions:
_repository.Requests.Where(r => ids.Any(a => a == r.Student.Id) && r.StartDate <= cycle.EndDate && r.EndDate >= cycle.StartDate)
So basically, I can checking if Student id is in my id list and checking if dates match.

Your filtered variable contains a query which is a question, and it doesn't contain the answer. If you request the answer by calling .ToList(), that is when the query is executed. And that is the reason why it is slow, because only when you call .ToList() is the query executed by your database.
It is called Deferred execution. A google might give you some more information about it.
If you show some of your conditions, we might be able to say why it is slow.

In addition to Maarten's answer I think the problem is about two different situation
some condition is complex and results in complex and heavy joins or query in your database
some condition is filtering on a column which does not have an index and this cause the full table scan and make your query slow.
I suggest start monitoring the query generated by Entity Framework, it's very simple, you just need to set Log function of your context and see the results,
using (var context = new MyContext())
{
context.Database.Log = Console.Write;
// Your code here...
}
if you see something strange in generated query try to make it better by breaking it in parts, some times Entity Framework generated queries are not so good.
if the query is okay then the problem lies in your database (assuming no network problem).
run your query with an SQL profiler and check what's wrong.
UPDATE
I suggest you to:
add index for StartDate and EndDate Column in your table (one for each, not one for both)

ToList executes the query against DB, while first line is not.
Can you show some conditions code here?
To increase the performance you need to optimize query/create indexes on the DB tables.

Your first line of code only returns an IQueryable. This is a representation of a query that you want to run not the result of the query. The query itself is only runs on the databse when you call .ToList() on your IQueryable, because its the first point that you have actually asked for data.
Your adjustment to add the .Select only adds to the existing IQueryable query definition. It doesnt change what conditions have to execute. You have essentially changed the following, where you get back 8 records:
select * from Requests where [some conditions];
to something like:
select '' from Requests where [some conditions];
You will still have to perform the full query with the conditions giving you 8 records, but for each one, you only asked for an empty string, so you get back 8 empty strings.
The long and the short of this is that any performance problem you are having is coming from your "some conditions". Without seeing them, its is difficult to know. But I have seen people in the past add .Where clauses inside a loop, before calling .ToList() and inadvertently creating a massively complicated query.

Jaanus. The most likely reason of this issue is complecity of generated SQL query by entity framework. I guess that your filter condition contains some check of other tables.
Try to check generated query by "SQL Server Profiler". And then copy this query to "Management Studio" and check "Estimated execution plan". As a rule "Management Studio" generatd index recomendation for your query try to follow these recomendations.

Multiple operations under linq2Entities

I've been using Linq2Entities for quite some time now on small scale programs. So usually my queries are basic (return elements with a certain value in a specific column, add/update element, remove element,...).
Now i'm moving to a larger scale program and given my experience went with Linq2Entities and everything was fine until i had to remove a large number of elements.
In Linq2Entities i can only find a remove method that takes on entity - and so to remove 1000 elemnts i need first to retrieve them then apply a remove one by one then call savechanges.
Either i am missing something or i feel a) this is really bad performance-wise and b) i would be better off making this a procedure in the database.
Am i right in my assumptions? Should massive removals / additions be made as procedures only? or is there a way to do this efficiently with Linq2Entities that i am missing?

If you have the primary key values then you can use the other features of the Context by creating the objects manually, setting their key(s), and attaching them to the context and setting their state to delete.
var ids = new List<Guid>();
foreach (var id in ids)
{
Employee employee = new Employee();
employee.Id = id;
entityEntry = context.Entry(employee);
entityEntry.State = EntityState.Deleted;
}
context.SaveChanges();

Linq2Sql: Random double-insert bug - a real poser

OK, so you think you're a real debugger? Try this one:
I've got a Linq2Sql project where suddenly we've discovered that occasionally, seemingly randomly, we get double-inserts of the same row.
I have subclassed DataContext and overridden SubmitChanges. Inside there I have made a call to GetChangeSet(), both before and after the call to base.SubmitChanges(), and used a text file logger to record the objects in the Inserts collection. Plus I hang on to a reference to the inserted objects long enough to record their autonumbered ID.
When the double-insert happens, I see in the DB that instead of one row each being inserted into MyTableA and MyTableB, there are two each. SQL Profiler shows four insert statements, one after the other in rapid succession:
insert into MyTableA(...
insert into MyTableB(...
insert into MyTableA(...
insert into MyTableB(...
I check in the debug log, and there are only two objects in the Inserts collection: one of MyClassA and one of MyClassB. After the call to base.SubmitChanges(), the changeset is empty (as it should be). And the autonumber IDs show the larger value of the newly inserted rows.
Another piece of useful information: the bug never happens when stepping through in debug mode; only when you run without breakpoints. This makes me suspect it's something to do with the speed of execution.
We have been using the same DataContext subclass for over a year now, and we've never seen this kind of behavior before in our product. It only happens with MyClassA and MyClassB.
To summarize:
From the debug log, everything looks like it's working correctly.
On SQL Profiler, you can see that a double-insert is happening.
This behavior happens frequently but unpredictably, only to the two classes mentioned, excepting that it never happens when stepping through code in debug mode.
EDIT - New information:
Inside my DataContext subclass, I have the following code:
try {
base.SubmitChanges(ConflictMode.ContinueOnConflict);
} catch (ChangeConflictException) {
// Automerge database values for members that client has not modified.
foreach (ObjectChangeConflict occ in ChangeConflicts) {
occ.Resolve(RefreshMode.KeepChanges);
}
}
// Submit succeeds on second try.
base.SubmitChanges(ConflictMode.FailOnFirstConflict);
MyTableA and MyTableB both have a mandatory foreign key OtherTableID referencing OtherTable. The double insert happens when a ChangeConflictException happens during an update of the common parent table OtherTable.
We're on the scent, now...

When I've had a problem like this before it is usually down to multiple threads executing the same code at the same time.
Have you tried using the lock{} command to make sure the insert is only being used by a single thread?
MSDN Lock

looks like it's a BUG in Linq2Sql! Here's a repeatable experiment for you:
using (var db1 = new MyDataContext()) {
var obj1 = db1.MyObjects.Single(x => x.ID == 1);
obj1.Field1 = 123;
obj1.RelatedThingies.Add(new RelatedThingy {
Field1 = 456,
Field2 = "def",
});
using (var db2 = new MyDataContext()) {
var obj2 = db2.MyObjects.Single(x => x.ID == 1);
obj2.Field2 = "abc";
db2.SubmitChanges();
}
try {
db1.SubmitChanges(ConflictMode.ContinueOnConflict);
} catch (ChangeConflictException) {
foreach (ObjectChangeConflict occ in ChangeConflicts) {
occ.Resolve(RefreshMode.KeepChanges);
}
}
base.SubmitChanges(ConflictMode.FailOnFirstConflict);
}
Result: MyObject record with ID = 1 gets updated, Field1 value is 123 and Field2 value is "abc". And there are two new, identical records inserted to RelatedThingy, with MyObjectID = 1, Field1 = 456 and Field2 = "def".
Explain THAT!
UPDATE: After logging this on Microsoft Connect, the nice folks at MS asked me to put together a little demo project highlighting the bug. And wouldn't you know it - I couldn't reproduce it. It seems to be connected to some weird idiosyncrasy of my project. Don't have time to investigate further, and I found a workaround, anyway...

FWIW, we recently found this problem with our retry logic for SubmitChanges. We were doing an InsertAllOnSubmit. When a ChangeConflictException occurred, we would retry with a Resolve(RefreshMode.KeepChanges,true) on each ObjectChangeConflict.
We redid the work a different way (retry logic to re-perform the entire transaction) and that seems to fix the problem.