We Keep Coding

Speeding up LINQ queries (selecting data from table)

I have written a code which looks like this:
using(var ctx = new myentitiesContext())
{
var currentLoggedUser = ctx.Users.FirstOrDefault(x=>x.Email==User.Identity.Name);
var items = ctx.Items.Where(x=>x.Sales>0 && x.UserId==currentLoggedUser.UserId).ToList();
}
As you can see it's a simple select from the DB. But the tricky part is that sometimes I can select a large quantity of data (50-100k records at a time). So what I've been wondering, are there any ways to tweak the LINQ to perform faster when the data is being pulled out of the table?
I've already created indexes in my table on FK UserId, so that part is done.
My question here is, is there any way to speed up LINQ queries via some tweaks in context configuration section, or perhaps by creating compiled queries, or via some other method ?
P.S. guys, would something like this work good:
ctx.Configuration.AutoDetectChangesEnabled = false;
// my queries...
ctx.Configuration.AutoDetectChangesEnabled = true;

In addition with the things that the rest of the users have written. You could disable lazy loading. That way if the Items Db Table has references to other tables they will not get loaded along with the Items unless you absolutely need it. Check these links
thecodegarden
mehdi
One more think that i would recommend is that you must log the sql queries that your linq expressions create and try to optimise them with your DBA. You could do this by adding an Action<string> delegate on the DbContext.Database.Log that will emit everything between a connection.Open() and a connection.Close(). You could also take the sql query out of your IQueryableor IQueryable<T> calling the .ToString() method on your IQueryable variable.

You should make projection first. For example, this:
var items = ctx.Items.Where(x=>x.Sales>0 && x.UserId==currentLoggedUser.UserId).ToList();
will be better if you write it like this:
var items = ctx.Items.Where(x.UserId==currentLoggedUser.UserId).Where(x2=>x2.Sales>0 ).ToList();
And if you don't need all the object you have to use the "Select" clause before the "Where" and project just the properties that you need to minimize the cost, like this:
ctx.Items.Select(e=>new {e.UserID,e.Sales}).Where(x.UserId==currentLoggedUser.UserId).Where(x2=>x2.Sales>0 ).ToList();

Efficiency in Populating complex object properties (lists) with LINQ to Entities

Alright, this one will have quite a bit of code in it, and may be somewhat long. That said, I'm really having trouble finding the best way to do this, and any help would be immensely appreciated. I am here to learn, specifically the most efficient way to write these types of queries.
NOTE: The system I'm working with right now is an ASP.NET MVC site written in VB.NET, using Entity Framework 6.
With all that out of the way, I believe my best bet here is to just show an example, and see what options are available. My issue comes in to play when I need to populate an object's properties utilizing a LINQ to Entities query hitting the database, but some of those properties are lists themselves of both simple integers, as well as more complex objects. For example, take this query:
Note that in the code example below I removed a complex WHERE clause and some other irrelevant to the question properties I populate to make it smaller and easier to read.
//NOTE: If any context is needed to better understand this, I can provide it. I do provide a bit in the comments. Actual code uses ' for comments (VB).
/*Context: db is an instance of the EF context. Code is representing threads
and replies in an "activity feed". Threads can have private recipients and
uploads, both of which are grabbed from the database. These are the two
properties I'm populating which this question addresses further down.*/
//Get all my activity feed threads
model.ActivityFeedThreads = (From thds In db.tblThreads
From thps In db.tblThreadParticipants.Where(Function(w) w.thpThread_thdID = thds.thdID).DefaultIfEmpty()
Join prns In db.tblPersons On thds.thdOwner_prnID Equals prns.prnID
Join emps In db.tblEmployees On prns.prnID Equals emps.empPerson_prnID
Where (thps Is Nothing And thds.thdPrivacy_lvlID = ActivityThread.ActivityThreadPrivacy.PublicThread) _
Select New ActivityThread With {.thdID = thds.thdID,
.Content = thds.thdContent,
.ThreadOwner_prnID = thds.thdOwner_prnID,
.PrivacyOption_lvlID = thds.thdPrivacy_lvlID,
.ThreadDate = thds.thdDateCreated}).Distinct().Take(20).OrderByDescending(Function(o) o.ThreadDate).ToList()
Now, one property which I need to populate that is not included in that query is "AttachedFiles", which is a list of type "AppFile", a custom class in our system. In addition, there is a second property called "PrivateRecipients", which is a list of simple Int32s. I have tried to work those in to the above query without success, so I have to loop through the resulting list and populate them, resulting in numerous hits to the database. Below is my current code:
//I need to get the private recipients for this post.. Is there a way to work this in to the above query? What's the most efficient solution here?
For Each threadToEdit In model.ActivityFeedThreads
threadToEdit.PrivateRecipients = db.tblThreadParticipants.Where(Function(w) w.thpThread_thdID = threadToEdit.thdID).Select(Function(s) s.thpPerson_prnID).ToList()
Next
//Again, loop through, grab all attached files. Similar situation as above.. Can I work it in to the query?
For Each threadToEdit In model.ActivityFeedThreads
threadToEdit.AttachedFiles = (From flks In db.tblFileLinks
Join fils In db.tblFiles On flks.flkFile_filID Equals fils.filID
Where flks.flkTarget_ID = threadToEdit.thdID And flks.flkLinkType_lvlID = AppFile.FileLinkType.Thread_Upload
Select New AppFile With {.filID = fils.filID,
.Location = fils.filLocation,
.FileURL = fils.filLocation,
.Name = fils.filName,
.FileName = fils.filName,
.MIMEType = fils.filMIMEType}).ToList()
Next
As you can see, in both situations, I'm having to loop through and hit the database a bunch of times, and I have to imagine as the data grows this will become a bit of a bottleneck.. Is there a better way to go about this with LINQ to Entities? Should I change my approach altogether?
Thank you in advance for your time/help, it's greatly appreciated!

How does linq actually execute the code to retrieve data from the data source?

I will start working on xamarin shortly and will be transferring a lot of code from android studio's java to c#.
In java I am using a custom classes which are given arguments conditions etc, convert them to SQL statements and then loads the results to the objects in the project's model
What I am unsure of is wether linq is a better option for filtering such data.
For example what would happen currently is somethng along these lines
List<Customer> customers = (new CustomerDAO()).get_all()
Or if I have a condition
List<Customer> customers = (new CustomerDAO()).get(new Condition(CustomerDAO.Code, equals, "code1")
Now let us assume I have transferred the classes to c# and I wish to do somethng similar to the second case.
So I will probably write something along the lines of:
var customers = from customer
in (new CustomerDAO()).get_all()
where customer.code.equals("code1")
select customer
I know that the query will only be executed when I actually try to access customers, but if I have multiple accesses to customers ( let us say that I use 4 foreach loops later on) will the get_all method be called 4 times? or are the results stored at the first execution?
Also is it more efficient (time wise because memory wise it is probably not) to just keep the get_all() method and use linq to filter the results? Or use my existing setup which in effect executes
Select * from Customers where code = 'code1'
And loads the results to an object?
Thanks in advance for any help you can provide
Edit: yes I do know there is sqlite.net which pretty much does what my daos do but probably better, and at some point I will probably convert all my objects to use it, I just need to know for the sake of knowing

if I have multiple accesses to customers ( let
us say that I use 4 foreach loops later on) will the get_all method be
called 4 times? or are the results stored at the first execution?
Each time you enumerate the enumerator (using foreach in your example), the query will re-execute, unless you store the materialized result somewhere. For example, if on the first query you'd do:
var customerSource = new CustomerDAO();
List<Customer> customerSource.Where(customer => customer.Code.Equals("code1")).ToList();
Then now you'll be working with an in-memory List<Customer> without executing the query over again.
On the contrary, if each time you'd do:
var filteredCustomers = customerSource.Where(customer => customer.Code.Equals("code1"))
foreach (var customer in filteredCustomers)
{
// Do stuff
}
Then for each enumeration you'll be exeucting the said query over again.
Also is it more efficient (time wise because memory wise it is
probably not) to just keep the get_all() method and use linq to filter
the results? Or use my existing setup which in effect executes
That really depends on your use-case. Lets imagine you were using LINQ to EF, and the customer table has a million rows, do you really want to be bringing all of them in-memory and only then filtering them out to use a subset of data? It would usually be better to full filtered query.

C# - Concatenate an in memory IList and IQueryable?

Suppose I have a List containing one string value. Suppose I also have an IQueryable that contains several strings from a database. I want to be able to concatenate these two containers into one list and then be able to call methods such as .Skip or .Take on the list. I want to be able to do this in such a way that when I combine the two containers I don't load all of the DB data into memory (only after I call .Skip and .Take). Basically, I want to do something like this (pseudocode):
IQueryable someQuery = myEntities.GetDBQuery(); // Gets "test2", "test3"
IList inMemoryList = new List();
inMemoryList.Add("test");
IList finalList = inMemoryList.Union(someQuery) // Can I do something like this without loading DB data into memory? finalList should contain all 3 strings.
// At this point it is fine to load the filtered query into memory.
foreach (string myString in finalList.Skip(100).Take(200))
{
// Do work...
}
How can I achieve this?

If I didn't misunderstand, you are trying to query the data, part of which comes from memory and others from database, like this:
//the following code will not compile, just for example
var dbQuery = BuildDbQuery();
var list = BuildListInMemory();
var myQuery = (dbQuery + list).OrderBy(aa).Skip(bb).Take(cc).Select(dd);
//and you don't want to load all records into memory by dbQuery
//because you only need some of them
The short answer is NO, you can't. Consider the .OrderBy method, all data have to be in a same "place", otherwise the code can't sort them. So the code loads all records in database by dbQuery into memory(now they are in a same place) and then sorts all of them including those in list. That probably causes a memory issue when dbQuery gives thousands of rows.
HOW TO RESOLVE
Pass the data in list into database (as parameters of dbQuery) so that the query happens in database. This is easy if your list has only a few items.
If list also has lots of records that will makes dbQuery too complex, you can try to query twice, one for dbQuery and one for list. For example, you have 10,000 users in database and 1,000 users in your memory list, and you want to get the top 10 youngest users. You don't need to load 10,000 users into memory and then find the youngest 10. Instead, you find 10 youngest (ResultA) in dbQuery and load into memory, and 10 youngest (ResultB) in memory list, and then compare between ResultA and ResultB.

I entirely agree with Danny's answer when he says you need to somehow find a way to include in memory user list into db so that you achieve what you want. As for the example which you sought in your comment, without knowing data structure of your User object, seems difficult. However assuming you would be able to connect the dots. Here is my suggested approach:
Create temporary table with identical structure that of your regular user table in your db and insert all your inmemory users into it
Write a query to Union temporary and regular table both identical in structure so that should be easy.
Return the result in your application and use it performing standard Linq operations
If you want exact code which you can use as it is then you will have to provide your User object structure - fields type etc in db to enable me to write the code.

You specify that your query and your list are both sequences of strings. someQuery can be performed completely on the database side (not in-memory)
Let's make your sequences less generic:
IQueryable<string> someQuery = ...
IList<string> myList = ...
You also specify that myList contains only one element.
string myOneAndOnlyString = myList.Single();
As your list is in-memory, this has to be performed in-memory. But because the list has only one element, this won't take any time.
The query that you request:
IQueryable<string> correctQuery = someQuery
.Where(item => item.Equals(myOneandOnlyString)
.Skip(skipCount)
.Take(takeCount)
Use your SQL server profiler to check the used SQL and see that the request is completely performed in one SQL statement.

Linq to XML how many times is this XML file getting read?

My site navigation has a concept of categories that have a description, image, and pages associated with them.
In _ViewStart.cshtml I have the following LINQ query and then store the results in PageData because I might be using the categories more than once on a given page and didn't want to re-run the query.
XDocument navigation = XDocument.Load(Server.MapPath("~/App_Data/Navigation.xml"));
IEnumerable<Category> categories = from category in navigation.Root.Descendants("category")
select new Category(
category.Attribute("name").Value,
category.Element("description").Value,
new CategoryImage(
category.Element("image").Element("path").Value,
category.Element("image").Element("cssClass").Value,
category.Element("image").Element("description").Value
),
(from page in category.Descendants("page") select new BetterSolutions.ViewModels.ProductPage(page.Attribute("display").Value, page.Value)).ToList()
);
PageData["categories"] = categories;
When I watch what happens through the debugger, anytime I access PageData["categories"] it keeps going back to the query in _ViewStart.cshtml.
When I change the above code by adding parenthesis around the LINQ query and adding .ToList() at the end, it appears to execute once and then never again.
What is the way I should be doing this? I think that adding the .ToList() is correct since the query appears to be only running once, but I might be misunderstanding how deferred execution within LINQ to XML is actually working.

What is the way I should be doing this? I think that adding the .ToList() is correct since the query appears to be only running once, but I might be misunderstanding how deferred executing within LINQ to XML is actually working.
What you see is deferred execution. When you create your categories it's not a collection of items: it's just a query which will be executed when results are needed. And that query definition is stored in PageData["categories"]. So every time to take it from there you have to execute it to get results.
Adding ToList() makes the results necessary right away, because you ask for list of results. And after that that list is stored in PageData["categories"]. That's why you don't have to execute query over and over again: because you already have the results stored in a list, and you don't even know where results came from before.
What is the right way to go? It depends. If you expect the file to change between PageData["categories"] calls and you need it to always return accurate results, you may stay with your current solution. If file does not change or it's OK not to read the file just once and discard all changes made to the file during program execution you should use ToList() to make it performing better and avoid unnecessary file access.
Update
My main answer is not completely correct. Even without ToList() accessing PageData["categories"] would not make the file access again, because the file is already completely loaded and parsed into XDocument instance. But it would travers the document itself to execute the query.

it Looks like you forgot to put to ToList() call for categories:
the simple fix would be :
PageData["categories"] = categories.ToList();