I have a complex Entity Framework query. My performance bottleneck is not actually querying the database, but translating the IQueryable into query text.
My code is something like this:
var query = context.Hands.Where(...)
if(x)
query = query.where(...)
....
var result = query.OrderBy(...)
var page = result.skip(500 * pageNumber).Take(500).ToList(); //loong time here, even before calling the DB
do
{
foreach(var h in page) { ... }
pageNumber += 1;
page = result.skip(500 * pageNumber).Take(500).ToList(); //same here
}
while(y)
What can I do? I am using DbContext (with SQLite), so I can't use precompiled query (and even then, it would be cumbersome with query building algorithm like this).
What I basically need, is to cache a "page" query and only change the "skip" and "take" parameters, without recompiling it from the ground up each time.
Your premise is incorrect. Because you have a ToList call at the end of your query you are querying the database where you've indicated, to construct the list. You're not deferring execution any longer. That's why it takes so long. You aren't spending a long time constructing the query, it's taking a long time to go to the database and actually execute it.
If it helps you can use the following method to do the pagination for you. It will defer fetching each page until you ask for the next one:
public static IEnumerable<IEnumerable<T>> Paginate<T>(
this IQueryable<T> query, int pagesize)
{
int pageNumber = 0;
var page = query.Take(pagesize).ToList();
while (page.Any())
{
yield return page;
pageNumber++;
page = query.Skip(pageNumber * pagesize)
.Take(pagesize)
.ToList();
}
}
So if you had this code:
var result = query.OrderBy(...);
var pages = result.Paginate();//still haven't hit the database
//each iteration of this loop will go query the DB once to get that page
foreach(var page in pages)
{
//use page
}
If you want to get an IEnumerable<IQueryable<T>> in which you have all of the pages as queries (meaning you could add further filters to them before sending them to the database) then the major problem you have is that you don't know how many pages there will be. You need to actually execute a given query to know if it's the last page or not. You either need to fetch each page as you go, as this code does, or you need to query the count of the un-paged query at the start (which means one more DB query than you would otherwise need). Doing that would look like:
public static IEnumerable<IQueryable<T>> Paginate<T>(
this IQueryable<T> query, int pagesize)
{
//note that this is hitting the DB
int numPages = (int)Math.Ceiling(query.Count() / (double)pagesize);
for (int i = 0; i < numPages; i++)
{
var page = query.Skip(i * pagesize)
.Take(pagesize);
yield return page;
}
}
Related
I'm not sure External Source is the correct phrasing, but essentially I have a view in my database that points to a table in a different database. Not always, but from time to time I get an ORA-12537 Network Session: End of File exception. I'm using Entity Framework, so I tried breaking it up so instead of using one massive query, it does a handful of queries to generate the final result. But this has had a mixed-to-no impact.
public List<SomeDataModel> GetDataFromList(List<string> SOME_LIST_OF_STRINGS)
{
var retData = new List<SomeDataModel>();
const int MAX_CHUNK_SIZE = 1000;
var totalPages = (int)Math.Ceiling((decimal)SOME_LIST_OF_STRINGS.Count / MAX_CHUNK_SIZE);
var pageList = new List<List<string>>();
for(var i = 0; i < totalPages; i++)
{
var chunkItems = SOME_LIST_OF_STRINGS.Skip(i * MAX_CHUNK_SIZE).Take(MAX_CHUNK_SIZE).ToList();
pageList.Add(chunkItems);
}
using (var context = new SOMEContext())
{
foreach(var pageChunk in pageList)
{
var result = (from r in context.SomeEntity
where SOME_LIST_OF_STRINGS.Contains(r.SomeString)
select r).ToList();
result.ForEach(x => retData.Add(mapper.Map<SomeDataModel>(x)));
}
}
return retData;
}
I'm not sure if there's a different approach to dealing with this exception or not, or if breaking up the query has any desired effect. It's probably worth noting that SOME_LIST_OF_STRINGS is pretty large (about 21,000 on average), so totalPages usually sits around 22.
Sometimes, that error can be caused by an excessively large "IN" list in the SQL. For example:
SELECT *
FROM tbl
WHERE somecol IN ( ...huge list of stuff... );
Enabling application or database level tracing could help reveal whether the SQL that's being constructed behind the scenes has a large IN list.
A workaround might be to INSERT "...huge list of stuff..." into a table and then use something similar to the query below in order to avoid the huge list of literals.
SELECT *
FROM tbl
WHERE somecol IN ( select stuff from sometable );
Reference*:
https://support.oracle.com/knowledge/More%20Applications%20and%20Technologies/2226769_1.html
*I mostly drew my conclusions from the part of this reference that's not publicly viewable.
I'm having a problem with an application I made for my company. We are taking queries out of an ERP system. People can search for an article, and then the application shows them all relevant technical data, pictures and/or datasheets.
Problem is: it's loading very slow. Queries seem to run fine, but the generation of the view takes ages.
This is my search code (don't mind the Dutch parts):
public IQueryable<Item> GetItems(string vZoekString)
{
db.Configuration.LazyLoadingEnabled = false;
//Split de ZoekString
var searchTerms = vZoekString.ToLower().Split(null);
// TODO: alles in een db query
//Resultaten oplijsten
var term = searchTerms[0];
var results = db.item_general.Where(c => c.ITEM.Contains(term) || c.DSCA.Contains(term));
//Zoeken op alle zoektermen
if (searchTerms.Length > 1)
{
for (int i = 0; i < searchTerms.Length; i++)
{
var tempTerm = searchTerms[i];
results = results.Where(c => c.ITEM.Contains(tempTerm) || c.DSCA.Contains(tempTerm));
}
}
//Show
return results;
And then, these results are returned to the view like this:
public ActionResult SearchResults(string vZoekString, string filterValue, int? pageNo)
{
//Bewaking zoekstring
if (vZoekString != null)
{
pageNo = 1;
}
else
{
vZoekString = filterValue;
}
//De zoekstring doorgeven
if (vZoekString == null)
{
return RedirectToAction("Index", "Home");
}
else
{
ViewBag.ZoekString = vZoekString;
}
//Ophalen Items via Business-Object
//var vItems = new SearchItems().GetItems(vZoekString);
SearchItems vSearchItems = new SearchItems();
IQueryable<Item> vItems = vSearchItems.GetItems(vZoekString);
//Nummering
int pageSize = 10;
int page = (pageNo ?? 1);
//Show
return View(vItems.OrderBy(x => x.ITEM).AsNoTracking().ToPagedList(page, pageSize));
}
What can be wrong in my situation? Am I overlooking something?
UPDATE:
I've checked my code, and it seems that everything works very quickly, but it takes more than 10 seconds when it reaches .ToPagedList(). So my guess is that there is something wrong with that. I'm using the paged list from Nuget.
While I can't evaluate your view code without seeing it the problem could very well be in the database query.
An IQueryable does not actually load anything from the database until you use the results. So the database query will only be run after the View code has started.
Try changing the View call to:
var items = vItems.OrderBy(x => x.ITEM).AsNoTracking().ToPagedList(page, pageSize);
return View(items);
And then check to see if the View is still the bottleneck.
(This should probably be a comment instead, but I don't have the reputation....)
In most cases where you face performance issues with MVC and EF it is due to returning entities to views and getting stung by lazy loading. The reason for this is that when ASP.Net is told to send an object to the browser it needs to serialize it. The process of serialization iterates over the entity which touches lazy-load proxies, triggering those related entities to load one at a time.
You can detect this by running a profiler against your database, set a breakpoint point prior to the end of your action, then watch what queries execute as the action call returns. Lazy loading due to serialization will appear as a number of individual (TOP 1) queries being executed in rapid succession after the action has completed prior to the page rendering.
The simplest suggestion to avoid this pain is don't return entities from controllers.
IQueryable<Item> vItems = vSearchItems.GetItems(vZoekString);
var viewModels = vItems.OrderBy(x => x.ITEM)
.Select(x => new ItemViewModel
{
ItemId = x.ItemId,
// .. Continue populating view model. If model contains a hierarchy of data, map those to related view models as well.
}).ToPagedList(page, pageSize);
return View(viewModels);
The benefit of this approach:
The .Select() will result in a query that only retrieves the data you actually need to populate the view models. (The data your view needs) Faster query, less data across the wire between DB server -> App server -> Browser
This doesn't result in any lazy loads.
The caveat of this approach:
You need to take care that the .Select() expression will go to SQL, so no .Net or private functions for stuff like translating/formatting data. Populate raw values into the view model then expose properties on the view model to do the translation which will serialize that formatted data to the client. Basic stuff like FullName = x.FirstName + " " + x.LastName is fine, but avoid things like OrderDate = DateTime.Parse(x.DateAsISO) if the DB stored dates as strings for example.
You can leverage mappers like Automapper to assist with mapping between Entity and ViewModel. Provided the mapping tool inspects/traverses the destination to populate rather than the source, you should be good. Automapper does support integration within IQueryable so it would be a worthwhile investment to research that if you want to leverage a mapper.
We can implementing pagination in C#, ASP.NET in ActionResult function like this
public ActionResult Index(int? page)
{
Entities db = new Entities();
return View(db.myTable.ToList().ToPagedList(page ?? 1,8 ));
}
how to implementing pagination in JsonResult function that sending result to ajax in html view?
public JsonResult GetSearchingData(string SearchBy, string SearchValue)
{
var subCategoryToReturn = myList.Select(S => new { Name = S.Name });
return Json(subCategoryToReturn, JsonRequestBehavior.AllowGet);
}
stop thinking in terms of UI and start thinking in terms of data.
you have some data you want to paginate. That's it. Forget about MVC at this point or JSONResult or anything else that has nothing to do with the data.
One thing to be aware of, this code you posted above:
db.myTable.ToList().ToPagedList(page ??1,8 )
if you do it like this, your entire table will be returned from the database and then it will be paginated. What you want is to only return the data already paginated, so instead of returning 100 records and then only taking the first 20, only return the 20.
Don't use ToList() at that point, use something like this instead:
var myData = db.myTable.Skip(pageNumber * pageSize).Take(pageSize)
I don't have any running code to check this, but hopefully you get the idea, only return the data already paginated, so only return the data you will display and nothing more. The UI can send the page index you click on, the pageSize can be a predefined number stored in appSettings for example.
You can use, .Skip() for skipping first n elements and Take() for fetching the next n rows.
int pageNumber = 0;
int ItemsPerPage= 10;
var Results = db.myTable.ToList().Skip(ItemsPerPage* pageNumber).Take(numberOfObjectsPerPage);
Suppose that you only need 10 items Per page. Number of Pages would be equal to
TotalPages = Math.Ceiling(TotalItems/ItemsPerPage); // IF 201 Items then 201/10 ceiling = 21 pages
Now you can make Pagination buttons in html. I suggest you use Jquery Pagniation Library
https://esimakin.github.io/twbs-pagination/
I am running search query as following to bring results from Dynamics CRM. Search is working fine but it is brining results based on relevance. We want to order them in descending order of 'createdon' field. As we are displaying only 10 results per page, so I can't sort the result returned by this query.
Is there any way to order based on a field?
public IEnumerable<SearchResult> Search(string term, int? pageNumber, int
pageSize, out int totalHits, IEnumerable<string> logicalName)
{
var searchProvider = SearchManager.Provider;
var query = new CrmEntityQuery(term, pageNumber.GetValueOrDefault(1), pageSize, logicalNames);
return GetSearchResults(out totalHits, searchProvider, query);
}
private IEnumerable<SearchResult> GetSearchResults(out int totalHits,
SearchProvider searchProvider, CrmEntityQuery query)
{
using (ICrmEntityIndexSearcher searcher = searchProvider.GetIndexSearcher())
{
Portal.StoreRequestItem("SearchDeduplicateListForAuthorisation", new List<Guid>());
var results = searcher.Search(query);
totalHits = results.ApproximateTotalHits;
return from x in results
select new SearchResult(x);
}
}
Not used Lucene myself, so cant comment on that.
However, if you were doing this in basic CRM. You would use a QueryExpression with an OrderExpression. Then when you page the results they are paged in order.
Here is an example of a QueryExpression, with an OrderExpression, and paging.
Page large result sets with QueryExpression
Presumably at some point the data is being pulled out of CRM, either within Lucene, or your own code, maybe in CrmEntityQuery? Then you can add the sort there.
I am working on an application in which I would like to implement paging. I have the following class that implements detached criteria -
public class PagedData : DetachedCriteria
{
public PagedData(int pageIndex, int pageSize) : base(typeof(mytype))
{
AddOrder(Order.Asc("myId"));
var subquery = DetachedCriteria.For(typeof(mytype2))
.SetProjection(Projections.Property("mytype.myId"));
Add(Subqueries.PropertyIn("myId", subquery));
SetFirstResult((pageIndex - 1) * pageSize);
SetMaxResults(pageSize);
}
}
This works fine - it returns exactly the data that I am trying to retrieve. The problem I am running into is getting the total row count for my page navigation. since I am using the setfirstresults and setmaxresults in my detached criteria, the row count is always limited to the pageSize variable that is coming in.
My question is this: How can I get the total row count? Should I just create another detachedcriteria to calculate the row count? If so, will that add round trips to the db? Would I be better off not using detacedcriteria and using a straight criteria query in which I can then utilize futures? Or can I somehow use futures with what I am currently doing.
Please let me know if any further information is needed.
Thanks
I do it like this, inside my class which is used for paged criteria access:
// In order to be able to determine the NumberOfItems in a efficient manner,
// we'll clone the Criteria that has been given, and use a Projection so that
// NHibernate will issue a SELECT COUNT(*) against the ICriteria.
ICriteria countQuery =
CriteriaTransformer.TransformToRowCount (_criteria);
NumberOfItems = countQuery.UniqueResult<int> ();
Where NumberOfItems is a property (with a private setter) inside my 'PagedCriteriaResults' class.
The PagedCriteriaResults class takes an ICriteria instance in its constructor.
you can create a second DetachedCriteria to get to row count with the build-in CriteriaTransformer
DetachedCriteria countSubquery = NHibernate.CriteriaTransformer.TransformToRowCount(subquery)
this will of course result in a second call to the db
Discussed here:
How can you do paging with NHibernate?
Drawing on the two answers above i created this method for paged searching using detached criteria.
Basically i just take an ordinary detached criteria and after i've created the real ICriteria from the session, i transform it to a rowcount critera and then use Future on both of them. Works great!
public PagedResult<T> SearchPaged<T>(PagedQuery query)
{
try
{
//the PagedQuery object is just a holder for a detached criteria and the paging variables
ICriteria crit = query.Query.GetExecutableCriteria(_session);
crit.SetMaxResults(query.PageSize);
crit.SetFirstResult(query.PageSize * (query.Page - 1));
var data = crit.Future<T>();
ICriteria countQuery = CriteriaTransformer.TransformToRowCount(crit);
var rowcount = countQuery.FutureValue<Int32>();
IList<T> list = new List<T>();
foreach (T t in data)
{
list.Add(t);
}
PagedResult<T> res = new PagedResult<T>();
res.Page = query.Page;
res.PageSize = query.PageSize;
res.TotalRowCount = rowcount.Value;
res.Result = list;
return res;
}
catch (Exception ex)
{
_log.Error("error", ex);
throw ex;
}
}