I'm getting into using the Repository Pattern for data access with the Entity Framework and LINQ as the underpinning of implementation of the non-Test Repository. Most samples I see return AsQueryable() when the call returns N records instead of List<T>. What is the advantage of doing this?
AsQueryable just creates a query, the instructions needed to get a list. You can make futher changes to the query later such as adding new Where clauses that get sent all the way down to the database level.
AsList returns an actual list with all the items in memory. If you add a new Where cluse to it, you don't get the fast filtering the database provides. Instead you get all the information in the list and then filter out what you don't need in the application.
So basically it comes down to waiting until the last possible momment before committing yourself.
Returning IQueryable<T> has the advantage, that the execution is defferer until you really start enumerating the result and you can compose the query with other queries and still get server side execution.
The problem is that you cannot control the lifetime of the database context in this method - you need an open context and must ensure that it stays open until the query gets executed. And then you must ensure that the context will be disposed. If you return the result as a List<T>, T[], or something similar, you loose deffered execution and server side execution of composed queries, but you win control over the lifetime of the database context.
What fits best, of course, depends on the actual requirements. It's another question without a single truth.
AsQueryable is an extension method for IEnumerable<T> that could do two things:
If the IEnumerable<T> implements IQueryable<T> justs casts, doing nothing.
Otherwise creates a 'fake' IEnumerable<T> (EnumerableQuery<T>) that implements every method compiling the lambdas and calling to Enumerable extension methods.
So in most of the cases using AsQueryable is useless, unless u are forced to pass a IQueryable to a method and u have a IEnumerable instead, it's a hack.
NOTE: AsQueryable is a hack, IQueryable of course is not!
Returning IQueryable<T> will defer execution of the query until its results are actually used. Until then, you can also perform additional database query operations on the IQueryable<T>; on a List you're restricted to generally less-efficient in-memory operations.
IQueryable: This is kind of deferred execution - lazy loading so your query is evaluated and hit to database. If we add any additional clause it will hit the Db with the required query with filters.
IEnumerable: Is eager loading thus records will be loaded first in-memory operation and then the operation will be performed.
So depending on use case like while paging on a huge number of records we should use IQueryable<T>, if short operation and doesn't create huge memory storage use IEnumerable.
Related
Long story short, we have a C# WebApi that provides mapped instances from our DAL into DTOS.
Each controller exposes a IEnumerable<T> Get()
I have a test that finds all the controllers, and call this get method.
This flushes out any mapping errors that only occur on execution.
A while I noticed that to be sure you had to actually execute the enumeration otherwise the the mapping never takes place.
So I have this code
if (results is IEnumerable enumerable)
{
var counter = enumerable.Cast<object>().Count();
Trace.Write($"{counter} results found.");
}
This executes the cast (which invokes the mapping) and gives me a count for output reasons.
This worked until the result set got massive - it nows just times out.
Most of my IEnumerables are actually IQueryable underneath
I need a way to Take(10) on an IQueryable that can work on reflected data/executed types. This will let the mapping work but on a smaller data set.
Help!
If I only have to use that result one time i.e. in iterating and displaying in view. I generally do 1st one, is it wiser?
IEnumerable<model> result = _someService.GetValues();
return View(result);
OR
IList<model> result = _someService.GetValues().ToList();
return View(result);
I checked in profiler and db is queried only once for that data using EF6.
To clear its a plain SELECT and both has data and there is no adjacent list to kick in lazy loading.
Note: I know IEnumerable has one method GetEnumerator which executes everytime its called it executes for data whereas ToList() executes only once.
See Programmers.SE: Why should I use List<T> over IEnumerable<T>?:
There are times when doing a ToList() on your linq queries can be important to ensure your queries execute at the time and in the order that you expect them to.
This is exactly the case here.
If _someService.GetValues() returns an IEnumerable<model> which is an Entity Framework query through deferred execution, calling ToList() in your controller will execute the query there: in your controller, where you can handle the exceptions that may occur. For example a query timeout.
If you don't call .ToList(), that same exception will be thrown instead at #foreach (var item in Model) in your view. You don't want that.
Also, if you use lazy loading, the context will be disposed by the time the query result end up in your view and throw an exception again, so in that case you'll have to materialize the query results in your controller by calling .ToList().
People get all high up in a tree when it comes to IEnumerable<T> versus IList<T>. They are generally parroting each other from some general advice (found here, amongst others), but they do not understand that that advice is for API design. You are not designing an API, you are implementing an application with very specific requirements: the view may never throw an exception. In order to fulfill that requirement, you must execute your queries in your controller, hence call .ToList() there.
The user asking the question linked above is right in the comment they posted:
all queries should be executed so that the model is done and loaded ready for the view. I.e. the view should receive everything and not be querying the database.
Razor rendering your model to your view may not have any side-effects. Executing a query is a side-effect, so that may not happen while rendering your view.
Even when calling .ToList() in your controller, your #model directive in the view can and should still be of IEnumerable<T> though!
Problem: I have millions of rows from database to be processed.
I need to implement a method that will return a "stream"(?) of database rows.
I don't want to load all of them into memory at once.
I was thinking about returning a lazy IEnumerable<Record> and use yield.
The method would handle loading consecutive records using SqlDataReader.
But what will happen when a client will call .Count() on my IEnumerable? Counting all the records would mean a need to fetch them all.
Is there any good modern way to return a stream of objects not storing all of them in memory, just process one by one? My method should return a stream of records.
It seems like Reactive Extensions might solve the problem for me but I have never used it.
Any ideas?
Thanks
First, why reinvent the wheel? Entity Framework makes it easier to do stuff like this and adds all of the abstraction for you. The DbSet<TEntity> on the DbContext object implements IQueryable<TEntity> and IEnumerable<T> so you can:
Execute a Count() (with and without a lamda filter argument) with an extension method when you need to figure out the number of records (or some other aggregate function)
You can loop through them as an IEnumerable which opens a connection and reads 1 record at a time each time the MoveNext method is called from the connection.
If you do want to load everything in memory at once (I understand you don't based on your description) you could call extension method ToList or ToArray.
If you insist on using ADO.NET and manually doing this (I understand with legacy code there is not always a choice to use EF) then opening a data reader from the connection is the best approach. This will read in each next record with each corresponding call to method Read() and this is the least expensive way to go about doing a read of records in the DB.
If you want a Count then I suggest you write a new sql query which returns a count executed on your Database Server using Sql akin to
SELECT COUNT(field) FROM table
as this is best practice. Do not iterate and sum up all of your records from a reader with some custom work around to execute a sum in memory, that would be waste of resources not to mention creating complex code with no benefit.
For count query the db, and return to the user.
On the other hand, you need to implement count only for ICollection, IEnumerable does not require that. just return IEnumerable for iteration on the records.
just notice that you handle correctly the connection to the db.
If I run against dlinq, for example
var custq = data.StoreDBHelper.DataContext.Customers as IEnumerable <Data.SO.Customer>;
I Thought it was not much of a difference against running:
var custq = data.StoreDBHelper.DataContext.Customers as IQueryable <Data.SO.Customer>;
As, IQueryable inherits from IEnumerable.
But I discovered the following, if you call:
custq.Sum()
then the program will process this as you called .toList() it you use the 'as IEnumerable'
because the memory on the progam raised to the same level, when i tried, custq.ToList.Sum()
but not on the 'as IQueryable' (because the issue then running on sql server) and did not affect the memory usage of the program.
My question is simply this, should you not use 'as IEnumerable' on Dlinq? But 'as IQueryable' as an general rule? I know that if you are running standard iterations, it gets the same result, between 'as IEnumerable'and 'as IQueryable'.
But is it just the summary functions and delegate expressions in where statement that there will be a difference - or will you in general get better performance if you use 'as IQueryable'? (for standard iteration and filter functions on DLinq entities)
Thanks !
Well, depends on what you want to do...
Casting it as IEnumerable will return an object you can enumerate... and nothing more.
So yes, if you call Count on an IEnumerable, then you enumerate the list (so you actually perform your Select query) and count each iteration.
On the other hand, if you keep an IQueryable, then you may enumerate it, but you could also perform database operations like Were, OrderBy or count. Then this will delay execution of the query and eventually modify it before running it.
Calling OrderBy on an enumerable browse all results and order them in memory. Calling OrderBy on a queryable simply adds ORDER BY at the end of your SQL and let database do the ordering.
In general, it is better to keep it as an IQueryable, yes... Unless you want to count them by actually browsing them (instead of doing a SELECT COUNT(*)...)
When using .FromCache() on an IQueryable result set, should I additionally call .ToList(), or can I just return the IEnumerable<> returned by the materialized query with FromCache?
I am assuming you are using a derivative of the code from http://petemontgomery.wordpress.com/2008/08/07/caching-the-results-of-linq-queries/ . If you look at the FromCache implementation, you will see the that the query.ToList() is already called. This means that the evaluated list is what is cached. So,
You do NOT need to call ToList()
That depends entirely on what you want to do with it. If you're just going to foreach over it once then you may as well just leave it as an IEnumerable. There's no need to build up a list just to discard it right away.
If you plan to iterate over it multiple times it's probably best to ToList it, so that you're not accessing the underlying IQueryable multiple times. You should also ToList it if it's possible for the underlying query to change over time and you don't want those changes to be reflected in your query.
If you are likely to not need to iterate all of the items (you may end up stopping after the first item, or half way, or something like that) then it's probably best to leave it as an IEnumerable to potentially avoid even fetching some amount of data in the first place.
If the method has no idea how it's going to be used, and it's just a helper method that will be used by not-yet-written code, then consider returning IEnumerable. The caller can call ToList on it if they have a compelling reason to turn it into a list.
For me, as a general rule, I leave such queries as IEnumerable unless I have some compelling reason to make it a List.