I need to insert around 2500 rows using EF Code First.
My original code looked something like this:
foreach(var item in listOfItemsToBeAdded)
{
//biz logic
context.MyStuff.Add(i);
}
This took a very long time. It was around 2.2 seconds for each DBSet.Add() call, which equates to around 90 minutes.
I refactored the code to this:
var tempItemList = new List<MyStuff>();
foreach(var item in listOfItemsToBeAdded)
{
//biz logic
tempItemList.Add(item)
}
context.MyStuff.ToList().AddRange(tempItemList);
This only takes around 4 seconds to run. However, the .ToList() queries all the items currently in the table, which is extremely necessary and could be dangerous or even more time consuming in the long run. One workaround would be to do something like context.MyStuff.Where(x=>x.ID = *empty guid*).AddRange(tempItemList) because then I know there will never be anything returned.
But I'm curious if anyone else knows of an efficient way to to a bulk insert using EF Code First?
Validation is normally a very expensive portion of EF, I had great performance improvements by disabling it with:
context.Configuration.AutoDetectChangesEnabled = false;
context.Configuration.ValidateOnSaveEnabled = false;
I believe I found that in a similar SO question--perhaps it was this answer
Another answer on that question rightly points out that if you really need bulk insert performance you should look at using System.Data.SqlClient.SqlBulkCopy. The choice between EF and ADO.NET for this issue really revolves around your priorities.
I have a crazy idea but I think it will help you.
After each adding 100 items call SaveChanges. I have a feeling Track Changes in EF have a very bad performance with huge data.
I would recommend this article on how to do bulk inserts using EF.
Entity Framework and slow bulk INSERTs
He explores these areas and compares perfomance:
Default EF (57 minutes to complete adding 30,000 records)
Replacing with ADO.NET Code (25 seconds for those same 30,000)
Context Bloat- Keep the active Context Graph small by using a new context for each Unit of Work (same 30,000 inserts take 33 seconds)
Large Lists - Turn off AutoDetectChangesEnabled (brings the time down to about 20 seconds)
Batching (down to 16 seconds)
DbTable.AddRange() - (performance is in the 12 range)
As STW pointed out, the DetectChanges method called every time you call the Add method is VERY expensive.
Common solution are:
Use AddRange over Add
SET AutoDetectChanges to false
SPLIT SaveChanges in multiple batches
See: Improve Entity Framework Add Performance
It's important to note that using AddRange doesn't perform a BulkInsert, it's simply invoke the DetecthChanges method once (after all entities is added) which greatly improve the performance.
But I'm curious if anyone else knows of an efficient way to to a bulk
insert using EF Code First
There is some third party library supporting Bulk Insert available:
See: Entity Framework Bulk Insert library
Disclaimer: I'm the owner of Entity Framework Extensions
This library allows you to perform all bulk operations you need for your scenarios:
Bulk SaveChanges
Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge
Example
// Easy to use
context.BulkSaveChanges();
// Easy to customize
context.BulkSaveChanges(bulk => bulk.BatchSize = 100);
// Perform Bulk Operations
context.BulkDelete(customers);
context.BulkInsert(customers);
context.BulkUpdate(customers);
// Customize Primary Key
context.BulkMerge(customers, operation => {
operation.ColumnPrimaryKeyExpression =
customer => customer.Code;
});
EF is not really usable for batch/bulk operations (I think in general ORMs are not).
The particular reason why this is running so slowly is because of the change tracker in EF. Virtually every call to the EF API results in a call to TrackChanges() internally, including DbSet.Add(). When you add 2500, this function gets called 2500 times. And each call gets slower and slower, the more data you have added. So disabling the change tracking in EF should help a lot:
dataContext.Configuration.AutoDetectChangesEnabled = false;
A better solution would be to split your big bulk operation into 2500 smaller transactions, each running with their own data context. You could use msmq, or some other mechanism for reliable messaging, for initiating each of these smaller transactions.
But if your system is build around a lot a bulk operations, I would suggest finding a different solution for your data access layer than EF.
While this is a bit late and the answers and comments posted above are very useful, I will just leave this here and hope it proves useful for people who
had the same problem as I did and come to this post for answers. This post
still ranks high on Google (at the time of posting this answer) if you search for a way
to bulk-insert records using Entity Framework.
I had a similar problem using Entity Framework and Code First in an MVC 5 application. I had a user submit a form that caused tens of thousands of records
to be inserted into a table. The user had to wait for more than 2 and a half minutes while 60,000 records were being inserted.
After much googling, I stumbled upon BulkInsert-EF6 which is also available
as a NuGet package. Reworking the OP's code:
var tempItemList = new List<MyStuff>();
foreach(var item in listOfItemsToBeAdded)
{
//biz logic
tempItemList.Add(item)
}
using (var transaction = context.Transaction())
{
try
{
context.BulkInsert(tempItemList);
transaction.Commit();
}
catch (Exception ex)
{
// Handle exception
transaction.Rollback();
}
}
My code went from taking >2 minutes to <1 second for 60,000 records.
Although a late reply, but I'm posting the answer because I suffered the same pain.
I've created a new GitHub project just for that, as of now, it supports Bulk insert/update/delete for Sql server transparently using SqlBulkCopy.
https://github.com/MHanafy/EntityExtensions
There're other goodies as well, and hopefully, It will be extended to do more down the track.
Using it is as simple as
var insertsAndupdates = new List<object>();
var deletes = new List<object>();
context.BulkUpdate(insertsAndupdates, deletes);
Hope it helps!
EF6 beta 1 has an AddRange function that may suit your purpose:
INSERTing many rows with Entity Framework 6 beta 1
EF6 will be released "this year" (2013)
public static void BulkInsert(IList list, string tableName)
{
var conn = (SqlConnection)Db.Connection;
if (conn.State != ConnectionState.Open) conn.Open();
using (var bulkCopy = new SqlBulkCopy(conn))
{
bulkCopy.BatchSize = list.Count;
bulkCopy.DestinationTableName = tableName;
var table = ListToDataTable(list);
bulkCopy.WriteToServer(table);
}
}
public static DataTable ListToDataTable(IList list)
{
var dt = new DataTable();
if (list.Count <= 0) return dt;
var properties = list[0].GetType().GetProperties();
foreach (var pi in properties)
{
dt.Columns.Add(pi.Name, Nullable.GetUnderlyingType(pi.PropertyType) ?? pi.PropertyType);
}
foreach (var item in list)
{
DataRow row = dt.NewRow();
properties.ToList().ForEach(p => row[p.Name] = p.GetValue(item, null) ?? DBNull.Value);
dt.Rows.Add(row);
}
return dt;
}
Related
I have 200k rows in my table and I need to filter the table and then show in datatable. When I try to do that, my sql run fast. But when I want to get row count or run the ToList(), it takes long time. Also when I try to convert it to list it has 15 rows after filter, it has not huge data.
public static List<Books> GetBooks()
{
List<Books> bookList = new List<Books>();
var v = from a in ctx.Books select a);
int allBooksCount = v.Count(); // I need all books count before filter. but it is so slow is my first problem
if (isFilter)
{
v = v.Where(a => a.startdate <= DateTime.Now && a.enddate>= DateTime.Now);
}
.
.
bookList = v.ToList(); // also toList is so slow is my second problem
}
There's nothing wrong with the code you've shown. So either you have some trouble in the database itself, or you're ruining the query by using IEnumerable instead of IQueryable.
My guess is that either ctx.Books is IEnumerable<Books> (instead of IQueryable<Books>), or that the Count (and Where etc.) method you're calling is the Enumerable version, rather than the Queryable version.
Which version of Count are you actually calling?
First, to get help you need to provide quantitative values for "fast" vs. "too long". Loading entities from EF will take longer than running a raw SQL statement in a client tool like TOAD etc. Are you seeing differences of 15ms vs. 15 seconds, or 15ms vs. 150ms?
To help identify and eliminate possible culprits:
Eliminate the possibility of a long-running DbContext instance tracking too many entities bogging down performance. The longer a DbContext is used and the more entities it tracks, the slower it gets. Temporarily change the code to:
List<Books> bookList = new List<Books>();
using (var context = new YourDbContext())
{
var v = from a in context.Books select a);
int allBooksCount = v.Count(); // I need all books count before filter. but it is so slow is my first problem
if (isFilter)
{
v = v.Where(a => a.startdate <= DateTime.Now && a.enddate>= DateTime.Now);
}
.
.
bookList = v.ToList();
}
Using a fresh DbContext ensures queries are not sifting through in-memory entities after running a query to find tracked instances to return. This also ensures we are running against IQueryable off the Books DbSet within the context. We can only guess what "ctx" in your code actually represents.
Next: Look at a profiler for MySQL, or have your database log out SQL statements to capture exactly what EF is requesting. Check that the Count and ToList each trigger just one query against the database, and then run these exact statements against the database. If there are more than these two queries being run then something odd is happening behind the scenes that you need to investigate, such as that your example doesn't really represent what your real code is doing. You could be tripping client side evaluation (if using EF Core 2) or lazy loading. The next thing I would look at is if possible to look at the execution plan for these queries for hints like missing indexes or such. (my DB experience is primarily SQL Server so I cannot provide advice on tools to use for MySQL)
I would log the actual SQL queries here. You can then use DESCRIBE to look at how many rows it hits. There are various tools that can further analyse the queries if DESCRIBE isn't sufficient. This way you can see whether it's the queries or the (lack of) indices that is the problem. Next step has to be guided by that.
I have a function in my asp.net core app which updates a bunch of records based on a certain criteria I write in a where clause ... I read that ToList() has bad performance , so is there a better and faster way than using tolist and foreach ???
This is my current way doing it , I would appreciate it if someone provides a more efficient way
public async Task UpdateCatalogOnTenantApproval(int tenantID)
{
var catalogQuery = GetQueryable();
var catalog = await catalogQuery.Where(x => x.IdTenant == tenantID).ToListAsync();
catalog.ForEach(c => { c.IsApprovedByAdmin = true; c.IsActive = true; });
Context.UpdateRange(catalog);
await Context.SaveChangesAsync(); ;
}
read that ToList() has bad performance ,
That is wrong. ToList has as good a performance as you will get - submit a bad query which is overly complex and which results in bad SQL that SQL Server will take ages to execute and it is slow.
Also, many people think "ToList" is slow (as in: in the profiler). You see, yo ustart with a db context, take a set of entities there, add some where clauses - all fast. Then ToList and it takes "long" (compared to the rest). Well, THAT is where the query is sent to the sql server ;) WHere (x=>whatever) takes "no time" because all it does is add some nodes to the expression tree, not executing the query. THAT is mostly what people mix up - delayed execution which exeutes only when asked for the results.
And third, some people like "ToList().Where() and complain about performance. Filter as much as possible no the DB.
All three reasons are why people think ToList is slow - but all it shows is a lack of understanding of how LINQ and SQL operate.
Entity Framework does not handle bulk update operations by default -- hence your existing code. If you really want to do these bulk operations, then you have two options:
Write the SQL yourself and use the ExecuteSqlCommand() method
to execute it; or
Look at 3rd party extensions, such as https://entityframework-extensions.net/
We can reduce query cost by selecting a subset of data before attaching for EF to track, and then updating.
However, it may be just pointless micro-optimization that does not perform significantly better unless you are processing massive amount of records.
// select pk for EF to track, and the 2 fields to be modified
var catalog = await catalogQuery.Where(x => x.IdTenant == tenantID)
.Select(x => new Catelog{x.CatelogId, x.IsApprovedByAdmin, x.IsActive }).ToListAsync();
//next we attach range here to let EF track the list
Context.AttachRange(catalog);
//perform your update as usual, this will be flagged as modified if changed
catalog.ForEach(c => { c.IsApprovedByAdmin = true; c.IsActive = true; });
//save and let EF update based on modified fields.
await Context.SaveChangesAsync();
Let me explain to you what you have done and what you are trying to do.
You are partially right about the performance issues related to ToList and ToListAsync as they are mainly responsible to upload entities to the memory and track them.
Based on that if your request is expected to deal intensively with light data you are not required to enhance your code. if it is not, however, there are many open approaches each one has its pros and cons and you have to treat and balance between them for each case you do not want to use the dual app-SQL requests.
let's be more realistic by talking about your case:
1- we assume that your method is a resource-consuming by (loading high volume of data, intensively called, or both)
2- I see the modification is too static by updating all of the rows by c.IsApprovedByAdmin = true; c.IsActive = true;
form (1) and (2) I suggest to write a stored procedure or ExexcuteSqlCammand (as Bryan Lewis suggested) that does this for you
because (3) the stored procedures, triggers, and all the SQL based operation are hard-maintainable and are highly potential for hidden exceptions. In your case, however, you less likely to fell into that as your code is too basic and you could reduce more the risk by construct your query from dynamic elements such as nameof(yourClassName that is the table name).YouProperty and the like ...
Anyway, this is an example to show that there is no ideal approach and you have study each case alone.
Finally, I do not agree with the 3d parties extensions as most of freely provided developed by unprofessionals and tracking exceptions caused by them are nightmares, and the paid versions are too expensive and not 0-exception extensions. The 3d party extension are more oriented to the complex bulk update/delete and/or huge data.
e.g.
await Context.UpdateAsync(e=> new Catalog
{ Archived = e.LastUpdate >
DateTime.UtcNow.AddYears(-99)? false : true
});
Would an Entity Framework LINQ-to-Entities query return all records (even 10 million rows) from a database, or would there be any limitation on retrieval record size?
Entity Framework and LINQ don't have any limitations for how many rows they can fetch. A problem you might face is making your server out of memory, since you're trying to retrieve that amount of data at once.
You should consider using something like Dapper as Valkyriee mentioned in the comments, or at least disable proxy if you still want to use Entity Framework:
using(var db = new MyDbContext())
{
db.Configuration.ProxyCreationEnabled = false;
var data = db.Users.ToList(); // suppose you have 10 milion users
}
...just be aware of what disabling proxy will cause. I'd still recommend using Dapper for this purpose.
Normally fetching 10 million records from database at one shot is not a good practice. You can use Entities recommended pagination functionalities.
I'm using Microsoft SQL Server and Entity Framework. I have N (for example 10 000) items to insert. Before inserting each item I need to insert or update existing group. It doesn't work well because of low performance. It's because I'm generating too many queries. Each time in loop I'm looking for group by querying Groups table by three (already indexed) parameters.
I was thinking about querying first all groups by using WHERE IN query (Groups.Where(g => owners.Contains(g.OwnerId) && .. ), but as I remember such queries are limited by number of parameters.
Maybe I should write a stored procedure?
Here is my example code. I'm using IUnitOfWork pattern for wrapping the EF DbContext:
public async Task InsertAsync(IItem item)
{
var existingGroup = await this.unitOfWork.Groups.GetByAsync(item.OwnerId, item.Type, item.TypeId);
if (existingGroup == null)
{
existingGroup = this.unitOfWork.Groups.CreateNew();
existingGroup.State = GroupState.New;
existingGroup.Type = item.Code;
existingGroup.TypeId = item.TypeId;
existingGroup.OwnerId = item.OwnerId;
existingGroup.UpdatedAt = item.CreatedAt;
this.unitOfWork.Groups.Insert(existingGroup);
}
else
{
existingGroup.UpdatedAt = item.CreatedAt;
existingGroup.State = GroupState.New;
this.unitOfWork.Groups.Update(existingGroup);
}
this.unitOfWork.Items.Insert(item);
}
foreach(var item in items)
{
InsertAsync(item);
}
await this.unitOfWork.SaveChangesAsync();
There are three key elements to improve performance when bulk inserting:
Set AutoDetectChangesEnabled and ValidateOnSaveEnabled to false:
_db.Configuration.AutoDetectChangesEnabled = false;
_db.Configuration.ValidateOnSaveEnabled = false;
Break up your inserts into segments, wich use the same DbContext, then recreate it. How large the segment should be varies from use-case to use-case, I made best performance at around 100 Elements before recreating the Context. This is due to the observing of the elements in the DbContext.
Also make sure not to recreate the context for every insert.
(See Slauma's answer here Fastest Way of Inserting in Entity Framework)
When checking other tables, make sure to use IQueryable where possible and to work only where necessary with ToList() or FirstOrDefault(). Since ToList() and FirstOrDefault() loads the objects. (See Richard Szalay's answer here What's the difference between IQueryable and IEnumerable)
These tricks helped me out the most when bulk inserting in a scenario as you described. There are also other possibilities. For example SP's, and the BulkInsert function.
We are using EF 6.0, .NET 4.5 and using code first approach and our database has around 170 entities(tables) and the main table holding around 150,000 records
On first load of the entity framework it takes around 25 seconds.
I am trying to improve this time as this is too slow and as the number of records increases it becomes slower.
I have tried generating native images, tried using pre generated interactive views but I couldn't achieve any significant improvements.
Can anyone please help me on this?
Thanks.
You can consider the Entity Framework Pre-Generated Mapping Views.You can use EF Power Tools to create pre-generate views.
Using pre-generated views moves the cost of view generation from model
loading (run time) to compile time. While this improves startup
performance at runtime, you will still experience the pain of view
generation while you are developing. There are several additional
tricks that can help reduce the cost of view generation, both at
compile time and run time.
You can refer this for knowing more about it : Entity Framework Pre-Generated Mapping Views
You can use Caching in the Entity Framework to improve the performance of your app.
There are 3 types of caching.
1. Object caching – the ObjectStateManager built into an ObjectContext
instance keeps track in memory of the objects that have been
retrieved using that instance. This is also known as first-level
cache.
2. Query Plan Caching - reusing the generated store command when a
query is executed more than once.
3. Metadata caching - sharing the metadata for a model across different
connections to the same model.
You can refer this article to read more about it : Performance Considerations for EF 6
I recently had a simple query that runs super quick in SSMS that was taking way, way too long to run using the Entity Framework in my C# program.
This page has been extremely helpful, when trouble shooting EF performance problems in general:
https://www.simple-talk.com/dotnet/net-tools/entity-framework-performance-and-what-you-can-do-about-it/
..but in this case, nothing helped. So in the end, I did this:
List<UpcPrintingProductModel> products = new List<UpcPrintingProductModel>();
var sql = "select top 75 number, desc1, upccode "
+ "from MailOrderManager..STOCK s "
+ "where s.number like #puid + '%' "
;
var connstring = ConfigurationManager.ConnectionStrings["MailOrderManagerContext"].ToString();
using (var connection = new SqlConnection(connstring))
using (var command = new SqlCommand(sql, connection)) {
connection.Open();
command.Parameters.AddWithValue("#puid", productNumber);
using (SqlDataReader reader = command.ExecuteReader()) {
while (reader.Read()) {
var product = new UpcPrintingProductModel() {
ProductNumber = Convert.ToString(reader["number"]),
Description = Convert.ToString(reader["desc1"]),
Upc = Convert.ToString(reader["upccode"])
};
products.Add(product);
}
}
}
(For this particular query, I just completely bypassed the EF altogether, and used the old standby: System.Data.SqlClient.)
You can wrinkle your nose in disgust; I certainly did - but it didn't actually take that long to write, and it executes almost instantly.
You also can circumvent this issue asynchronously "warming" your dbcontexts at application start moment.
protected void Application_Start()
{
// your code.
// Warming up.
Start(() =>
{
using (var dbContext = new SomeDbContext())
{
// Any request to db in current dbContext.
var response1 = dbContext.Addresses.Count();
}
});
}
private void Start(Action a)
{
a.BeginInvoke(null, null);
}
I also recommend use such settings as: (if they fit your app)
dbContext.Configuration.AutoDetectChangesEnabled = false;
dbContext.Configuration.LazyLoadingEnabled = false;
dbContext.Configuration.ProxyCreationEnabled = false;
Skip validation part ( ie Database.SetInitializer<SomeDbContext>(null);)
Using .asNoTraking() on GET queries.
For additional information you can read:
https://msdn.microsoft.com/en-in/data/hh949853.aspx
https://www.fusonic.net/en/blog/3-steps-for-fast-entityframework-6.1-code-first-startup-performance/
https://www.fusonic.net/en/blog/ef-cache-deployment/
https://msdn.microsoft.com/en-us/library/dn469601(v=vs.113).aspx
https://blog.3d-logic.com/2013/12/14/using-pre-generated-views-without-having-to-pre-generate-views-ef6/
In some cases EF does not use Query Plan Caching. For example if you use contans, any, all methods or use constants in you query. You can try NihFix.EfQueryCacheOptimizer. It convert your query expression that allow EF use cahce.