Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
I am adding multiple entities in the database using AddRange in Entity Framework:
foreach (string tagNumber in notPresent)
{
element = new TagMaster { Name = Guid.NewGuid().ToString(), IsActive = true };
element.TagCollections.Add(new TagCollection { TagNumber = tagNumber });
newTagMasters.Add(element);
}
dbContext.TagMasters.AddRange(newTagMasters);
dbContext.SaveChanges();
What I was expecting is that by adding the complete collection in context using AddRange method, there would be a single query that will be sent to database. But to my surprise, I see multiple insert statements for each record to be inserted.
Any Insights?
The problem you are running in is that sadly the entity framework commands know NO bulk inserts. Instead they generate 1 statement per line that you want to insert.
There is no workaround to this.
The only possiblity to get 1 single statement that does all the inserts is to use specific classes or libraries. As example here SqlBulkCopy which needs no external lib to be downloaded to work.
Here a link to the msdn site:
https://msdn.microsoft.com/de-de/library/system.data.sqlclient.sqlbulkcopy(v=vs.110).aspx
The usage is quite easy. You only give the constructor your connection (after opening it beforehand!) and tell it what it shall write to teh server and what the destination table name is. Then you only need to close the connection afterwards again.
sqlcon.Open();
using (SqlBulkCopy sqlBulkCopyVariable= new SqlBulkCopy(sqlcon))
{
sqlBulkCopyVariable.BulkCopyTimeout = 600; // 10 minutes timeout
sqlBulkCopyVariable.DestinationTableName = "targetTableName";
sqlBulkCopyVariable.WriteToServer(MyData);
}
sqlcon.Close();
The WriteToServer takes DataTable, DataReader or even arrays of DataRow. The exact implementation there would depend on how you want to give the data to it. So far from my personal experience: That class is quite fast and generates only 1 single statement. BUT it is only there for SqlClients. Thus if you have a different client you need to look up which class or external library would be best fitting for you.
I am afraid insertions through Linq is not optimized as you would expect. It does that by multiple insert statements as you observed. You could instead bypass Linq in those cases and use the bulk copying alternatives instead (ie: for MS SQL server use SqlBulkCopy class, for postgreSQL use Copy etc).
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
In a nutshell, my project is receiving data faster than it can process and then write it to a database (EF6 to SQL Server 2016), and I'm not sure what the best-practice approach is (ditch EF? Offload to database via Service Broker? Something else?) Write events are not being handled fast enough, so they result in cascading event logjams and fatal memory crashes.
The write events are (I want them to be) low-priority, and I'm using async tasks for them. The write events involve a lot of data and a lot of relationships, and EF is just not handling them efficiently (I'm using AddRange, but EF is just sending everything in many single inserts, which I've read is its regular behavior).
I've tried paring back the relationships, and I've moved more processing over to the database, and I've tried using a batched "Delayed"Queue (an observable queue implementation that triggers an "empty me" event when a threshold is met), so that the inbound write events can be handled very quickly (just dump the request in the queue and move on), but this didn't get me anywhere (not surprising, I suppose, since I've basically added a message queue on top of the built-in message queue?).
Please correct me if I'm wrong, but it seems to me that EF is not the right tool for something as write-heavy and relationship-heavy as what I have (I know there are bulk-write extensions...). So, in an effort to resolve this sensibly, would it make sense to bypass EF and do my own bulk-write queries, or is this an appropriate use for Service Broker? With Service Broker, I could just send a dataset in one sproc, which just adds the dataset to the queue, frees the frontend to move on, and the database can handle and build the relationships whenever. Are these solutions sensible or best practice, or am I barking up the wrong tree (or putting lipstick-on-a-pig maybe)?
Thank you.
Please correct me if I'm wrong, but it seems to me that EF is not the
right tool for something as write-heavy and relationship-heavy as what
I have
You are right.
By default like you said, Entity Framework perform one database round-trip for every record to save which is INSANELY slow.
Disclaimer: I'm the owner of Entity Framework Extensions
(The library is not free)
This library allows you to improve Entity Framework Performance.
I'm not sure if our library can help you but it worth a try if you save multiple entities at once.
By example, the BulkSaveChanges is exactly like SaveChanges but way faster by dramatically reducing the database round-trip required.
Bulk SaveChanges
Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge
Example
// Easy to use
context.BulkSaveChanges();
// Easy to customize
context.BulkSaveChanges(bulk => bulk.BatchSize = 100);
// Perform Bulk Operations
context.BulkDelete(endItems);
context.BulkInsert(endItems);
context.BulkUpdate(endItems);
// Customize Primary Key
context.BulkMerge(endItems, operation => {
operation.ColumnPrimaryKeyExpression =
endItem => endItem.Code;
});
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
How to make AddOrUpdate() generating update t-sql?
How to make AddOrUpdate() generating t-sql sent to sql server in sequence?
How to create multiple columns 'reference' index in EF6?
come on
This is to gather all the solutions of pitfall I encountered on initializing
System.Data.Entity.Migrations.DbMigrationsConfiguration.Seed().
I am T-SQL master, but I'm not EF master due to I always on the MS SQL side.
I found DbContext.AddOrUpdate(new instance(){}) didn't generate the update T-sql, and throw exception when EF encountered inserting the repeat row error in sql server.
Here is the solution gave by roelant-m:
migrationdbsett addorupdate trouble
When I solved the AddOrUpdate(new Instance(){}) inserting same row problem, I encountered the EF optimizing problem, yeah, he is so smart that optimize all the AddOrUpdate( new instance()) in method Seed(). and then he throw exceptions.
Due to the Initializing database must be in sequence (some table has referenced the other tables.)
Here is the solution gave by endi zhupani:
how to make the addorupdate run in order in dbmigrationconfiguration derived
When I solved the initializing AddOrUpdate(new Instance(){}) in sequence, I encountered the multiple column 'references' index, some element of the index are navigation property.
Here is solution gave by sampath:
how to create multi columns reference index join index in csharp model
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I'm just starting out with server side code and Visual Studio with C#, but something I can't figure out with my googling is how do people use C# and Visual Studio to access a database and pull results and display them on the html documment?
I have an SQL Server 2012, I believe I know how to connect to the database itself, but unless I use classic ASP I don't understand how to pull it from the database and display it using C# and or ASP.NET.
I suggest you start out with Ado.Net. Once you have a good understanding of the principles, look into Entity Framework.
Robin Nadeau is right, what you'll probably want to start with is Ado.Net controls. In reference to your example, how I would go about it is using a DataGridView and binding it with a BindingSource + DataSet. When you drop a DataGridView on the form, it will give you a chance to run through a wizard to easily set up the BindingSource and Dataset.
To filter stuff from there, you'll need to hook up the buttons etc. with code such as this:
bindingSource1.Filter = string.Format("VideoTitle LIKE '{0}'", txtSearch.Text);
Hopefully that helps you get started.
If you are a beginner and don't have enough time to learn a complete framework. I would suggest to use the WebMatrix.Data Wrapper. Life's simple with it. 6 Commands is all you need to know to make 90% of the applications. Just install the Nuget Package and you're good to go.
Those 6 Commands are
1. Open & Close Connection
var db=Database.Open("name of connection string in web.config");
db.Close();
2. SELECT Query
foreach(var row in db.Query("SELECT * FROM tablename")
{
servervariable1=row.column1;
}
3. SELECT Single Row
var row=db.QuerySingle("SELECT * FROM tablename WHERE Key=#0",PassedVariable);
if(row!=null)
{
// do the operation
}
else
{
// handle code if the row is not found
}
4. SELECT Single Value
try{
datatype variable=db.QueryValue("SELECT columnname FROM tablename WHERE Key=#0",Key);
}
catch{
// handle code if row doesn't exists
}
5. INSERT Query
db.Execute("INSERT INTO tablename(column1,column2) Values(#0,#1)",inputvar1,inputvar2);
6. DELETE Query
db.Execute("DELETE * FROM tablename WHERE Key=#0",key);
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 8 years ago.
Improve this question
I am trying to do a fuzzy match of records in two Account table using .NET Entity Framework.
I wrote some code like this but it has bad performance like 1 min a record.
ARSalesforceEntities arsf = new ARSalesforceEntities(); //dbcontext
Salesforce_FFEntities ffsf = new Salesforce_FFEntities(); //dbcontext
var araccounts = arsf.Accounts; //dbset contains 400000 records
var ffaccounts = ffsf.Accounts; //dbset contains 6000 records
IDCONV byName = new IDCONV();
IDCONV byAddress = new IDCONV();
foreach (var ffaccount in ffaccounts)
{
Console.WriteLine(++count);
foreach (var araccount in araccounts)//this line goes every slow like 1 min
{
Basically, I am comparing the records in two tables with complicated logic.
How can I greatly improve the performance of the code?
Thank you
This line:
var ffaccounts = ffsf.Accounts;
is what's hitting you hard. You're basically assigning the IQueryable to a variable which, when accessed in your inner loop, re-queries the database everytime. I imagine simply adding ToList() on the end will drastically improve your performance:
var ffaccounts = ffsf.Accounts.ToList();
That's assuming of course that it's acceptable to materialise the 6000 rows into memory.
If not, then you might want to consider writing the logic in SQL and doing all the work in the DB instead...
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
Is it OK to always fire the create statements rather than checking them like
SELECT * FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_NAME = 'my_table_name'
in SQL SERVER CE and C#.
Because irrespective of the approach, there is always a single query that will be fired. Isnt it ???
I am using trasaction for creating all tables. So, I am sure if there is one table present, then definitely all of them are created.
So if i use the same transaction, then the transaction will fail right at the very first create table statement: But is this good performance wise???
Check if table exists or always fire create statement
If the table already exists in database, you will get an exception.
You can check for table existence, delete and then create it, something like:
if OBJECT_ID('dbo.my_table_name', 'U') IS NOT NULL
DROP TABLE dbo.my_table_name;
--and then
CREATE TABLE ....
If you mean try to create the table and look for an exception, then no. Exceptions are expensive, and if they are not handled properly can terminate your program unexpectedly.
The best practice would be to check for the table's existence.