I have a worker thread, whose job is to insert Objects that are stored in a Queue into the database.
We are currently using Entity framework to do the inserts. Now my question is, do I need to make a new Db Instance for every insert? or can I safely re-use the same db instance over and over?
private static void MainWorker()
{
while (true)
{
try
{
if (IncomingDataQueue.Any())
{
if (IncomingDataQueue.TryDequeue(out var items))
{
//Insert into db
using (var db = GetNewDbInstance())
{
if (db != null)
{
db.DataRaw.AddRange(items);
db.SaveChanges();
//Skip everything and continue to the next loop
continue;
}
}
}
}
}
catch (Exception ex)
{
Debug.WriteException("Failed to insert DB Data", ex);
//Delay here in case we are hitting the db 2 hard.
Thread.Sleep(100);
}
//Wait here as we did not have any items in the queue, so wait before checkign again
Thread.Sleep(20);
}
}
Here is my function which gets a new DB Instance:
private static DbEntities GetNewDbInstance()
{
try
{
var db = new DbEntities();
db.Configuration.ProxyCreationEnabled = false;
db.Configuration.AutoDetectChangesEnabled = false;
return db;
}
catch (Exception ex)
{
Debug.WriteLine("Error in getting db instance" + ex.Message);
}
return null;
}
Now I have not had any issues to date, however, I worry that this solution will not scale well if we are for example doing 1000s of inserts per minute?
I then also worry that with 1 static db instance that we could get memory leaks or that object will keep growing and not manage it's db connections properly?
What is the correct way to use EF with long term db connections?
Related
I've been searching everywhere, to try and get over this issue but I just can't figure this out.
I'm trying to make many changes to the DB with one single transaction using LINQ to SQL.
I've created a .dbml that represents the SQL Table, then I use basicaly this code:
foreach (var _doc in _r.Docs)
{
try
{
foreach (var _E in _Es)
{
Entity _newEnt = CreateNewEnt(_EListID, _doc, _fileName, _E);
_db.Etable.InsertOnSubmit(_newEnt);
_ECount++;
if (_ECount % 1000 == 0)
{
_db.SubmitChanges();
}
}
}
catch (Exception ex)
{
throw;
}
}
But when I do a SQL Profiler, the commands are all executed individually. It won't even start an SQL Transaction.
I've tried using TransactionScope (using statement and Complete()) and DbTransaction (BeginTransaction() and Commit()), none of them did anything at all, it just keeps on executing all commands individually, inserting everything like it was looping through all the inserts.
TransactionScope:
using(var _tans = new TransactionScope())
{
foreach (var _doc in _r.Docs)
{
try
{
foreach (var _E in _Es)
{
Entity _newEnt = CreateNewEnt(_EListID, _doc, _fileName, _E);
_db.Etable.InsertOnSubmit(_newEnt);
_ECount++;
if (_ECount % 1000 == 0)
{
_db.SubmitChanges();
}
}
}
catch (Exception ex)
{
throw;
}
}
_trans.Complete();
}
DbTransaction:
_db.Transaction = _db.Connection.BeginTransaction();
foreach (var _doc in _r.Docs)
{
try
{
foreach (var _E in _Es)
{
Entity _newEnt = CreateNewEnt(_EListID, _doc, _fileName, _E);
_db.Etable.InsertOnSubmit(_newEnt);
_ECount++;
if (_ECount % 1000 == 0)
{
_db.SubmitChanges();
}
}
}
catch (Exception ex)
{
throw;
}
}
_db.Transaction.Commit();
I also tried commiting transactions everytime I Submit the changes, but still nothing, just keeps on executing everything individually.
Right now I'm at a loss and wasting time :\
GSerg was right and pointed me to the right direction, Transactions do not mean multiple commands in one go, they just allow to "undo" all that was made inside given transaction if need be. Bulk statements do what I want to do.
You can download a Nuget Package directly from Visual Studio called "Z.LinqToSql.Plus" that helps with this. It extends DataContext from LINQ, and allows to do multiple insertions, updates or deletes in bulks, which means, in one single statement, like this:
foreach (var _doc in _r.Docs)
{
try
{
foreach (var _E in _Es)
{
Entity _newEnt = CreateNewEnt(_EListID, _doc, _fileName, _E);
_dictionary.add(_ECount, _newEnt); //or using a list as well
_ECount++;
if (_ECount % 20000 == 0)
{
_db.BulkInsert(_dictionary.Values); //inserts in bulk, there are also BulkUpdate and BulkDelete
_dictionary = new Dictionary<long, Entity>(); //restarts the dictionary to prepare for the next bulk
}
}
}
catch (Exception ex)
{
throw;
}
}
As in the code, I can even insert 20k entries in seconds. It's a very useful tool!
Thank you to everyone who tried helping! :)
I've got a plugin on Update (pre-op) of InvoiceDetail, in which I'm retrieving the associated Invoice to get some more information from it (ie: the tax profile that was selected at the invoice level) in CRM 2016.
Here's how I do it:
//xrmObjects is an object containing all useful objects in plugins/workflow...
var invoice = RetrieveEntity(xrmObjects.Service, xrmObjects.TracingService, image["invoiceid"] as EntityReference, new ColumnSet("invoiceid", "pricelevelid", "customerid", "opportunityid", "xtc_tax_definition"));
This specific line of code above triggers another Update on InvoiceDetail
Here's the method invoked above:
public static Entity RetrieveEntity(IOrganizationService service, ITracingService tracingService, EntityReference target, ColumnSet columnSet)
{
Entity entity = new Entity();
try
{
entity = CrmServiceExtensions.ExecuteWithRetry<RetrieveResponse>(service, new RetrieveRequest
{
Target = target,
ColumnSet = columnSet
}).Entity;
}
catch (Exception ex)
{
tracingService.Trace($"Error retrieving {target.LogicalName}: {ex.Message}");
throw;
}
return entity;
}
Here's ExecuteWithRetry:
public static T ExecuteWithRetry<T>(IOrganizationService service, OrganizationRequest request)
where T : OrganizationResponse
{
T response = null;
int i = 0;
// Maximum of five iterations.
while (i < 5)
{
try
{
response = (T)service.Execute(request);
// If the Execute does not throw an Exception, break the loop
break;
}
catch (System.Web.Services.Protocols.SoapException e)
{
// Retry if the SoapException is a "Generic SQL Error",
// otherwise rethrow the SoapException.
// "Generic SQL Error" might indicate a deadlock.
if (e.Detail.InnerText.ToLower().Contains("generic sql error"))
{
++i;
// Wait (sleep thread) for i * 1000 milliseconds.
// So, first iteration waits 1 second,
// while fifth iteration will wait 5 seconds.
System.Threading.Thread.Sleep(i * 1000);
}
else throw;
}
}
if (i >= 5)
{
throw new Exception("ExecuteWithRetry: too many retries");
}
return response;
}
I have validated that nothing funky is happening, the update message on InvoiceDetail is triggered again at the line response = (T)service.Execute(request);
I also tried by using early-bound and a context to retrieve the invoice but the LoadProperty methods which loads the invoice does the same thing....
using (XrmServiceContext ctx = new XrmServiceContext(xrmObjects.Service))
{
Xrm.InvoiceDetail image = xrmObjects.PluginContext.PreEntityImages["invoicedetail"].ToEntity<Xrm.InvoiceDetail>();
try
{
ctx.LoadProperty(image, "invoice_details");
}
catch (Exception ex)
{
throw new InvalidPluginExecutionException($"Error retrieving invoice details' invoice: {ex.Message}");
}
}
I can't see anything in my steps configuration that would do this. Any ideas?
Instead of using LoadProperty, I simply retrieved the invoice manually like so
var invoice = ctx.InvoiceSet.SingleOrDefault(x => x.Id == image.InvoiceId.Id);
Instead of:
ctx.LoadProperty(image, "invoice_details");
For some reason, LoadProperty is triggering unwanted update message on child invoice details...
my requirement is to process multiple cost files, which has million of records. after processing and validation I have to add those records to database.
For better performance I am using "yield" in foreach loop and return one record at a time, process that record and immediately add that one record to database with file number. During this file reading process if I come across any data validation error, I throw InvalidRecordException.
My requirement is to delete all the records from table related that file. in short, even if one record is invalid I want to mark that file as invalid file and not add even a single record of that file to database.
can anyone help me here, how can i make use of TransactionScope here.
public class CostFiles
{
public IEnumerable<string> FinancialRecords
{
get
{
//logic to get list of DataRecords
foreach (var dataRecord in DataRecords)
{
//some processing... which can throw InvalidRecord exception
yield return dataRecord;
}
yield break;
}
}
}
public void ProcessFileRecords(CostFiles costFile, int ImportFileNumber)
{
Database db = new Database();
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required))
{
try
{
foreach (var record in costFile.FinancialRecords)
{
db.Add(record, ImportFileNumber);
}
}
catch(InvalidRecordException ex)
{
//here i want to delete all the records from the table where import file number is same as input paramter ImportFileNumber
}
}
}
The purpose of a transaction scope is to create an "all or nothing" scenario, so either the whole transaction commits, or nothing at all commits. It looks like you already have the right idea (at least in terms of the TransactionScope. The scope won't actually commit the records to the database until you call TransactionScope.Complete(). If Complete() is not called, then the records are discarded when you leave the transaction scope. You could easily do something like this:
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required))
{
bool errorsEncountered = false;
try
{
foreach (var record in costFile.FinancialRecords)
{
db.Add(record, ImportFileNumber);
}
}
catch(InvalidRecordException ex)
{
//here i want to delete all the records from the table where import file number is same as input paramter ImportFileNumber
errorsEncountered = true;
}
if (!errorsEncountered)
{
scope.Complete();
}
}
Or you can just let the Add throw an exception and handle it outside of the transaction scope instead, as the exception will cause Complete() not to be called, and therefore no records added. This method has the additional advantage of stopping processing of additional records when we already know it will do nothing.
try
{
using (var scope = new TransactionScope(TransactionScopeOptions.Required))
{
foreach(var record in costFile.FinancialRecords)
{
db.Add(record, ImportFileNumber);
}
// if an exception is thrown during db.Add(), then Complete is never called
scope.Complete()
}
catch(Exception ex)
{
// handle your exception here
}
}
EDIT If you don't want your transaction elevated to a distributed transaction (which may have additional security/network requirements), make sure you reuse the same SqlConnection object for every database call within your transaction scope.
using (var conn = new SqlConnection("myConnectionString"))
{
conn.Open();
using (var scope = new TransactionScope(...))
{
foreach(var foo in foos)
{
db.Add(foo, conn);
}
scope.Complete();
}
}
I'm getting strange things since updated to EF6,no sure this is related or not, but used to be good
I'm doing a set of work, then save it to DB , then do another , save another.
after a while,i check SQL server by sp_who2 , i found many dead connections from my computer.
Job is huge then there goes to 700 connections,
I have to kill them all manually in cycle.
program like:
while (jobDone == false)
{
var returnData=doOneSetJob();
myEntity dbconn= new myEntity;
foreach( var one in retrunData)
{
dbconn.targetTable.add(one );
try
{
dbconn.savechange();
/// even i put a dispose() here , still lots of dead connections
}
catch
{
console.writeline("DB Insertion Fail.");
dbconn.dispose();
dbconn= new myEntity();
}
}
dbconn.dispose()
}
You should consider refactoring your code so that your connection is cleaned up after your job is complete. For example:
using (var context = new DbContext())
{
while (!jobDone)
{
// Execute job and get data
var returnData = doOneSetJob();
// Process job results
foreach (var one in returnData)
{
try
{
context.TargetTable.Add(one);
context.SaveChanges();
}
catch (Exception ex)
{
// Log the error
}
}
}
}
The using statement will guarantee that your context is cleaned up properly, even if an error occurs while you are looping through the results.
In this case you should use a using statement. Taken from MSDN:
The using statement ensures that Dispose is called even if an exception occurs while you are calling methods on the object. You can achieve the same result by putting the object inside a try block and then calling Dispose in a finally block; in fact, this is how the using statement is translated by the compiler.
So, your code would look better like this:
using(var dbconn = new DbContext())
{
while (!jobDone)
{
foreach(var one in retrunData)
{
try
{
targetTable row = new TargetTable();
dbconn.TargetTable.add(row);
dbconn.SaveChanges();
}
catch (Exception ex)
{
Console.WriteLine("DB Insertion Fail.");
}
}
}
}
This way, even if your code fails at some point, the Context, resources and connections will be properly disposed.
I am working on an app that is inserting a lot of new objects (rows) and relationships between them. But at a certain point when an error occurs I want all the changes to the DataContext be disgarded and "thrown away". So that after an error I have a clean copy of the DataContext that matches the state of the database.
Edit
Alternatively, you could make use of the DataContext.Transaction, and use that to .Commit() or .Rollback() your changes.
ORIG
Just throw away that DataContext & Re-instantiate it.
Something like...
public void MyMethod(string connStr)
{
try
{
DataClasses1DataContext dc = new DataClasses1DataContext(connStr);
for (int i = 0; i < 100; i++)
{
try
{
//Do Stuff
//Insert Objects
dc.SubmitChanges();
}
catch (Exception ex) //So if it bombs in the loop, log your exception
{
Log(ex);
}
finally //Reinstantiate your DC
{
dc = new DataClasses1DataContext(connStr);
}
}
}
catch (Exception bigEx)
{
Log(bigEx);
}
}
You could also use the TransactionScope in a using statement. If you don't call .Complete() on the TransactionScope, all changes are rolled back when it is disposed (which happens when leaving the using statement).