I've been searching everywhere, to try and get over this issue but I just can't figure this out.
I'm trying to make many changes to the DB with one single transaction using LINQ to SQL.
I've created a .dbml that represents the SQL Table, then I use basicaly this code:
foreach (var _doc in _r.Docs)
{
try
{
foreach (var _E in _Es)
{
Entity _newEnt = CreateNewEnt(_EListID, _doc, _fileName, _E);
_db.Etable.InsertOnSubmit(_newEnt);
_ECount++;
if (_ECount % 1000 == 0)
{
_db.SubmitChanges();
}
}
}
catch (Exception ex)
{
throw;
}
}
But when I do a SQL Profiler, the commands are all executed individually. It won't even start an SQL Transaction.
I've tried using TransactionScope (using statement and Complete()) and DbTransaction (BeginTransaction() and Commit()), none of them did anything at all, it just keeps on executing all commands individually, inserting everything like it was looping through all the inserts.
TransactionScope:
using(var _tans = new TransactionScope())
{
foreach (var _doc in _r.Docs)
{
try
{
foreach (var _E in _Es)
{
Entity _newEnt = CreateNewEnt(_EListID, _doc, _fileName, _E);
_db.Etable.InsertOnSubmit(_newEnt);
_ECount++;
if (_ECount % 1000 == 0)
{
_db.SubmitChanges();
}
}
}
catch (Exception ex)
{
throw;
}
}
_trans.Complete();
}
DbTransaction:
_db.Transaction = _db.Connection.BeginTransaction();
foreach (var _doc in _r.Docs)
{
try
{
foreach (var _E in _Es)
{
Entity _newEnt = CreateNewEnt(_EListID, _doc, _fileName, _E);
_db.Etable.InsertOnSubmit(_newEnt);
_ECount++;
if (_ECount % 1000 == 0)
{
_db.SubmitChanges();
}
}
}
catch (Exception ex)
{
throw;
}
}
_db.Transaction.Commit();
I also tried commiting transactions everytime I Submit the changes, but still nothing, just keeps on executing everything individually.
Right now I'm at a loss and wasting time :\
GSerg was right and pointed me to the right direction, Transactions do not mean multiple commands in one go, they just allow to "undo" all that was made inside given transaction if need be. Bulk statements do what I want to do.
You can download a Nuget Package directly from Visual Studio called "Z.LinqToSql.Plus" that helps with this. It extends DataContext from LINQ, and allows to do multiple insertions, updates or deletes in bulks, which means, in one single statement, like this:
foreach (var _doc in _r.Docs)
{
try
{
foreach (var _E in _Es)
{
Entity _newEnt = CreateNewEnt(_EListID, _doc, _fileName, _E);
_dictionary.add(_ECount, _newEnt); //or using a list as well
_ECount++;
if (_ECount % 20000 == 0)
{
_db.BulkInsert(_dictionary.Values); //inserts in bulk, there are also BulkUpdate and BulkDelete
_dictionary = new Dictionary<long, Entity>(); //restarts the dictionary to prepare for the next bulk
}
}
}
catch (Exception ex)
{
throw;
}
}
As in the code, I can even insert 20k entries in seconds. It's a very useful tool!
Thank you to everyone who tried helping! :)
Related
I have a worker thread, whose job is to insert Objects that are stored in a Queue into the database.
We are currently using Entity framework to do the inserts. Now my question is, do I need to make a new Db Instance for every insert? or can I safely re-use the same db instance over and over?
private static void MainWorker()
{
while (true)
{
try
{
if (IncomingDataQueue.Any())
{
if (IncomingDataQueue.TryDequeue(out var items))
{
//Insert into db
using (var db = GetNewDbInstance())
{
if (db != null)
{
db.DataRaw.AddRange(items);
db.SaveChanges();
//Skip everything and continue to the next loop
continue;
}
}
}
}
}
catch (Exception ex)
{
Debug.WriteException("Failed to insert DB Data", ex);
//Delay here in case we are hitting the db 2 hard.
Thread.Sleep(100);
}
//Wait here as we did not have any items in the queue, so wait before checkign again
Thread.Sleep(20);
}
}
Here is my function which gets a new DB Instance:
private static DbEntities GetNewDbInstance()
{
try
{
var db = new DbEntities();
db.Configuration.ProxyCreationEnabled = false;
db.Configuration.AutoDetectChangesEnabled = false;
return db;
}
catch (Exception ex)
{
Debug.WriteLine("Error in getting db instance" + ex.Message);
}
return null;
}
Now I have not had any issues to date, however, I worry that this solution will not scale well if we are for example doing 1000s of inserts per minute?
I then also worry that with 1 static db instance that we could get memory leaks or that object will keep growing and not manage it's db connections properly?
What is the correct way to use EF with long term db connections?
my requirement is to process multiple cost files, which has million of records. after processing and validation I have to add those records to database.
For better performance I am using "yield" in foreach loop and return one record at a time, process that record and immediately add that one record to database with file number. During this file reading process if I come across any data validation error, I throw InvalidRecordException.
My requirement is to delete all the records from table related that file. in short, even if one record is invalid I want to mark that file as invalid file and not add even a single record of that file to database.
can anyone help me here, how can i make use of TransactionScope here.
public class CostFiles
{
public IEnumerable<string> FinancialRecords
{
get
{
//logic to get list of DataRecords
foreach (var dataRecord in DataRecords)
{
//some processing... which can throw InvalidRecord exception
yield return dataRecord;
}
yield break;
}
}
}
public void ProcessFileRecords(CostFiles costFile, int ImportFileNumber)
{
Database db = new Database();
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required))
{
try
{
foreach (var record in costFile.FinancialRecords)
{
db.Add(record, ImportFileNumber);
}
}
catch(InvalidRecordException ex)
{
//here i want to delete all the records from the table where import file number is same as input paramter ImportFileNumber
}
}
}
The purpose of a transaction scope is to create an "all or nothing" scenario, so either the whole transaction commits, or nothing at all commits. It looks like you already have the right idea (at least in terms of the TransactionScope. The scope won't actually commit the records to the database until you call TransactionScope.Complete(). If Complete() is not called, then the records are discarded when you leave the transaction scope. You could easily do something like this:
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required))
{
bool errorsEncountered = false;
try
{
foreach (var record in costFile.FinancialRecords)
{
db.Add(record, ImportFileNumber);
}
}
catch(InvalidRecordException ex)
{
//here i want to delete all the records from the table where import file number is same as input paramter ImportFileNumber
errorsEncountered = true;
}
if (!errorsEncountered)
{
scope.Complete();
}
}
Or you can just let the Add throw an exception and handle it outside of the transaction scope instead, as the exception will cause Complete() not to be called, and therefore no records added. This method has the additional advantage of stopping processing of additional records when we already know it will do nothing.
try
{
using (var scope = new TransactionScope(TransactionScopeOptions.Required))
{
foreach(var record in costFile.FinancialRecords)
{
db.Add(record, ImportFileNumber);
}
// if an exception is thrown during db.Add(), then Complete is never called
scope.Complete()
}
catch(Exception ex)
{
// handle your exception here
}
}
EDIT If you don't want your transaction elevated to a distributed transaction (which may have additional security/network requirements), make sure you reuse the same SqlConnection object for every database call within your transaction scope.
using (var conn = new SqlConnection("myConnectionString"))
{
conn.Open();
using (var scope = new TransactionScope(...))
{
foreach(var foo in foos)
{
db.Add(foo, conn);
}
scope.Complete();
}
}
I need to insert new rows in two of my tables first table's auto generated id field is one of the field of second table .currently I'm using transaction for inserting data. My current code is given below
using (var context = new ApplicationDatabaseEntities())
{
using (var transaction = context.Database.BeginTransaction())
{
try
{
foreach (var item in itemList)
{
context.MyFirstEntity.Add(item);
context.SaveChanges();
mySecondEntity.MyFirstEntityId = item.Id;
context.MySecondEntity.Add(mySecondEntity.MyFirstEntityId );
context.SaveChanges();
}
transaction.Commit();
}
catch (Exception ex)
{
transaction.Rollback();
Console.WriteLine(ex);
}
}
}
The above code is working fine .my question is, is this the correct way? I mean if I have 1000 or 2000 items for insertion does my current code affect the performance
Code can be improved with implicit transaction:
foreach (var item in itemList)
{
context.MyFirstEntity.Add(item);
mySecondEntity.MyFirstEntity = item;
context.MySecondEntity.Add(mySecondEntity);
}
context.SaveChanges();
Note: instead of id I've used navigation property.
It depends. If the whole batch has to be transactional then you have no other way as do in one big transaction. If transaction has to be guaranteed for tupples only then if the time of transaction is big enough you may face some locks. Then you can just do transaction in the loop for each tupple.
Also you can do what you are doing in one go without explicit transaction. You can SaveChanges after the loop and it will be transactional:
foreach (var item in itemList)
{
context.MyFirstEntity.Add(item);
mySecondEntity.MyFirstEntity = item;
context.MySecondEntity.Add(mySecondEntity);
}
context.SaveChanges();
You should wrap your transaction in a using statement and rollback when an exception is thrown.
using (DbContextTransaction transaction = context.Database.BeginTransaction())
{
try
{
foreach (var item in itemList)
{
context.MyFirstEntity.Add(item);
mySecondEntity.MyFirstEntity = item;
context.MySecondEntity.Add(mySecondEntity);
}
transaction.Commit();
}
catch (Exception e)
{
transaction.Rollback();
}
}
More info here.
I'm getting strange things since updated to EF6,no sure this is related or not, but used to be good
I'm doing a set of work, then save it to DB , then do another , save another.
after a while,i check SQL server by sp_who2 , i found many dead connections from my computer.
Job is huge then there goes to 700 connections,
I have to kill them all manually in cycle.
program like:
while (jobDone == false)
{
var returnData=doOneSetJob();
myEntity dbconn= new myEntity;
foreach( var one in retrunData)
{
dbconn.targetTable.add(one );
try
{
dbconn.savechange();
/// even i put a dispose() here , still lots of dead connections
}
catch
{
console.writeline("DB Insertion Fail.");
dbconn.dispose();
dbconn= new myEntity();
}
}
dbconn.dispose()
}
You should consider refactoring your code so that your connection is cleaned up after your job is complete. For example:
using (var context = new DbContext())
{
while (!jobDone)
{
// Execute job and get data
var returnData = doOneSetJob();
// Process job results
foreach (var one in returnData)
{
try
{
context.TargetTable.Add(one);
context.SaveChanges();
}
catch (Exception ex)
{
// Log the error
}
}
}
}
The using statement will guarantee that your context is cleaned up properly, even if an error occurs while you are looping through the results.
In this case you should use a using statement. Taken from MSDN:
The using statement ensures that Dispose is called even if an exception occurs while you are calling methods on the object. You can achieve the same result by putting the object inside a try block and then calling Dispose in a finally block; in fact, this is how the using statement is translated by the compiler.
So, your code would look better like this:
using(var dbconn = new DbContext())
{
while (!jobDone)
{
foreach(var one in retrunData)
{
try
{
targetTable row = new TargetTable();
dbconn.TargetTable.add(row);
dbconn.SaveChanges();
}
catch (Exception ex)
{
Console.WriteLine("DB Insertion Fail.");
}
}
}
}
This way, even if your code fails at some point, the Context, resources and connections will be properly disposed.
I want to read data to a list from database.
I tried the following code
public List<T> StoredProcedureForIList<T>(string spName, params IDataParameter[] commandParameters)
{
List<T> list = new List<T>();
T item;
Type listItemType = typeof(T);
item = (T)Activator.CreateInstance(listItemType);
list.Add(item);
using (IDatabaseConnection connection = new DatabaseConnection())
{
IDbCommand cmd = connection.CreateCommandForStoredProcedure(spName);
foreach (SqlParameter par in commandParameters)
{
cmd.Parameters.Add(par);
}
try
{
using (IDataReader reader = cmd.ExecuteReader())
{
while (reader != null && reader.Read())
{
for (int i = 0; i < reader.FieldCount; i++)
{
var prop = listItemType.GetProperty(reader.GetName(i));
prop.SetValue(item, reader[i], null);
}
list.Add(item);
}
}
}
catch(Exception ex)
{ }
return list;
}
}
But the problem is that when the for loop starts the reader loses data.
The data reader ResultView value is Enumeration yielded no results.
My guess is that some error occurs during execution of your loop. This technique...
try
{
...
}
catch(Exception ex)
{ }
...ensures that this error is ignored and all you get is an incomplete result. As you have noticed, this makes debugging quite hard. So don't do that.
Thus, the solution is:
remove the try-catch block (i.e., replace try { ... } catch(Exception ex) {} by ...),
run the code again,
note the error that occurs,
if you understand the error
fix it
else
ask again on StackOverflow, in a new question.
And, never, never write catch (Exception ex) {} again. ;-) Do proper error handling, or don't do error handling at all.
The reader won't be dropping rows; the reader is pretty well-tested. Swallowing exceptions won't help. If I had to guess, the problem here is that you are adding the same item over and over. In fact, you add it N+1 times (you add it once at the top even if no rows are returned).
However, can I suggest: just use something like dapper, which does everything above, except a: it gets it right, and b: it is highly optimized (it emits custom IL to avoid constant reflection, and caches that IL). It would be something akin to:
var list = connection.Query<T>(procName, namedArgs,
commandType: CommandType.StoredProcedure).ToList();
where namedArgs would be, to pass in #id and #name, for example:
new {id=123, name="abc"}
i.e.
int id = ...
string name = ...
var list = connection.Query<T>("MyProc", new {id, name},
commandType: CommandType.StoredProcedure).ToList();