In my C# program I am using Entity Framework to synchronize a local SQL Server database with QuickBooks data. Getting the data from QuickBooks does not seem to have any issues. However I am running into a stumbling block when doing batch commits of entities.
Currently I am building up the DataContext with a configurable number of entities and then committing the entities in batch. So far the batch has not failed, but what if it does? My idea to combat this would be to iterate over the batch and submit each entity one at a time and then log the one(s) that is/are causing the commit failure.
However I do not see a way to do this with the data context since it appears to be an all or nothing matter when using SaveChanges(). Is there a way to handle what I am trying to accomplish, or should I be going about dealing with the failures in a completely different way?
Here is the code that I currently have, in case you want to take a look at it:
int itemsCount = 0;
int itemsSynced = 0;
int itemsFailed = 0;
ArrayList exceptions = new ArrayList();
int batchSliceCount = Properties.Settings.Default.SyncBatchSize; //Getting the max batch size from the settings
int index = 1; //Index used for keeping track of current batch size on data context
List<Customer> currentBatch = new List<Customer>(); // List to hold curent batch
db = new DataContext(DatabaseHelper.GetLocalDatabaseConnectionString());
foreach (var customer in QBResponse.customers)
{
itemsCount++;
try
{
string debugMsg = "Saving Customer with the Following Details....." + Environment.NewLine;
debugMsg += "ListId: " + customer.CustomerListId + Environment.NewLine;
debugMsg += "FullName: " + customer.FullName + Environment.NewLine;
int progressPercentage = (itemsCount * 100) / opResponse.retCount;
UpdateStatus(Enums.LogLevel.Debug, debugMsg, progressPercentage);
var dbCustomer = db.Customers.FirstOrDefault(x => x.CustomerListId == customer.CustomerListId);
if (dbCustomer == null)
{
// customer.CopyPropertiesFrom(customer, db);
Customer newCustomer = new Customer();
newCustomer.CopyCustomer(customer, db);
newCustomer.AddBy = Enums.OperationUser.SyncOps;
newCustomer.AddDateTime = DateTime.Now;
newCustomer.EditedBy = Enums.OperationUser.SyncOps;
newCustomer.EditedDateTime = DateTime.Now;
newCustomer.SyncStatus = true;
db.Customers.Add(newCustomer);
currentBatch.Add(newCustomer);
}
else
{
//dbCustomer.CopyPropertiesFrom(customer, db);
dbCustomer.CopyCustomer(customer, db);
dbCustomer.EditedBy = Enums.OperationUser.SyncOps;
dbCustomer.EditedDateTime = DateTime.Now;
dbCustomer.SyncStatus = true;
currentBatch.Add(dbCustomer);
}
try
{
if (index % batchSliceCount == 0 || index == opResponse.customers.Count()) //Time to submit the batch
{
UpdateStatus(Enums.LogLevel.Information, "Saving Batch of " + batchSliceCount + "Customers to Local Database");
db.SaveChanges();
itemsSynced += currentBatch.Count();
currentBatch = new List<Customer>();
db.Dispose();
db = new DataContext(DatabaseHelper.GetLocalDatabaseConnectionString());
}
}
catch (Exception ex)
{
string errorMsg = "Error occured submitting batch. Itterating and submitting one at a time. " + Environment.NewLine;
errorMsg += "Error Was: " + ex.GetBaseException().Message + Environment.NewLine + "Stack Trace: " + ex.GetBaseException().StackTrace;
UpdateStatus(Enums.LogLevel.Debug, errorMsg, progressPercentage);
//What to do here? Is there a way to properly iterate over the context and submit a change one at a time?
}
}
catch (Exception ex)
{
//Log exception and restart the data context
db.Dispose();
db = new DataContext(DatabaseHelper.GetLocalDatabaseConnectionString());
}
Thread.Sleep(Properties.Settings.Default.SynchronizationSleepTimer);
index++;
}
it depends on the exceptions you want to recover from...
If you are just looking for a way to retry if the connection was interrupted you could use a custom Execution Strategy based on DbExecutionStrategy that retries if specific errors occur as demonstrated in this CodeProject article.
Related
I am struggling a few days already with a problem of growing memory consumption by console application in .Net Core 2.2, and just now I ran out of ideas what else I could improve.
Im my application I have a method that triggers StartUpdatingAsync method:
public MenuViewModel()
{
if (File.Exists(_logFile))
File.Delete(_logFile);
try
{
StartUpdatingAsync("basic").GetAwaiter().GetResult();
}
catch (ArgumentException aex)
{
Console.WriteLine($"Caught ArgumentException: {aex.Message}");
}
Console.ReadKey();
}
StartUpdatingAsync creates 'repo' and instance is getting from DB a list of objects to be updated (around 200k):
private async Task StartUpdatingAsync(string dataType)
{
_repo = new DataRepository();
List<SomeModel> some_list = new List<SomeModel>();
some_list = _repo.GetAllToBeUpdated();
await IterateStepsAsync(some_list, _step, dataType);
}
And now, within IterateStepsAsync we are getting updates, parsing them with existing data and updating DB. Inside of each while I was creating new instances of all new classes and lists, to be sure that old ones are releasing memory, but it didnt help. Also I was GC.Collect() at the end of the method, what also is not helping. Please note, that method below triggers lots of parralel Tasks, but they supposed to be disposed within it, am I right?:
private async Task IterateStepsAsync(List<SomeModel> some_list, int step, string dataType)
{
List<Area> areas = _repo.GetAreas();
int counter = 0;
while (counter < some_list.Count)
{
_repo = new DataRepository();
_updates = new HttpUpdates();
List<Task> tasks = new List<Task>();
List<VesselModel> vessels = new List<VesselModel>();
SemaphoreSlim throttler = new SemaphoreSlim(_degreeOfParallelism);
for (int i = counter; i < step; i++)
{
int iteration = i;
bool skip = false;
if (dataType == "basic" && (some_list[iteration].Mmsi == 0 || !some_list[iteration].Speed.HasValue)) //if could not be parsed with "full"
skip = true;
tasks.Add(Task.Run(async () =>
{
string updated= "";
await throttler.WaitAsync();
try
{
if (!skip)
{
Model model= await _updates.ScrapeSingleModelAsync(some_list[iteration].Mmsi);
while (Updating)
{
await Task.Delay(1000);
}
if (model != null)
{
lock (((ICollection)vessels).SyncRoot)
{
vessels.Add(model);
scraped = BuildData(model);
}
}
}
else
{
//do nothing
}
}
catch (Exception ex)
{
Log("Scrape error: " + ex.Message);
}
finally
{
while (Updating)
{
await Task.Delay(1000);
}
Console.WriteLine("Updates for " + counter++ + " of " + some_list.Count + scraped);
throttler.Release();
}
}));
}
try
{
await Task.WhenAll(tasks);
}
catch (Exception ex)
{
Log("Critical error: " + ex.Message);
}
finally
{
_repo.UpdateModels(vessels, dataType, counter, some_list.Count, _step);
step = step + _step;
GC.Collect();
}
}
}
Inside of the method above, we are calling _repo.UpdateModels, where is updated DB. I tryed two approaches, with using EC Core and SqlConnection. Both with similar results. Below you can find both of them.
EF Core
internal List<VesselModel> UpdateModels(List<Model> vessels, string dataType, int counter, int total, int _step)
{
for (int i = 0; i < vessels.Count; i++)
{
Console.WriteLine("Parsing " + i + " of " + vessels.Count);
Model existing = _context.Vessels.Where(v => v.id == vessels[i].Id).FirstOrDefault();
if (vessels[i].LatestActivity.HasValue)
{
existing.LatestActivity = vessels[i].LatestActivity;
}
//and similar parsing several times, as above
}
Console.WriteLine("Saving ...");
_context.SaveChanges();
return new List<Model>(_step);
}
SqlConnection
internal List<VesselModel> UpdateModels(List<Model> vessels, string dataType, int counter, int total, int _step)
{
if (vessels.Count > 0)
{
using (SqlConnection connection = GetConnection(_connectionString))
using (SqlCommand command = connection.CreateCommand())
{
connection.Open();
StringBuilder querySb = new StringBuilder();
for (int i = 0; i < vessels.Count; i++)
{
Console.WriteLine("Updating " + i + " of " + vessels.Count);
//PARSE
VesselAisUpdateModel existing = new VesselAisUpdateModel();
if (vessels[i].Id > 0)
{
//find existing
}
if (existing != null)
{
//update for basic data
querySb.Append("UPDATE dbo." + _vesselsTableName + " SET Id = '" + vessels[i].Id+ "'");
if (existing.Mmsi == 0)
{
if (vessels[i].MMSI.HasValue)
{
querySb.Append(" , MMSI = '" + vessels[i].MMSI + "'");
}
}
//and similar parsing several times, as above
querySb.Append(" WHERE Id= " + existing.Id+ "; ");
querySb.AppendLine();
}
}
try
{
Console.WriteLine("Sending SQL query to " + counter);
command.CommandTimeout = 3000;
command.CommandType = CommandType.Text;
command.CommandText = querySb.ToString();
command.ExecuteNonQuery();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
connection.Close();
}
}
}
return new List<Model>(_step);
}
Main problem is, that after tenths/hundreds of thousands of updated models my console application memory consumption increases continuously. And I have no idea why.
SOLUTION my problem was inside of ScrapeSingleModelAsync method, where I was using incorrectly HtmlAgilityPack, what I could debug thanks to cassandrad.
Your code is messy, with huge amount of different objects with unknown lifetime. It's hardly possible to figure out the problem just looking at it.
Consider using profiling tools, for example Visual Studio's Diagnostic Tools, they will help you to find what objects are living too long in the heap. Here is overview of its functions related to memory profiling. Highly recomended to be read.
In short, you need to take two snapshots and look at what objects are taking the most memory. Let's look at simple example.
int[] first = new int[10000];
Console.WriteLine(first.Length);
int[] secod = new int[9999];
Console.WriteLine(secod.Length);
Console.ReadKey();
Take the first snapshot when your function works at least once. In my case, I took snapshot when the first huge space has been alocated.
After that, let your app be working some time so the difference in memory usage become noticeable, take the second memory snapshot.
You'll notice that another snapshot is added with info about how much is the difference. To get more specific info, click on one or another blue label of the latest snapshot to open snapshots comparison.
Following my example, we can see that there is change in count of int arrays. By default int[] wasn't visible in the table, so I had to uncheck Just My Code in filtration options.
So, this is what needs to be done. After you figure out what objects increase in count or size over time, you can locate where these objects are create and optimize this operation.
I'm trying to remove all Lookup Tables that start with a specific prefix inside all families.
The method "sizeTableManager.RemoveSizeTable(tableToRemove)" returns true as if it succeeded but when I go edit the families in the project and bring up the Lookup Tables list they are still there.
The transaction seems to be committing with no errors too, which is even more puzzling...
Any ideas as to what I'm doing wrong?
This is my code so far:
string existingPrefix = "ExistingPrefix_";
Document doc = this.ActiveUIDocument.Document;
FilteredElementCollector collector = new FilteredElementCollector(doc);
ICollection<Element> elements = collector.OfClass(typeof(Family)).ToElements();
foreach (var element in elements)
{
using (Transaction t = new Transaction(doc, "flush old lookup tables"))
{
t.Start();
FamilySizeTableManager sizeTableManager = FamilySizeTableManager.GetFamilySizeTableManager(doc, element.Id);
if(sizeTableManager != null)
{
foreach (var tableToRemove in sizeTableManager.GetAllSizeTableNames())
{
if(tableToRemove.StartsWith(existingPrefix))
{
bool result = sizeTableManager.RemoveSizeTable(tableToRemove);
if(result)
{
// TaskDialog.Show("Success", "Removed " + tableToRemove + " from " + element.Name);
var test = "test";
}
else
{
TaskDialog.Show("Warning", "Unable to remove " + tableToRemove + " from " + element.Name);
}
}
}
}
var commitResult = t.Commit();
}
}
Thanks in advance!
Well, turns out that the RemoveSizeTable() method was indeed working, but that I had to regenerate the document for the changes to take effect:
using (Transaction t = new Transaction(doc, "regenerate document"))
{
t.Start();
doc.Regenerate();
t.Commit();
}
Discovered this after noticing that saving the document caused the changes to take effect.
I have a code the gets data from my server. Every data will be inserted in my local database (SQLite Database) one by one. Every once in a whole I am getting these two errors. My codes below is where the exception are always appearing.
SQLite.SQLiteException: Busy
SQLite.SQLiteException: database is locked
What is/are the cause(s) why I always get these exceptions?
Update:
I added await conn.CloseAsync(); in every end of the code is that correct?
here is my first code, it will inquire to the server if there were updates and if there are updates it will insert or replace(update) the data in my local database:
var db = DependencyService.Get<ISQLiteDB>();
var conn = db.GetConnection();
string apifile = "sync-retailer-outlet-server-update-api.php";
var lastchecked = Preferences.Get("retaileroutletchangelastcheck", String.Empty, "private_prefs");
int count = 0;
var uri = new Uri(string.Format("http://" + domain + "/TBSApp/app_api/" + apifile + "?Host=" + host + "&Database=" + database + "&ContactID=" + contact + "&LastChecked=" + lastchecked, string.Empty));
try
{
SyncStatus("Getting retailer outlet data from server");
var response = await client.GetAsync(uri);
if (response.IsSuccessStatusCode)
{
var content = await response.Content.ReadAsStringAsync();
if (!string.IsNullOrEmpty(content))
{
var dataresult = JsonConvert.DeserializeObject<List<RetailerGroupData>>(content, settings);
var datacount = dataresult.Count;
for (int i = 0; i < datacount; i++)
{
SyncStatus("Saving retailer outlet server update to local database (" + (count + 1) + " out of " + dataresult.Count + ")");
var item = dataresult[i];
var retailerCode = item.RetailerCode;
var insertdata = new RetailerGroupTable
{
RetailerCode = retailerCode
};
await conn.InsertOrReplaceAsync(insertdata);
count++;
}
}
}
}
catch (Exception ex)
{
Crashes.TrackError(ex);
}
await conn.CloseAsync();
Here is the other code that creates the local database table everytime the StartPage is loaded.
public async void CreateTableAsync()
{
try
{
var db = DependencyService.Get<ISQLiteDB>();
var conn = db.GetConnection();
if (conn != null)
{
try
{
await conn.CreateTableAsync<UserTable>();
await conn.CreateTableAsync<ContactsTable>();
await conn.CreateTableAsync<ActivityTable>();
await conn.CreateTableAsync<CAFTable>();
await conn.CreateTableAsync<RetailerGroupTable>();
await conn.CreateTableAsync<UserEmailTable>();
await conn.CreateTableAsync<UserLogsTable>();
await conn.CreateTableAsync<SubscriptionTable>();
await conn.CreateTableAsync<ProvinceTable>();
await conn.CreateTableAsync<TownTable>();
}
catch (Exception ex)
{
Console.Write("Creating table error " + ex.Message);
}
}
}
catch (Exception ex)
{
Crashes.TrackError(ex);
await DisplayAlert("Application Error", "Error:\n\n" + ex.Message.ToString() + "\n\n Please contact your administrator", "Ok");
}
}
await conn.CloseAsync();
using (SQLiteConnection connection = db.GetConnection())
{
// Do whatever you want to do with your active connection
}
Note that you cannot reuse that connection outside the scope of the using block and have to get a new connection the same way the next time you want to access your database.
According to your description and code, I assume you use async-style inserts and are on different threads and thus an insert is timing out waiting for the lock of a different insert to complete. You can use synchronous inserts to avoid this condition.
I want to reprocess 3.5 million data from database. We are using oracle pl/sql 11g. And I tried to fetch 1000 rows per batch for reprocessing. But after processing 42000 rows it throws out of memory exception.
This is my code
internal void dataReprocessor(int rowCount)
{
DataTable dataTable = new DataTable();
int MaxProcessid = AssetProcessorDbHandler.GetMaxProcessId();
startCount=AssetProcessorDbHandler.GetErrorCount();
ShowStatus("Max. processid:" + MaxProcessid);
ShowStatus("Fetching " + rowCount + " rows....");
do
{
dataTable.Clear();
dataTable = AssetProcessorDbHandler.GetHundreadrows(rowCount, MaxProcessid);
if (dataTable.Rows.Count > 0)
{
for (int i = 0; i < dataTable.Rows.Count; i++)
{
int error = 0;
mainId = Convert.ToInt32(dataTable.Rows[i]["MAINID"]);
itemId = Convert.ToInt32(dataTable.Rows[i]["ITEMID"]);
siteid = Convert.ToInt32(dataTable.Rows[i]["SITE_ID"]);
equipmentId = Convert.ToInt32(dataTable.Rows[i]["EQUIPMENT_TYPE"]);
qrCode = dataTable.Rows[i]["QRCODE"].ToString();
string equipmentname = AssetProcessorDbHandler.Getequipmentname(equipmentId);
error = DbHandler.CheckQRCofEquipment(siteid, equipmentId, itemId, qrCode,ref assetStatus);
if (error == 1)//Not available in iMapp.
{
MESSAGE=string.Concat("QRCode ",qrCode," for ",equipmentname," is not available in iMapp!");
AssetProcessorDbHandler.UpdateSMSintoQRCtable(itemId, MESSAGE);
}
else if (error == 2)//Not available in that site.
{
int mappedsite = AssetProcessorDbHandler.Getmappedsiteid(qrCode);
MESSAGE = string.Concat("QRCode ", qrCode, " for ", equipmentname, " is mapped with site IN-", mappedsite);
AssetProcessorDbHandler.UpdateSMSintoQRCtable(itemId,MESSAGE);
}
else if (assetStatus == 1)//Equipment was in Finance verifiaction pending
{
MESSAGE = string.Concat("Finance verification pending of ", equipmentname, " for site IN-", siteid, " in iMapp. Please contact corporate finance team.");
AssetProcessorDbHandler.UpdateSMSintoQRCtable(itemId, MESSAGE);
}
AssetProcessorDbHandler.UpdateReprocessed(MaxProcessid + 1, itemId);
}
}
} while (dataTable.Rows.Count > 0);
ShowStatus("--Completed.--");
finalCount = AssetProcessorDbHandler.GetErrorCount();
ShowStatus(startCount + " ROWS PROCESSED.");
ShowStatus(finalCount + " ROWS ARE STILL ON ERROR.");
}
Anybody help me out please? The issue is escalating.
We had the same issue while we are always processing millions of records per day.
The problem is we had too many connections open to oracle database and did not close and dispose the objects well.
It is better to use the bellow snippet while interacting with oracle database.
using(var oracleConnection = new OracleConnection(connectionString))
{
oracleConnection.Open()
//do your stuff here.
}
You are never disposing your datatables. Stay with one datatable per loop and dispose it properly.
This part:
do
{
dataTable.Clear();
dataTable = AssetProcessorDbHandler.GetHundreadrows(rowCount, MaxProcessid);
if (dataTable.Rows.Count > 0)
{
Should read:
while(true)
{
using(var dataTable = AssetProcessorDbHandler.GetHundreadrows(rowCount, MaxProcessid))
{
if (dataTable.Rows.Count == 0) break;
The following code gives me the error (I get it from the MessageBox.Show() in the catch block)
"Exception in PopulateBla() : There is a file sharing violation. A
different process might be using the file [,,,,,,]
CODE
using (SqlCeCommand cmd = new SqlCeCommand(SQL_GET_VENDOR_ITEMS, new SqlCeConnection(SQLCE_CONN_STR)))
{
cmd.Parameters.Add("#VendorID", SqlDbType.NVarChar, 10).Value = vendorId;
cmd.Parameters.Add("#VendorItemID", SqlDbType.NVarChar, 19).Value = vendorItemId;
try
{
cmd.Connection.Open();
using (SqlCeDataReader SQLCEReader = cmd.ExecuteReader(CommandBehavior.SingleRow))
{
if (SQLCEReader.Read())
{
itemID = SQLCEReader.GetString(ITEMID_INDEX);
packSize = SQLCEReader.GetString(PACKSIZE_INDEX);
recordFound = true;
}
}
}
catch (SqlCeException err)
{
MessageBox.Show(string.Format("Exception in PopulateControlsIfVendorItemsFound: {0}\r\n", err.Message));//TODO: Remove
}
finally
{
if (cmd.Connection.State == ConnectionState.Open)
{
cmd.Connection.Close();
}
}
}
SQL_GET_VENDOR_ITEMS is my query string.
What file sharing problem could be happening here?
UPDATE
This is the kind of code that makes that sort of refactoring recommended by ctacke below difficult:
public void setINVQueryItemGroup( string ID )
{
try
{
dynSQL += " INNER JOIN td_item_group ON t_inv.id = td_item_group.id AND t_inv.pack_size = td_item_group.pack_size WHERE td_item_group.item_group_id = '" + ID + "'";
}
catch( Exception ex )
{
CCR.ExceptionHandler( ex, "InvFile.setINVQueryDept" );
}
}
A SQL statement is being appended to by means of a separate method, altering a global var (dynSQL) while possibly allowing for SQL Injection (depending on where/how ID is assigned). If that's not enough, any exception thrown could mislead the weary bughunter due to indicating it occurred in a different method (doubtless the victim of a careless copy-and-paste operation).
This is "Coding Horror"-worthy. How many best practices can you ignore in a scant few lines of code?
Here's another example:
string dynSQL = "SELECT * FROM purgatory WHERE vendor_item = '" + VendorItem + "' ";
if (vendor_id != "")
{
dynSQL += "AND vendor_id = '" + vendor_id + "' ";
}
It could be done by replacing the args with "?"s, but the code to then determine which/how many params to assign would be 42X uglier than Joe Garagiola's mean cleats.
I really like Chris' idea of using a single connection to your database. You could declare that global to your class like so:
public ClayShannonDatabaseClass
{
private SqlCeConnection m_openConnection;
public ClayShannonDatabaseClass()
{
m_openConnection = new SqlCeConnection();
m_openConnection.Open();
}
public void Dispose()
{
m_openConnection.Close();
m_openConnection.Dispose();
m_openConnection = null;
}
}
I'm guessing your code is crashing whenever you attempt to actually open the database.
To verify this, you could stick an integer value in the code to help you debug.
Example:
int debugStep = 0;
try
{
//cmd.Connection.Open(); (don't call this if you use m_openConnection)
debugStep = 1;
using (SqlCeDataReader SQLCEReader = cmd.ExecuteReader(CommandBehavior.SingleRow))
{
debugStep = 2;
if (SQLCEReader.Read())
{
debugStep = 3;
itemID = SQLCEReader.GetString(ITEMID_INDEX);
debugStep = 4;
packSize = SQLCEReader.GetString(PACKSIZE_INDEX);
debugStep = 5;
recordFound = true;
}
}
}
catch (SqlCeException err)
{
string msg = string.Format("Exception in PopulateControlsIfVendorItemsFound: {0}\r\n", err.Message);
string ttl = string.Format("Debug Step: {0}", debugStep);
MessageBox.Show(msg, ttl); //TODO: Remove
}
// finally (don't call this if you use m_openConnection)
// {
// if (cmd.Connection.State == ConnectionState.Open)
// {
// cmd.Connection.Close();
// }
// }
I'm guessing your error is at Step 1.
Provided the file isn't marked read-only (you checked that, right?), then you have another process with a non-sharing lock on the file.
The isql.exe database browser that comes with SQL CE is a common culprit if it's running in the background.
Depending on your version of SQLCE, it's quite possible that another process has an open connection (can't recall what version started allowing multiple process connections), so if you have any other app in the background that has it open, that may be a problem too.
You're also using a boatload of connections to that database, and they don't always get cleaned up and released immediately up Dispose. I'd highly recommend building a simple connection manager class that keeps a single (or more like two) connections to the database and just reuses them for all operations.