Last week I had a bug that I tried to recreate. A piece of existing code did a mass insert on the database. Though, a value was not allowed to be duplicate in the database The old code persisted the changes to the database after every AddObject with a SaveChanges. This was not performing very well. So, instead of saving it every time I saved it after every 1000 records (and did this in a transaction,but that is not relevant for this sample).
this code below gives a overview of what I did.
TestEntities test = new TestEntities();
for (int i = 0; i < 10; i++)
{
for (int j = 0; j < 10; j++)
{
string q = j.ToString();
if (!test.warehouses.Any(x => x.combi == q))
{
warehouse wh = new warehouse();
wh.combi = j.ToString();
wh.ean = j.ToString();
test.warehouses.AddObject(wh);
}
}
}
test.SaveChanges();
What I did not know was that entity framework only queries the data in the database and not the pending data so the any query gives no result on the database (and I was assuming that there would be a result.) Hence, this results in unwanted duplicates in the code above.
For now I solved it to store all soon to be added objects in memory and then store them in the database. This works, but gives a lot of overhead in the code. Is there a way to work around this issue? Can you tell EF to work on the database and the pending changes? Are all ORM's working like this?
Thanks,
Patrick
Faced with similar problem I decided to check database as you do and check local data:
if( !test.warehouses.Local.Any(x => x.combi == q )
&& !test.warehouses.Any(x => x.combi == q ) )
This should work.
Edit:
But it doesn't. However, this does:
var isLocal = ( test as System.Data.Objects.ObjectContext ).ObjectStateManager
.GetObjectStateEntries( System.Data.EntityState.Added )
.Any( x => (x.Entity as warehouse).combi == q );
if( !isLocal && !test.warehouses.Any( x => x.combi == q ) ) {
// ...
}
I have to note that IDbSet<> (used by my code first instead of ObjectSet<> created by model designer) has .Local property so there's no need to query ObjectStateManager. I know that code first is completely different thing but you can be very efficient with it, too: design database with Microsoft SQL Server Management Studio and run excellent Reverse Engineer Code First from Entity Framework Power Tools with default or customized (if needed) T4 templates to get almost anything. http://en.wiktionary.org/wiki/your_mileage_may_vary
Related
I am trying to do large inserts using entity framework, so to reduce context bloat I am trying to recreate the context every x loops. However, after recreating the context the data from the previous context is still in the local information.
I have tried calling Dispose() on the context and setting it to null before recreating it.
I found this question asked before where someone seems to have the same issue, but there was no solution found. Entity Framework Context - Recreating Object keeps old object in memory
ApplicationDbContext context = new ApplicationDbContext();
context.Configuration.AutoDetectChangesEnabled = false;
for (int i = 0; i < products.Count; i++)
{
context.Product.Add(products[i]);
if(i % 50 == 0)
{
context.SaveChanges();
context.Dispose();
context = null;
context = new ApplicationDbContext();
context.Configuration.AutoDetectChangesEnabled = false;
}
}
context.SaveChanges();
Context Product local
For clarification, this is what I mean when the local data is not cleared. The local data for the Products has 151 entries locally, despite having the context be recreated every 50 entries. This has lead to the products being added multiple times to the database.
Just to clarify what I am asking, how can I recreate the context so that it does not have an data left over from the previous context? Also, if anyone could explain what causes the data to be retained to begin with that would be nice to understand.
Edit:
I have tried restructuring the code as suggested by by Gert Arnold in the comments.
Here is the current version.
public void BatchAddProducts(List<ProductModel> products)
{
int loops = (int)Math.Ceiling((double)(products.Count / 50));
List<ProductModel> sublist = new List<ProductModel>();
for (int i = 0; i < loops; i++)
{
int toPull;
if((i * 50 + 50) > products.Count)
{
toPull = products.Count - (i * 50);
}
else
{
toPull = 50;
}
sublist = products.GetRange(i * 50, toPull);
ProductAdd(sublist);
}
}
private void ProductAdd(List<ProductModel> products)
{
using(ApplicationDbContext context = new ApplicationDbContext())
{
context.Product.Local.Clear();
context.Product.AddRange(products);
context.SaveChanges();
}
}
This still has the same issue. The context is retaining information from the version that should no longer exist, and entries that were added in the previous loop are being added to the database again. i.e. after 2 iterations 150 entries are added to the database, and after 3 there are 300 new entries in the database.
In response to the answer by Steve Py, I am checking the context.Product.Local information just before SaveChanges() is called. The duplicate data I mention is duplicates of the products that are in the context. So to refer to the example I made earlier, after 3 loops, there are 3 versions of a product from the 1st iteration, 2 from the 2nd, and 1 from the 3rd. I dont believe it is an issue with how the product list is being generated. That was my first thought when I encountered this, and I checked that code. This code is being written as an upgrade to improve large inserts, and the code that generates the products is older that worked fine with the old update system. Also, the list being passed to the function only has ~8000 Products, so it cant be that as far as I can tell.
I suspect that you may be encountering a different issue based on the statement "this has lead to the products being added multiple times to the database" and your process of verifying the local cache is resulting in the DBSet being loaded into memory.
I have tested the disposed DbContext behaviour around the .Local and it is behaving as expected, the previously loaded entities are not still referenced.
[Test]
public void TestLocalReset()
{
var context = new TestDbContext();
for (int count = 1; count <= 100; count++)
{
var note = new Note { Text = $"Note #{count}." };
context.Notes.Add(note);
context.SaveChanges();
if (count % 10 == 0)
{
int test = context.Notes.Local.Count; // <- Always 10
context.Dispose();
context = new TestDbContext();
test = context.Notes.Local.Count; // <- Always 0
}
}
context.Dispose();
}
Your example does not show how/where you are checking the .Local instance, but if you have any code that queries against your DbSet that could result in loading the entire results into memory and showing up in the .Local instance.
As for duplicate data, are you getting duplicate Products, or do you have other referenced entities beneath the Product that you are seeing duplicated? How are you building/populating these Products in your array? References that might have been loaded by the outermost DbContext to be referenced by these new products will not be tracked when you recreate the DbContext resulting in potentially duplicate related entities being inserted.
You may need to update your question to show the complete code you are using to create the Products and insert them as this will likely contain details explaining the behaviour you are seeing.
I'm not sure External Source is the correct phrasing, but essentially I have a view in my database that points to a table in a different database. Not always, but from time to time I get an ORA-12537 Network Session: End of File exception. I'm using Entity Framework, so I tried breaking it up so instead of using one massive query, it does a handful of queries to generate the final result. But this has had a mixed-to-no impact.
public List<SomeDataModel> GetDataFromList(List<string> SOME_LIST_OF_STRINGS)
{
var retData = new List<SomeDataModel>();
const int MAX_CHUNK_SIZE = 1000;
var totalPages = (int)Math.Ceiling((decimal)SOME_LIST_OF_STRINGS.Count / MAX_CHUNK_SIZE);
var pageList = new List<List<string>>();
for(var i = 0; i < totalPages; i++)
{
var chunkItems = SOME_LIST_OF_STRINGS.Skip(i * MAX_CHUNK_SIZE).Take(MAX_CHUNK_SIZE).ToList();
pageList.Add(chunkItems);
}
using (var context = new SOMEContext())
{
foreach(var pageChunk in pageList)
{
var result = (from r in context.SomeEntity
where SOME_LIST_OF_STRINGS.Contains(r.SomeString)
select r).ToList();
result.ForEach(x => retData.Add(mapper.Map<SomeDataModel>(x)));
}
}
return retData;
}
I'm not sure if there's a different approach to dealing with this exception or not, or if breaking up the query has any desired effect. It's probably worth noting that SOME_LIST_OF_STRINGS is pretty large (about 21,000 on average), so totalPages usually sits around 22.
Sometimes, that error can be caused by an excessively large "IN" list in the SQL. For example:
SELECT *
FROM tbl
WHERE somecol IN ( ...huge list of stuff... );
Enabling application or database level tracing could help reveal whether the SQL that's being constructed behind the scenes has a large IN list.
A workaround might be to INSERT "...huge list of stuff..." into a table and then use something similar to the query below in order to avoid the huge list of literals.
SELECT *
FROM tbl
WHERE somecol IN ( select stuff from sometable );
Reference*:
https://support.oracle.com/knowledge/More%20Applications%20and%20Technologies/2226769_1.html
*I mostly drew my conclusions from the part of this reference that's not publicly viewable.
I have 3 tables in my DB which I'm working with:
Theme [Theme_ID]
ThemeWorkplace [Theme_ID, Workplace_ID, ThemeWorkplace_ID]
UserTheme [User_ID, Theme_ID, UserTheme_ID, UserTheme_AccessType]
I need to change UserTheme_AccessType for all UserTheme.Theme_ID in current workplace with ThemeWorkplace.Workplace_ID = 2 and current user with User_ID = 1. If theme is no row in UserTheme for such user and such theme - I need to create it.
I wrote such a code, but it works too long time:
var themeList = (from t in m_Entities.Theme
where (from tw in m_Entities.ThemeWorkplace
where tw.Workplace.Workplace_ID == 2
select tw.Theme.Theme_ID).Contains(t.Theme_ID)
select t)
.ToList();
foreach (Theme theme in themeList)
{
var oldUserTheme = GetByUserTheme(user/*user is given*/, theme);
if (oldUserTheme == null)
{
/* create new User Theme with params, that I need*/
this.Add(newUserTheme, true);
}
else
{
/* here - changing found row */
oldUserTheme.UserTheme_AccessType = 2;
}
}
I understand that this code accesses the database too many times. I want to find a way to get rid of:
var oldUserTheme = GetByUserTheme(user/*user is given*/, theme);
In every foreach iteration. Could somebody please help me?
Adding code of GetByUserTheme():
private UserTheme GetByUserTheme(User user, Theme theme)
{
return m_Entities.UserTheme.FirstOrDefault(ut => ut.User.User_ID == user.User_ID && ut.Theme.Theme_ID == theme.Theme_ID);
}
First: All changes for entities that you have done in code will be pushed to database in one batch command when you call context.SaveChanges. So you will have one request to Database for select and one request for update.
But in your batch command will be many sql queries for cause EF generate sql for updates entities one by one (not all in one).
If you want update really many records in database, you should use sql script (call stored procedure or execute sqlquery) against using EntityFramework.
I don't know if I'm completely understanding your question and structure. But based on what I see, could this be a reasonable slution?
First you select the workplaces that have ID equal to 2. From that result you select the theme-ID's. All your userthemes that have a themeID that occurs in the former result will then be selected into 'userThemes'. From there, you iterate over the results and if the userID is empty you create a new UserTheme, otherwise you update it.
Quick note: the code below is no real working code. It's just code I wrote to exemplify my explanation, kind of pseudo-code if you will.. :)
var userThemes = entities.Userthemes
.Where(ut => entities.Workplaces
.Where(w => w.WorkPlaceID == 2)
.Select(s => s.ThemeID)
.Contains(ut.ThemeID));
foreach (UserTheme ut in userThemes)
{
if (ut.UserID.ToString() == "")
{
//Create
}
else
ut.UserThemeAccessType = 2;
}
I'm developing metro app using Windows 8 release preview and C#(VS 2012), I'm new to SQLite, I integrated SQLite 3.7.13 in my App and it is working fine, Observe my code below
var dbPath = Path.Combine(Windows.Storage.ApplicationData.Current.LocalFolder.Path, "Test.db");
using (var db = new SQLite.SQLiteConnection(dbPath))
{
var data = db.Table<tablename>().Where(tablename => tablename.uploaded_bool == false && tablename.Sid == 26);
try
{
int iDataCount = data.Count();
int id;
if (iDataCount > 0)
{
for (int i = 0; i < iDataCount; i++)
{
Elements = data.ElementAt(i);
id = Elements.id;
/*
Doing some code
*/
}
int i = db.Delete<tablename>(new tablename() { Sid = 26 });
}
}
catch (Exception ex)
{
}
}
where "Sid" is column in my database and with number "26" i will get n number of rows
So, using a for loop i need to do some code and after the for loop I need to delete records of Sid(26) in database, So at this line
int i = db.Delete<tablename>(new tablename() { Sid = 26 });
I'm getting unable to close due to unfinalised statements exception, So my question is how to finalise the statement in sqlite3,Apparently SQLite3 has a finalize method for destroying previous DB calls but I am not sure how to implement this. Please help me.
Under the covers sqlite-net does some amazing things in an attempt to manage queries and connections for you.
For example, the line
var data = db.Table<tablename>().Where(...)
Does not actually establish a connection or execute anything against the database. Instead, it creates an instance of a class called TableQuery which is enumerable.
When you call
int iDataCount = data.Count();
TableQuery actually executes
GenerateCommand("count(*)").ExecuteScalar<int>();
When you call
Elements = data.ElementAt(i);
TableQuery actually calls
return Skip(index).Take(1).First();
Take(1).First() eventually calls GetEnumerator, which compiles a SQLite command, executes it with TOP 1, and serializes the result back into your data class.
So, basically, every time you call data.ElementAt you are executing another query. This is different from standard .NET enumerations where you are just accessing an element in a collection or array.
I believe this is the root of your problem. I would recommend that instead of getting the count and using a for(i, ...) loop, you simply do foreach (tablename tn in data). This will cause all records to be fetched at once instead of record by record in the loop. That alone may be enough to close the query and allow you to delete the table during the loop. If not, I recommend you create a collection and add each SID to the collection during the loop. Then, after the loop go back and remove the sids in another pass. Worst case scenario, you could close the connection between loops.
Hope that helps.
It's quite a simple question: When someone click on "Edit Plan" in my ASP.NET MVC project - He doesn't edit but create a new plan with. You can see it more clearly in my answer to my qeustion here: How do I duplicate an object?
Now I want to do the same thing to its references, and I did it like that:
var featurePlans = db.FeaturePlanBaseSet.Where(f => f.PlanId == plan.Id).ToList();
db.PlanSet.AddObject(plan);
db.SaveChanges();
for (var i = 0; i < featurePlans.Count(); i++ )
{
featurePlans[i].Plan = plan;
db.FeaturePlanBaseSet.AddObject(featurePlans[i]);
}
Plan is added when I do AddObject, but Feature isn't.
I get this error:
An object with the same key already exists in the ObjectStateManager.
The existing object is in the Unchanged state.
I'll be glad to know why does it happens.
It does not appear that you save after adding FeaturePlanBaseSet. You need to call db.SaveChanges() last to save all changes.
EDIT: It also appears that you are reading an existing FeaturePlanBaseSet record from the database and then adding that record back. The next line will retrieve an existing record.
var featurePlans = db.FeaturePlanBaseSet.Where(f => f.PlanId == plan.Id).ToList();
When you add featurePlans[i], you are adding the existing record. If you plan to add a new record, Do it as follows:
for (var i = 0; i < featurePlans.Count(); i++ )
{
var featurePlan = new FeaturePlanBaseSet();
featurePlan.Plan = plan;
...set other properties
db.FeaturePlanBaseSet.AddObject(featurePlan);
}