We Keep Coding

EF Competing SaveChanges() Calls

I am building a batch processing system. Batches of Units come in quantities from 20-1000. Each Unit is essentially a hierarchy of models (one main model and many child models). My task involves saving each model hierarchy to a database as a single transaction (either each hierarchy commits or it rolls back). Unfortunately EF was unable to handle two portions of the model hierarchy due to their potential to contain thousands of records.
What I've done to resolve this is set up SqlBulkCopy to handle these two potentially high count models and let EF handle the rest of the inserts (and referential integrity).
Batch Loop:
foreach (var unitDetails in BatchUnits)
{
var unitOfWork = new Unit(unitDetails);
Task.Factory.StartNew(() =>
{
unitOfWork.ProcessX(); // data preparation
unitOfWork.ProcessY(); // data preparation
unitOfWork.PersistCase();
});
}
Unit:
class Unit
{
public PersistCase()
{
using (var dbContext = new CustomDbContext())
{
// Need an explicit transaction so that
// EF + SqlBulkCopy act as a single block
using (var scope = new TransactionScope(TransactionScopeOption.Required,
new TransactionOptions() {
IsolationLevel = System.Transaction.IsolationLevel.ReadCommitted
}))
{
// Let EF Insert most of the records
// Note Insert is all it is doing, no update or delete
dbContext.Units.Add(thisUnit);
dbContext.SaveChanges(); // deadlocks, DbConcurrencyExceptions here
// Copy Auto Inc Generated Id (set by EF) to DataTables
// for referential integrity of SqlBulkCopy inserts
CopyGeneratedId(thisUnit.AutoIncrementedId, dataTables);
// Execute SqlBulkCopy for potentially numerous model #1
SqlBulkCopy bulkCopy1 = new SqlBulkCopy(...);
...
bulkCopy1.WriteToServer(dataTables["#1"]);
// Execute SqlBulkCopy for potentially number model #2
SqlBulkCopy bulkCopy2 = new SqlBulkCopy(...);
...
bulkCopy2.WriteToServer(dataTables["#2"]);
// Commit transaction
scope.Complete();
}
}
}
}
Right now I'm essentially stuck between a rock and a hard place. If I leave the IsolationLevel set to ReadCommitted, I get deadlocks between EF INSERT statements in different Tasks.
If I set the IsolationLevel to ReadUncommitted (which I thought would be fine since I'm not doing any SELECTs) I get DbConcurrencyExceptions.
I've been unable to find any good information about DbConcurrencyExceptions and Entity Framework but I'm guessing that ReadUncommitted is essentially causing EF to receive invalid "rows inserted" information.
UPDATE
Here is some background information on what is actually causing my deadlocking issues while doing INSERTS:
http://connect.microsoft.com/VisualStudio/feedback/details/562148/how-to-avoid-using-scope-identity-based-insert-commands-on-sql-server-2005
Apparently this same issue was present a few years ago when Linq To SQL came out and Microsoft fixed it by changing how scope_identity() gets selected. Not sure why their position has changed to this being a SQL Server problem when the same issue came up with Entity Framework.

This issue is explained fairly well here: http://connect.microsoft.com/VisualStudio/feedback/details/562148/how-to-avoid-using-scope-identity-based-insert-commands-on-sql-server-2005
Essentially its an internal EF issue. I migrated my code to use Linq To SQL and it now works fine (no longer does the unnecessary SELECT for the identity value).
Relevant quote from the exact same issue in Linq To Sql which was fixed:
When a table has an identity column, Linq to SQL generates extremely
inefficient SQL for insertion into such a table. Assume the table is
Order and the identiy column is Id. The SQL generated is:
exec sp_executesql N'INSERT INTO [dbo].[Order]([Colum1], [Column2])
VALUES (#p0, #p1)
SELECT [t0].[Id] FROM [dbo].[Order] AS [t0] WHERE [t0].[Id] =
(SCOPE_IDENTITY()) ',N'#p0 int,#p1 int,#p0=124,#p1=432
As one can see instead of returning SCOPE_IDENTITY() directly by using
'SELECT SCOPE_IDENTITY()', the generated SQL performs a SELECT on the
Id column using the value returned by SCOPE_IDENTITY(). When the
number of the records in the table is large, this significantly slows
down the insertion. When the table is partitioned, the problem gets
even worse.

linq insert: getting the row number

I have a table with the primary key as the identity column. I'm doing an insert with linq-to-sql and I was wondering if there's a way to return the ID of the row that was inserted and how we know for sure that the insert happened.
Thanks for your suggestions.

The primary key property of the entity that was used for the insert will be updated after the call to SubmitChanges with the ID from the database.
Ex:
using (var dc = new MyDataContext())
{
MyEntity entity = new MyEntity();
dc.MyEntities.InsertOnSubmit(entity);
dc.SubmitChanges();
int pkValue = entity.PKColumn
}

With LINQ-to-SQL it will update any IDENTITY (etc) properties during the save automatically, so just look at .Id (or whatever).
Re knowing it happened: updates etc happen in a transaction; that transaction must have committed to have got out of the method without an exception. If you have an outer TransactionScope (or similar) then that it up to you to commit; in short - until you do commit you have a phantom record. As such, avoid passing that new id anywhere until you have committed all your transactions.

The identity of the inserted record should automatically populate the object which was added to the DataContext for the insert. Just check the ID on that object.
Additionally, you can look at the GetChangeSet() method on the DataContext to see what changes it has tracked if you want more manual information about the number of affected records, etc..

How to automatically generate identity for an Oracle database through Entity framework?

I'm using Oracle provider for Entity framework (beta), and I'm facing a problem.
Our tables have Id columns, which are set to be Identity in StoreGeneratedPattern. I thought that EF will automatically do "underlying works", such as create sequences, and get new identity for each record I add to the table. But when I run code to add a new record, such as:
var comment = new Comment
{
ComplaintId = _currentComplaintId,
Content = CommentContent.Text,
CreatedBy = CurrentUser.UserID,
CreatedDate = DateTime.Now
};
context.Comments.AddObject(comment);
context.SaveChanges();
an Exception still throws, which is
{"ORA-00001: unique constraint (ADMINMGR.CONSTRAINT_COMMENT)
violated"}
(CONSTRAINT_COMMENT is the constrain requires that comment identity
must be unique.
How do I solve this?
Thank you very much!

StoreGeneratedPattern="Identity" simply tells EF that the value will be generated DB-side on insert, and that it shouldn't supply a value in insert statements.
You still need to create a sequence in Oracle:
create sequence ComplaintIdSequence minvalue 1 maxvalue 9999999 start with 1 increment by 1;
and a trigger to make table inserts use it:
create or replace trigger CommplaintIdTrigger
before insert on comment for each row
begin
if :new.ComplaintId is null then select ComplaintIdSequence.nextval into :new.ComplaintId from dual;
endif;
end;

Oracle 12c has resolved it
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int SomeNumber { get; set; }

Another option would be:
Create a sequence the way Alextansc described.
Create a stored procedure that uses MySequence.nextval as it's primary key.
Map 'insert' for this model to your stored procedure and it works!
I've tested this using database first approach.
Using database first mapping to a stored procedure is pretty simple. Go to your edmx file and right click the model you want to map to a stored procedure. Click "stored procedure mappings." The dialog at the bottom of the page gives you three drop down menus for mapping insert, update, and delete to stored procedures.

I am using Oracle ODP.NET, Managed driver and Entity Framework 6. I created my tables using the code-first approach but wasn't able to add any records due to a null primary key.
The solution was to grant my user both:
'CREATE SEQUENCE' and
'CREATE TRIGGER'
permissions and re-create the schema.
I realized this after using the -verbose flag in the package management console

Instead of remember all of this SQL, you could easily do by using Mig# like this:
var schema = new DbSchema(ConnectionString, DbPlatform.Oracle12c);
schema.Alter(db => db.CreateTable("TableName")
.WithPrimaryKeyColumn("Id", DbType.Int32).AsIdentity()
...);
In this example, the Id column will have the required trigger and sequence generated by Mig# automatically.

Database doubly connected relationship inserting problem

I have two tables Plants and Information. For every plant there are many information, but for each plant there is a single MainInformation. So there is a one-to-many relationship and a one-to-one relationship between the two. The Information table has a PlantID and the Plants table has a MainInformationID. I want both fields in both tables not to be nulls. But now you can't insert either of the two records into their tables because each one requires their fields not be null, meaning they need the other record to be created first in order to create themselves. Perhaps this is not a good database design and something should be changed? (I am new to databases and entity framework)
I tried inserting into the database itself manually but I cant do it. I also tried this code with EntityFramework.
using (var context = new MyEntities())
{
var p = new Plant()
{
LatinName = "latinNameTest",
LocalName = "localNameTest",
CycleTime = 500
};
var i = new Information()
{
ShortDescription = "ShortDesc",
LongDescription = "LongDesc"
};
p.MainInformation = i;
i.Plant = p;
context.AddToPlants(p);
context.AddToInformation(i);
context.SaveChanges();
}

One of
The 1-1 FK column has to be NULL
The FK has to be disabled to allow parent insert before child
You have a single dummy Information row that is used by default in FL column
SQL Server does not allow deferred constraint checking without "code change" rights so even wrapping in a transaction won't work
Sounds like an EAV schema, which has other problems

You need to change the tables to allow for null. There is no other way to do this.

You may want to look at database transactions and how to use them with the Entity Framework. You can wrap both INSERTS into a single db transaction so the only results are both of them go in or neither go in.
Here is a link for transactions using EF. I didn't read through it but it seems to talk about them enough to get you started.

TSQL: UPDATE with INSERT INTO SELECT FROM

so I have an old database that I'm migrating to a new one. The new one has a slightly different but mostly-compatible schema. Additionally, I want to renumber all tables from zero.
Currently I have been using a tool I wrote that manually retrieves the old record, inserts it into the new database, and updates a v2 ID field in the old database to show its corresponding ID location in the new database.
for example, I'm selecting from MV5.Posts and inserting into MV6.Posts. Upon the insert, I retrieve the ID of the new row in MV6.Posts and update it in the old MV5.Posts.MV6ID field.
Is there a way to do this UPDATE via INSERT INTO SELECT FROM so I don't have to process every record manually? I'm using SQL Server 2005, dev edition.

The key with migration is to do several things:
First, do not do anything without a current backup.
Second, if the keys will be changing, you need to store both the old and new in the new structure at least temporarily (Permanently if the key field is exposed to the users because they may be searching by it to get old records).
Next you need to have a thorough understanding of the relationships to child tables. If you change the key field all related tables must change as well. This is where having both old and new key stored comes in handy. If you forget to change any of them, the data will no longer be correct and will be useless. So this is a critical step.
Pick out some test cases of particularly complex data making sure to include one or more test cases for each related table. Store the existing values in work tables.
To start the migration you insert into the new table using a select from the old table. Depending on the amount of records, you may want to loop through batches (not one record at a time) to improve performance. If the new key is an identity, you simply put the value of the old key in its field and let the database create the new keys.
Then do the same with the related tables. Then use the old key value in the table to update the foreign key fields with something like:
Update t2
set fkfield = newkey
from table2 t2
join table1 t1 on t1.oldkey = t2.fkfield
Test your migration by running the test cases and comparing the data with what you stored from before the migration. It is utterly critical to thoroughly test migration data or you can't be sure the data is consistent with the old structure. Migration is a very complex action; it pays to take your time and do it very methodically and thoroughly.

Probably the simplest way would be to add a column on MV6.Posts for oldId, then insert all the records from the old table into the new table. Last, update the old table matching on oldId in the new table with something like:
UPDATE mv5.posts
SET newid = n.id
FROM mv5.posts o, mv6.posts n
WHERE o.id = n.oldid
You could clean up and drop the oldId column afterwards if you wanted to.

The best you can do that I know is with the output clause. Assuming you have SQL 2005 or 2008.
USE AdventureWorks;
GO
DECLARE #MyTableVar table( ScrapReasonID smallint,
Name varchar(50),
ModifiedDate datetime);
INSERT Production.ScrapReason
OUTPUT INSERTED.ScrapReasonID, INSERTED.Name, INSERTED.ModifiedDate
INTO #MyTableVar
VALUES (N'Operator error', GETDATE());
It still would require a second pass to update the original table; however, it might help make your logic simpler. Do you need to update the source table? You could just store the new id's in a third cross reference table.

Heh. I remember doing this in a migration.
Putting the old_id in the new table makes both the update easier -- you can just do an insert into newtable select ... from oldtable, -- and the subsequent "stitching" of records easier. In the "stitch" you'll either update child tables' foreign keys in the insert, by doing a subselect on the new parent (insert into newchild select ... (select id from new_parent where old_id = oldchild.fk) as fk, ... from oldchild) or you'll insert children and do a separate update to fix the foreign keys.
Doing it in one insert is faster; doing it in a separate step meas that your inserts aren't order dependent, and can be re-done if necessary.
After the migration, you can either drop the old_id columns, or, if you have a case where the legacy system exposed the ids and so users used the keys as data, you can keep them to allow use lookup based on the old_id.
Indeed, if you have the foreign keys correctly defined, you can use systables/information-schema to generate your insert statements.

Is there a way to do this UPDATE via INSERT INTO SELECT FROM so I don't have to process every record manually?
Since you wouldn't want to do it manually, but automatically, create a trigger on MV6.Posts so that UPDATE occurs on MV5.Posts automatically when you insert into MV6.Posts.
And your trigger might look something like,
create trigger trg_MV6Posts
on MV6.Posts
after insert
as
begin
set identity_insert MV5.Posts on
update MV5.Posts
set ID = I.ID
from inserted I
set identity_insert MV5.Posts off
end

AFAIK, you cannot update two different tables with a single sql statement
You can however use triggers to achieve what you want to do.

Make a column in MV6.Post.OldMV5Id
make a
insert into MV6.Post
select .. from MV5.Post
then make an update of MV5.Post.MV6ID