I have a system implemented in C# API .Net Core 6. The database is SQL Server and I'm using Entity Framework. I have a table of individual stock items (each item has its own record). When I sell an item I record its sale date by changing the value of the 'Sold' field from NULL to DateTime.UtcNow. When I need to sell an item I want to get a reference to one of the items of that category in my stock table which I do with a ctx.Stock.Where(t=> t.Sold == null && t.Category==cat).First() that returns me a single item if it exists. That item then has its 'Sold' field updated.
Since the system is accessed via an API, it would seem possible that the code could be called twice (or more) at the same time, both calls would return the same item from the ctx.Stock.Where(t=> t.Sold == null && t.Category==cat).First() , both would then mark it as sold and when SaveChanges was called for both of them, it would save the later 'Sold' value and only 1 stock Record would be recorded as sold when in fact 2 sales had happened.
Now this could be avoided if the DbUpdateConcurrencyException is triggered when this type of situation occurs. Does anyone know if this would be the case? In that case I guess a retry loop could be implemented to attempt to get a new stock record using ctx.Stock.Where(t=> t.Sold == null && t.Category==cat).First() . Would this be the best way of doing it?
Otherwise would having a singleton service that has locks around the running of ctx.Stock.Where(t=> t.Sold == null && t.Category==cat).First() and SaveChanges of the populated 'Sold' field be a better bet? This could ensure there are no possible DbUpdateConcurrencyException as it would be sequential, but then maybe the locks would be a problem.
I'm not much of a fan of trusting the database. If you want to be able to scale your app horizontally and have access to Redis Cache, you could use RedLock to prevent the two calls from hitting the DB at the same time. Otherwise just use a simple lock. The service doesn't have to be a singleton... you just need to use a static object for the locking.
I have the following scenario:
A user can create orders having a given amount (e.g.: 500$)
There is a limit for the total orders' amount that can be added for a single day (e.g.: max 2000$/day)
At the moment, when creating a new order, this requirement is implemented as follows:
var newOrder = /* logic for creating the new order */;
var orders = _ordersRepository.GetAllBy(userId, date); // get the orders from the db
var totalAmount = orders.Sum(o => o.Amount);
if(totalAmount < MaximumAmount) {
newOrder.IsApproved = true;
}
else {
newOrder.IsApproved = false;
}
_ordersRepository.Add(newOrder);
_ordersRepository.SaveChanges(); // insert into the db
The problem with this approach is that it doesn't handle concurrent insertions properly in scenarios like:
Maximum orders limit: 2000$
In a single second, a user sends 10 requests for creating 10 new orders of 500$ each
The requests are handled concurrently and because of the short timeframe, the currently implemented check is executed before the new orders are saved in the database and therefore allows creating all of them. In the end, the user ends exceeding the maximum limit.
How could I solve this issue, ideally without having to call SaveChanges multiple times? I'm using Entity Framework Core 5 and SQL Server.
Thank you for the suggestions from the comments!
I've tried using transactions with the IsolationLevel set to Serializable, but then I've realized this would lock too many tables (the example from the question is a dummy one, the actual implementation is more complex).
I agree that sometimes it might be easier to have this kind of logic in the database, but adding a stored procedure for this will kind of break the current consistency and most probably leave the door open for other stored procedures. I'm not saying stored procedures are bad, just that in my current situation, even if it is a bit harder/more complex to achieve this without them, I believe it's worth it for consistency reasons.
The solution I've ended with
I've ended splitting the flow in 2 steps as follows:
// step 1
var newOrder = /* logic for creating the new order */;
_ordersRepository.Add(newOrder);
_ordersRepository.SaveChanges(); // insert into the db
// step 2
var orders = _ordersRepository.GetAllBy(userId, date); // get the orders from the db
var totalAmount = orders.Sum(o => o.Amount);
if(totalAmount < MaximumAmount) {
newOrder.IsApproved = true;
}
_ordersRepository.Update(newOrder);
_ordersRepository.SaveChanges(); // update the new order
Step 1 just creates the new order and inserts it into the database, the IsApproved flag being left to the default which is false.
Step 2 performs the daily limit validation and if the check passes, the IsApproved flag to true.
I know it's not an actual solution, but a workaround. Locking a table might have a too big performance impact, especially if the given table is used by multiple app features. With this solution, even if there was an issue in the Step 2, the order will be left with IsApproved=false so it won't have any impact and the user can either try again later, or somebody from support can handle it.
I am trying to read all new rows that are added to the database on a timer.
First I read the entire database and save it to a local data table, but I want to read all new rows that are added to the database. Here is how I'm trying to read new rows:
string accessDB1 = string.Format("SELECT * FROM {0} ORDER BY ID DESC", tableName);
setupaccessDB(accessDB1);
int dTRows = localDataTable.Rows.Count + 1;
localDataTable.Rows.Add();
using (readNext = command.ExecuteReader())
{
while (readNext.Read())
{
for (int xyz = 0; xyz < localDataTable.Columns.Count; xyz++)
{
// Code
}
break;
}
}
If only 1 row is added within the timer then this works fine, but when multiple rows are added this only reads the latest row.
So is there any way I can read all added rows.
I am using OledbDataReader.
Thanks in advance
For most tables the primary key is based an incremental value. This can be a very simple integer that is incremented by one, but it could also be a datetime based guid.
Anyway if you know the id of the last record. You can simple ask for all records that have a 'higher' id. In that way you do get the new records, but what about updated records? If you also want those you might want to use a column that contains a datetime value.
A little bit more trickier are records that are deleted from the database. You can't retrieve those with a basic query. You could solve that by setting a TTL for each record you retrieve from the database much like a cache. When the record is 'expired', you try to retrieve it again.
Some databases like Microsoft SQL Server also provide more advanced options into this regard. You can use query notifications via the broker services or enable change tracking on your database. The last one can even indicate what was the last action per record (insert, update or delete).
Your immediate problem lies here:
while (readNext.Read())
{
doSomething();
break;
}
This is what your loop basically boils down to. That break is going to exit the loop after processing the first item, regardless of how many items there are.
The first item, in this case, will probably be the last one added (as you state it is) since you're sorting by descending ID.
In terms of reading only newly added rows, there are a variety of ways to do it, some which will depend on the DBMS that you're using.
Perhaps the simplest and most portable would be to add an extra column processed which is set to false when a row is first added.
That way, you can simply have a query that looks for those records and, for each, process them and set the column to true.
In fact, you could use triggers to do this (force the flag to false on insertion) which opens up the possibility for doing it with updates as well.
Tracking deletions is a little more difficult but still achievable. You could have a trigger which actually writes the record to a separate table before deleting it so that your processing code has access to those details as well.
The following works
using (readNext = command.ExecuteReader())
{
while (readNext.Read())
{
abc = readNext.FieldCount;
for (int s = 1; s < abc; s++)
{
var nextValue = readNext.GetValue(s);
}
}
}
The For Loop reads the current row and then the While Loop moves onto the next row
I am developing an sync database application for one of my client i am done with synchronization but now i stuck at a part that i don`t want to sync complete table i just wana sync dynamically selected record through sync framework
using sqlce at client end
Any good link which make some help will also be preferable
i am sync ing some thing like below at server
public SyncContext GetChanges(SyncGroupMetadata groupMetadata, SyncSession syncSession)
{
return serverSyncProvider.GetChanges(groupMetadata, syncSession);
}
public SyncContext ApplyChanges(SyncGroupMetadata groupMetadata, DataSet dataSet, SyncSession syncSession)
{
return serverSyncProvider.ApplyChanges(groupMetadata, dataSet, syncSession);
}
you can add a filter to restrict the rows being synched. Sync Framework however does not support dynamic filtering. Sync framework does synchronization based on incremental syncs of what has changed since the last sync.
for example, assuming you have a customer table and you let a user pick which customer to sync.
you will most likely store the customers that was picked for synching in another table, let's say SelectedCustomer which would store the user id. and the customer id.
when a user adds or removes a customer from his selection, you would update the SelectedCustomer. However, when you sync Customer table, you won't get anything not unless the specific customer record has been changed since the last sync. Even if a customer was added to a specific user, the actual customer record is not updated, no update, no change detected.
Here is the article written by JuneT ;)
Sync Framework Provisioning
Go to Parameter-based Filters section
JuneT's answer is absolutely correct. The trick is if you select to a row to sync, update that row first before synching (so that tracking table will record the row as updated).
Update SelectedCustomer is not enough, the tracking record is syncronized but the real one is not. I had to create a insert/update trigger to update the local_create_peer_timestamp from the master tracking record.
I can't believe it is so hard to get someone to show me a simple working example. It leads me to believe that everyone can only talk like they know how to do it but in reality they don't.
I shorten the post down to only what I want the example to do. Maybe the post was getting to long and scared people away.
To get this bounty I am looking for a WORKING EXAMPLE that I can copy in VS 2010 and run.
What the example needs to do.
Show what datatype should be in my domain for version as a timestamp in mssql 2008
Show nhibernate automatically throwing the "StaleObjectException"
Show me working examples of these 3 scenarios
Scenario 1
User A comes to the site and edits Row1. User B comes(note he can see Row1) and clicks to edit Row1, UserB should be denied from editing the row until User A is finished.
Scenario 2
User A comes to the site and edits Row1. User B comes 30mins later and clicks to edit Row1. User B should be able to edit this row and save. This is because User A took too long to edit the row and lost his right to edit.
Scenario 3
User A comes back from being away. He clicks the update row button and he should be greeted with StaleObjectException.
I am using asp.net mvc and fluent nhibernate. Looking for the example to be done in these.
What I tried
I tried to build my own but I can't get it throw the StaleObjectException nor can I get the version number to increment. I tired opening 2 separate browser and loaded up the index page. Both browsers showed the same version number.
public class Default1Controller : Controller
{
//
// GET: /Default1/
public ActionResult Index()
{
var sessionFactory = CreateSessionFactory();
using (var session = sessionFactory.OpenSession())
{
using (var transaction = session.BeginTransaction())
{
var firstRecord = session.Query<TableA>().FirstOrDefault();
transaction.Commit();
return View(firstRecord);
}
}
}
public ActionResult Save()
{
var sessionFactory = CreateSessionFactory();
using (var session = sessionFactory.OpenSession())
{
using (var transaction = session.BeginTransaction())
{
var firstRecord = session.Query<TableA>().FirstOrDefault();
firstRecord.Name = "test2";
transaction.Commit();
return View();
}
}
}
private static ISessionFactory CreateSessionFactory()
{
return Fluently.Configure()
.Database(MsSqlConfiguration.MsSql2008
.ConnectionString(c => c.FromConnectionStringWithKey("Test")))
.Mappings(m => m.FluentMappings.AddFromAssemblyOf<TableA>())
// .ExposeConfiguration(BuidSchema)
.BuildSessionFactory();
}
private static void BuidSchema(NHibernate.Cfg.Configuration config)
{
new NHibernate.Tool.hbm2ddl.SchemaExport(config).Create(false, true);
}
}
public class TableA
{
public virtual Guid Id { get; set; }
public virtual string Name { get; set; }
// Not sure what data type this should be for timestamp.
// To eliminate changing to much started with int version
// but want in the end timestamp.
public virtual int Version { get; set; }
}
public class TableAMapping : ClassMap<TableA>
{
public TableAMapping()
{
Id(x => x.Id);
Map(x => x.Name);
Version(x => x.Version);
}
}
Will nhibernate stop the row from being retrieved?
No. Locks are only placed for the extent of a transaction, which in a web application ends when the request ends. Also, the default type of transaction isolation mode is Read committed which means that read locks are released as soon as the select statement terminates. If you are reading and making edits in the same request and transaction, you could place a read and write lock on the row at hand which would prevent other transactions from writing to or reading from that row. However, this type of concurrency control doesn't work well in a web application.
Or would the User B be able to still see the row but if he tried to save it would crash?
This would happen if [optimistic concurrency] was being used. In NHibernate, optimistic concurrency works by adding a version field. Save/update commands are issued with the version upon which the update was based. If that differs from the version in the database table, no rows are updated and NHibernate will throw.
What happens if User A say cancels and does not edit. Do I have to
release the lock myself or is there a timeout can be set to release
the lock?
No, the lock is released at the end of the request.
Overall, your best bet is to opt for optimistic concurrency with version fields managed by NHibernate.
How does it look in code? Do I setup in my fluent nhibernate to
generate a timestamp(not sure if I would timespan datatype).
I would suggest using a version column. If you're using FluentNhibernate with auto mappings, then if you make a column called Version of type int/long it will use that to version by default, alternatively you can use the Version() method in the mapping to do so (it's similar for timestamp).
So now I generated somehow the timestamp and the user is editing a
row(through a gui). Should I be storing the timestamp in memory or
something? Then when the user submits call from memory the timestamp
and id of the row and check?
When the user starts editing a row, you retrieve it and store the current version (the value of the version property). I would recommend putting the current version in a hidden field in the form. When the user saves his changes, you can either do a manual check against the version in the database (check that it's the same as the version in the hidden field), or you can set the version property to the value from the hidden field (if you are using databinding, you could do this automatically). If you set the version property, then when you try to save the entity, NHibernate will check that the version you're saving matches the version in the database, and throws an exception if it doesn't.
NHibernate will issue an update query something like:
UPDATE xyz
SET ,
Version = 16
WHERE Id = 1234 AND Version = 15
(assuming your version was 15) - in the process it will also increment the version field
If so that means the business logic is keeping track of the "row
locking" but in theory someone still could just go Where(x => x.Id ==
id) and grab that row and update at will.
If someone else updates the row via NHibernate, it will increment the version automatically, so when your user tries to save it with the wrong version you will get an exception which you need to decide how to handle (ie. try show some merge screen, or tell the user to try again with the new data)
What happens when the row gets updated? Do you set null to the timestamp?
It updates the version or timestamp (timestamp will get updated to the current time) automatically
What happens if the user never actually finishes updating and leaves. How does the row
every become unlocked again?
The row is not locked per se, it is instead using optimistic concurrency, where you assume that no-one will change the same row at the same time, and if someone does, then you need to retry the update.
Is there still a race condition what happens or is this next to
impossible in happening? I am just concerned 2 ppl try to get edit the
same row and both of them see it in their gui for editing but one is
actually going to get denied in the end because they lost the race
condition.
If 2 people try to edit the same row at the same time, one of them will lose if you're using optimistic concurrency. The benefit is that they will KNOW that there was a conflict, as opposed to either losing their changes and thinking that it updated, or overwriting someone else's changes without knowing about it.
So I did something like this
var test = session.Query.Where(x => x.Id ==
id).FirstOrDefault(); // send to user for editing. Has versioning on
it. user edits and send back the data 30mins later.
Codes does
test.Id = vm.Id; test.ColumnA = vm.ColumnA; test.Version = vm.Version;
session.Update(test); session.Commit(); So the above will work right?
The above will throw an exception if someone else has gone in and changed the row. That's the point of it, so you know that a concurrency issue has arisen. Typically you'd show the user a message saying "Someone else has changed this row" with the new row there and possibly their changes also so the user has to select which changes win.
but if I do this
test.Id = vm.Id;
test.ColumnA = vm.ColumnA;
session.Update(test);
session.Commit(); it would not commit right?
Correct as long as you haven't reloaded test (ie. you did test = new Xyz(), not test = session.Load() ) because the Timestamp on the row wouldn't match
If someone else updates the row via NHibernate, it will increment the
version automatically, so when your user tries to save it with the
wrong version you will get an exception which you need to decide how
to handle (ie. try show some merge screen, or tell the user to try
again with the new data)
Can I make it so when the record is grabbed this checked. I want to
keep it simple at first that only one person can edit at a time. The
other guy won't even be able to access the record to edit while
something is editing it.
That's not optimistic concurrency. As a simple answer you could add a CheckOutDate property which you set when someone starts editing it, and set it to null when they finish. Then when they start to edit, or when you show them the rows to edit you could exclude all rows where that CheckOutDate is newer than say the last 10 minutes (then you wouldn't need a scheduled task to reset it periodically)
The row is not locked per se, it is instead using optimistic
concurrency, where you assume that no-one will change the same row at
the same time, and if someone does, then you need to retry the update.
I am not sure what your saying does this mean I can do
session.query.Where(x => x.id == id).FirstOrDefault(); all day
long and it will keep getting me the record(thought it would keep
incrementing the version).
The query will NOT increment the version, only an update to it will increment the version.
I don't know that much about nHibernate itself, but if you are prepared to create some stored procs on the database it can >sort of< be done.
You will need one extra data column and two fields in your object model to store information against each row:
A 'hash' of all the field values (using SQL Server CHECKSUM 2008 and later or HASHBYTES for earlier editions) other than the hash field itself and the EditTimestamp field. This could be persisted to the table using INSERT/UPDATE triggers if needs be.
An 'edit-timestamp' of type datetime.
Change your procedures to do the following:
The 'select' procedure should include a where clause similar to 'edit-timestamp < (Now - 30 minutes)' and should update the 'edit-timestamp' to the current time. Run the select with appropriate locking BEFORE updating the row I'm thinking a stored procedure with hold locking such as this one here Use a persistent date/time rather than something like GETDATE().
Example (using fixed values):
BEGIN TRAN
DECLARE #now DATETIME
SET #now = '2012-09-28 14:00:00'
SELECT *, #now AS NewEditTimestamp, CHECKSUM(ID, [Description]) AS RowChecksum
FROM TestLocks
WITH (HOLDLOCK, ROWLOCK)
WHERE ID = 3 AND EditTimestamp < DATEADD(mi, -30, #now)
/* Do all your stuff here while the record is locked */
UPDATE TestLocks
SET EditTimestamp = #now
WHERE ID = 3 AND EditTimestamp < DATEADD(mi, -30, #now)
COMMIT TRAN
If you get a row back from this procedure then you 'have' the 'lock', otherwise, no rows will be returned and there's nothing to edit.
The 'update' procedure should add a where clause similar to 'hash = previously returned hash'
Example (using fixed values):
BEGIN TRAN
DECLARE #RowChecksum INT
SET #RowChecksum = -845335138
UPDATE TestLocks
SET [Description] = 'New Description'
WHERE ID = 3 AND CHECKSUM(ID, [Description]) = #RowChecksum
SELECT ##ROWCOUNT AS RowsUpdated
COMMIT TRAN
So in your scenarios:
User A edits a row. When you return this record from the database, the 'edit-timestamp' has been updated to the current time and you have a row so you know you can edit. User B would not get a row because the timestamp is still too recent.
User B edits the row after 30 minutes. They get a row back because the timestamp has passed more than 30 minutes ago. The hash of the fields will be the same as for user A 30 minutes ago as no updates have been written.
Now user B updates. The previously retrieved hash still matches the hash of the fields in the row, so the update statement succeeds, and we return the row-count to show that the row was updated. User A however, tries to update next. Because the value of the description field has changed, the hashvalue has changed, and so nothing is updated by the UPDATE statement. We get a result of 'zero rows updated' so we know that either the row has since been changed or the row was deleted.
There are probably some issues regarding scalability with all these locks going on and the above code could be optimised (might get problems with clocks going forward/back for example, use UTC), but I wrote these examples just to explain how it could work.
Outside of that I can't see how you can do this without utilising database level row-locking within the select transaction. It might be that you can request those locks via nHibernate, but that's beyond my knowledge of nHibernate I'm afraid.
Have you looked at the ISaveOrUpdateEventListener interface?
public class SaveListener : NHibernate.Event.ISaveOrUpdateEventListener
{
public void OnSaveOrUpdate(NHibernate.Event.SaveOrUpdateEvent e)
{
NHibernate.Persister.Entity.IEntityPersister p = e.Session.GetEntityPersister(null, e.Entity);
if (p.IsVersioned)
{
//TODO: check types etc...
MyEntity m = (MyEntity) e.Entity;
DateTime oldversion = (DateTime) p.GetVersion(m, e.Session.EntityMode);
DateTime currversion = (DateTime) p.GetCurrentVersion(m.ID, e.Session);
if (oldversion < currversion.AddMinutes(-30))
throw new StaleObjectStateException("MyEntity", m.ID);
}
}
}
Then in your Configuration, register it.
private static void Configure(NHibernate.Cfg.Configuration cfg)
{
cfg.EventListeners.SaveOrUpdateEventListeners = new NHibernate.Event.ISaveOrUpdateEventListener[] {new SaveListener()};
}
public static ISessionFactory CreateSessionFactory()
{
return Fluently.Configure().Database(...).
.Mappings(...)
.ExposeConfiguration(Configure)
.BuildSessionFactory();
}
And version the Properties you want to version in your Mapping class.
public class MyEntityMap: ClassMap<MyENtity>
{
public MyEntityMap()
{
Table("MyTable");
Id(x => x.ID);
Version(x => x.Timestamp);
Map(x => x.PropA);
Map(x => x.PropB);
}
}
The short answer to your question is you can't/shouldn't do this in a simple web application with nhibernates optimistic (version) and pessimistic (row locks) locking. The fact that your transactions are only as long as a request are your limiting factor.
What you CAN do is create another table and entity class, and mappings that manages these "locks". At the lowest level you need an Id of the object being edited and the Id of the user performing the editing, and a datetime of when the lock was acquired. I would make the Id of the object being edited the primary key since you want it to be exclusive...
When a user clicks on a row to edit, you can try to acquire a lock (create a new record in that table with the ids and current datetime). If the lock already exists for another user, than it will fail because you are trying to violate a primary key constraint.
If a lock is acquired, when the user clicks save you need to check that they still have a valid "lock" before performing the actual save. Then, perform the actual save and remove the lock record.
I would also recommend a background service/process that sweeps these locks periodically and removes the ones that have expired or are older than your time limit.
This is my prescribed way of dealing with "locks" in a web environment. Good luck!
Yes, it is possible to lock a row with nhibernate but if I understand well, your scenario is in a web context and then it is not the best practice.
The best practive is to use optimistic locking with automatic versioning as you mentioned.
Locking a row when page is opening and releasing it when page is unloading will quickly lead to dead lock the row (javascript issue, page not killed properly...).
Optimistic locking will make NHibernate throws an exception when flushing a transaction which contains objects modified by another session.
If you want to have true concurent modification of the same information you may try to think about a system which merge many users input inside a same document, but it is a system on its own, not managed by ORM.
You will have to choose a way to deal with session in a web environment.
http://nhibernate.info/doc/nh/en/index.html#transactions-optimistic
The only approach that is consistent with high concurrency and high
scalability is optimistic concurrency control with versioning.
NHibernate provides for three possible approaches to writing
application code that uses optimistic concurrency.
Hey you can try these sites
http://thesenilecoder.blogspot.ca/2012/02/nhibernate-samples-row-versioning-with.html
http://stackingcode.com/blog/2010/12/09/optimistic-concurrency-and-nhibernate