I have developed an application to online purchasing my products.
I have a product "Umbrellas" in my store with 100 pieces. I have developed an application to online purchasing my products.
But there is an issue when there is a concurrent purchasing.
If there is a two concurrent purchasing happening the AvailableQty will update incorrectly. Let's say there are two transactions happening concurrently with Purchasing Qty as 100 & 50. Ideally, the first transaction (purchase qty is 100) should be successful as we have 100 stocks available. But the second transaction should return an error because the stock is insufficient to process as with the first transaction the balance is 0. (100 - 100). But above scenario both transactions are successful and the balance shows as -50 now.
This will work correctly when there are two separate transactions. But this is an issue when this two transactions happening CONCURRENTLY. The reason for this problem is, when concurrent transactions the condition to check the availability hits same time, in that time the condition is satisfied as the DB table has not updated with the latest qty.
How can I correct this?
public bool UpdateStock(int productId, int purchaseQty)
{
using(var db = new MyEntities())
{
var stock = db.Products.Find(productId);
if (stock.AvailableQty >= purchaseQty) // Condition to check the availablity
{
stock.AvailableQty = stock.AvailableQty - purchaseQty;
db.SaveChanges();
return true;
}
else
{
return false;
}
}
}
This is typical thread concurrency issue which can be resolved in multiple ways, one of them is using simple lock statement:
public class StockService
{
private readonly object _availableQtyLock = new object();
public bool UpdateStock(int productId, int purchaseQty)
{
using (var db = new MyEntities())
{
lock (_availableQtyLock)
{
var stock = db.Products.Find(productId);
if (stock.AvailableQty >= purchaseQty) // Condition to check the availablity
{
stock.AvailableQty = stock.AvailableQty - purchaseQty;
db.SaveChanges();
return true;
}
return false;
}
}
}
}
Only one thread can get a exclusive rights to get a lock on _availableQtyLock, which means other thread will have to wait for the first thread to release lock on that object.
Take into account this is the simplest (and possibly slowest) way of dealing with concurrency, there are other ways to do thread synchronization, e.g. Monitor, Semaphore, fast SlimLock etc... Since it's hard to tell which one will suit your needs the best, you'll need to do proper performance/stress testing, but my advice would be to start with simplest.
Note: As others mentioned in comments, concurrency issues can be done on DB level as well, which indeed would be more suitable, but if you don't want/can't introduce any DB changes, this would be way to go
Related
I have a webapi and I want to make my logic inside this controller thread safe.
I want user can only update payroll when the last one updated and two update at the same time should not be happend.
As you can see in the code, I added a column in Payroll entity with the name of IsLock as boolean and try to handle multiple request for update in this way but it is not thread-safe.
How can I make it thread-safe ?
[HttpPut("{year}/{month}")]
public async Task<NoContentResult> Approve([FromRoute] int year, [FromRoute] int month)
{
var payroll = _dataContext.Payrolls
.SingleOrDefaultAsync(p =>
p.Month == month && p.Year == year);
if (payroll.IsLock)
{
throw new ValidationException(
$"The payroll {payroll.Id} is locked.");
}
try
{
payroll.IsLock = true;
_dataContext.Payrolls.Update(payroll);
await _dataContext.SaveChangesAsync(cancellationToken);
payroll.Status = PayrollStatus.Approved;
_dataContext.Payrolls.Update(payroll);
await _dataContext.SaveChangesAsync(cancellationToken);
payroll.IsLock = false;
_dataContext.Payrolls.Update(payroll);
await _dataContext.SaveChangesAsync(cancellationToken);
return NoContent();
}
catch (Exception)
{
payroll.IsLock = false;
_dataContext.Payrolls.Update(payroll);
await _dataContext.SaveChangesAsync(cancellationToken);
throw;
}
}
You are looking for Concurrency Tokens. Each row in the payroll table would have one. When a user loaded the edit interface for a payroll, the concurrency token would be sent to the client. The client would include the concurrency token in the request to update the payroll. The update would only succeed of the concurrency token had not changed - meaning that the data had not changed since the user fetched it to start the edit.
Entity Framework uses the concurrency tokens internally, as well, so it won't save changes from a stale entity (where the data has changed since it was loaded).
The current IsLocked solution has some flaws. If two API requests are received at the same time, both may read the payroll data and see that it isn't locked. Both requests would then lock the row, make competing changes, and release the lock without ever realizing there were simultaneous edits.
I have an API that people are calling and I have a database containing statistics of the number of requests. All API requests are made by a user in a company. There's a row in the database per user per company per hour. Example:
| CompanyId | UserId| Date | Requests |
|-----------|-------|------------------|----------|
| 1 | 100 | 2020-01-30 14:00 | 4527 |
| 1 | 100 | 2020-01-30 15:00 | 43 |
| 2 | 201 | 2020-01-30 14:00 | 161 |
To avoid having to make a database call on every request, I've developed a service class in C# maintaining an in-memory representation of the statistics stored in a database:
public class StatisticsService
{
private readonly IDatabase database;
private readonly Dictionary<string, CompanyStats> statsByCompany;
private DateTime lastTick = DateTime.MinValue;
public StatisticsService(IDatabase database)
{
this.database = database;
this.statsByCompany = new Dictionary<string, CompanyStats>();
}
private class CompanyStats
{
public CompanyStats(List<UserStats> userStats)
{
UserStats = userStats;
}
public List<UserStats> UserStats { get; set; }
}
private class UserStats
{
public UserStats(string userId, int requests, DateTime hour)
{
UserId = userId;
Requests = requests;
Hour = hour;
Updated = DateTime.MinValue;
}
public string UserId { get; set; }
public int Requests { get; set; }
public DateTime Hour { get; set; }
public DateTime Updated { get; set; }
}
}
Every time someone calls the API, I'm calling an increment method on the StatisticsService:
public void Increment(string companyId, string userId)
{
var utcNow = DateTime.UtcNow;
EnsureCompanyLoaded(companyId, utcNow);
var currentHour = new DateTime(utcNow.Year, utcNow.Month, utcNow.Day, utcNow.Hour, 0, 0);
var stats = statsByCompany[companyId];
var userStats = stats.UserStats.FirstOrDefault(ls => ls.UserId == userId && ls.Hour == currentHour);
if (userStats == null)
{
var userStatsToAdd = new UserStats(userId, 1, currentHour);
userStatsToAdd.Updated = utcNow;
stats.UserStats.Add(userStatsToAdd);
}
else
{
userStats.Requests++;
userStats.Updated = utcNow;
}
}
The method loads the company into the cache if not already there (will publish EnsureCompanyLoaded in a bit). It then checks if there is a UserStats object for this hour for the user and company. If not it creates it and set Requests to 1. If other requests have already been made for this user, company, and current hour, it increments the number of requests by 1.
EnsureCompanyLoaded as promised:
private void EnsureCompanyLoaded(string companyId, DateTime utcNow)
{
if (statsByCompany.ContainsKey(companyId)) return;
var currentHour = new DateTime(utcNow.Year, utcNow.Month, utcNow.Day, utcNow.Hour, 0, 0); ;
var userStats = new List<UserStats>();
userStats.AddRange(database.GetAllFromThisMonth(companyId));
statsByCompany[companyId] = new CompanyStats(userStats);
}
The details behind loading the data from the database are hidden away behind the GetAllFromThisMonth method and not important to my question.
Finally, I have a timer that stores any updated results to the database every 5 minutes or when the process shuts down:
public void Tick(object state)
{
var utcNow = DateTime.UtcNow;
var currentHour = new DateTime(utcNow.Year, utcNow.Month, utcNow.Day, utcNow.Hour, 0, 0);
foreach (var companyId in statsByCompany.Keys)
{
var usersToUpdate = statsByCompany[companyId].UserStats.Where(ls => ls.Updated > lastTick);
foreach (var userStats in usersToUpdate)
{
database.Save(GenerateSomeEntity(userStats.Requests));
userStats.Updated = DateTime.MinValue;
}
}
// If we moved into new month since last tick, clear entire cache
if (lastTick.Month != utcNow.Month)
{
statsByCompany.Clear();
}
lastTick = utcNow;
}
I've done some single-threaded testing of the code and the concept seem to work out as expected. Now I want to migrate this to be thread-safe but cannot seem to figure out how to implement it the best way. I've looked at ConcurrentDictionary which might be needed. The main problem isn't on the dictionary methods, though. If two threads call Increment simultaneously, they could both end up in the EnsureCompanyLoaded method. I know of the concepts of lock in C#, but I'm afraid to just lock on every invocation and slow down performance that way.
Anyone needed something similar and have some good pointers in which direction I could go?
When keeping counters in memory like this you have two options:
Keep in memory the actual historic value of the counter
Keep in memory only the differential increment of the counter
I have used both approaches and I've found the second to be simpler, faster and safer. So my suggestion is to stop loading UserStats from the database, and just increment the in-memory counter starting from 0. Then every 5 minutes call a stored procedure that inserts or updates the related database record accordingly (while zero-ing the in-memory value). This way you'll eliminate the race conditions at the loading phase, and you'll ensure that every call to Increment will be consistently fast.
For thread-safety you can use either a normal Dictionary
with a lock, or a ConcurrentDictionary without lock. The first option is more flexible, and the second more efficient. If you choose Dictionary+lock, use the lock only for protecting the internal state of the Dictionary. Don't lock while updating the database. Before updating each counter take the current value from the dictionary and remove the entry in an atomic operation, and then issue the database command while other threads will be able to recreate the entry again if needed. The ConcurrentDictionary class contains a TryRemove method that can be used to achieve this goal without locking:
public bool TryRemove (TKey key, out TValue value);
It also contains a ToArray method that returns a snapshot of the entries in the dictionary. At first glance it seems that the ConcurrentDictionary suits your needs, so you could use it as a basis of your implementation and see how it goes.
To avoid having to make a database call on every request, I've
developed a service class in C# maintaining an in-memory
representation of the statistics stored in a database:
If you want to avoid Update race conditions, you should stop doing exactly that.
Databases by design, by purpose prevent simple update race conditions. This is a simple counting-up operation. A single DML statement. Implicity protected by transactions, journaling and locks. Indeed that is why calling them a lot is costly.
You are fighting the concurrency already there, by adding that service. You are also moving a DB job outside of the DB. And Moving DB jobs outside of the DB, is just going to cause issues.
If your worry is speed:
Please read the Speed Rant.
Maybe a Dsitributed Database Design is the droid you are looking for? They had a massive surge in popularity since Mobile Devices have proliferated, both for speed and reliability reasons.
In general, to make your code thread-safe:
Use concurrent collections, such as ConcurrentDictionary
Make sure to understand concepts such as lock statement, Monitor.Wait and Mintor.PulseAll in tutorials. Locks can be slow if IO operations (such as disk write/read) it being locked on, but for something in RAM it is not necessary to worrry about. If you have really some lengthy operation such as IO or http requests, consider using ConcurrentQueue and learn about the consumer-producer pattern to process work in queues by many workers (example)
You can also try Redis server to cache database without need to design something from zero.
You can also make your service singleton, and update database only after value changes. For reading value, you have already stored it in your service.
I wrote a library, referenced by numerous applications, that tracks who is online and which application and page they are viewing.
The data is stored, using EF6, in a Sql Server 2008 table which tracks their username (primary key), application, page and timestamp. I only want to store the latest request for each person so each username should only be stored once.
The library code, which is called from the Global.asax of each application looks like this:
public static void Add(ApplicationType application, string username, string pageRequested)
{
using (var db = new CommonDAL()) // EF context
{
var exists = db.ActiveUsers.Find(username);
if (exists != null)
db.ActiveUsers.Remove(exists);
var activeUser = new ActiveUser() { ApplicationID = application.Value(), Username = username, PageRequested = pageRequested, TimeRequested = DateTime.Now };
db.ActiveUsers.Add(activeUser);
db.SaveChanges();
}
}
I'm intermittently getting the error Violation of PRIMARY KEY constraint 'PK_tblActiveUser_Username'. Cannot insert duplicate key in object 'dbo.tblActiveUser'. The duplicate key value is (xxxxxxxx)
What I can only guess is happening is Request A comes in, removes the existing username. Request B (from same user) then comes in, tries to remove the username, sees nothing exists. Request A then adds the username. Request B then tries to add the username. The error frequently seems to be triggered when a web server sends a client a 401 status, which again points to multiple requests within a short period of time triggering this.
I'm having trouble mocking this race condition using unit tests as I haven't done much async programming before, but tried to create async tests with delays to mock multiple simultaneous slow requests. I've tried to use using (var transaction = new TransactionScope()) and using (var transaction = db.Database.BeginTransaction(System.Data.IsolationLevel.ReadCommitted)) to lock the requests so request A can complete before request B begins but can't verify either one fixes the issue as I can't mock the situation reliably.
1) Which is the right way to prevent the exception (Most recent request is the one that ultimately is stored)?
2) Which is the right way to to write a unit test to prove this is working?
Since you only want to store the latest item, you could use a last update wins and avoid the race condition on who can insert first, the database handles the locks and the last to call update (which is the most recent) is what is in the table.
Something like the following should handle any primary key errors if you run into concurrency issues on the edge case that a brand new user has 2 requests at the same time and avoid an "infinite" loop of errors (well until a stack overflow exception any way).
public static void Add(ApplicationType application,
string username,
string pageRequested,
int recursionCount = 0)
{
using (var db = new CommonDAL()) // EF context
{
var exists = db.ActiveUsers.Find(username);
if (exists != null)
{
exists.propa = "someVal";
}
else
{
var activeUser = new ActiveUser
{
ApplicationID = application.Value(),
Username = username,
PageRequested = pageRequested,
TimeRequested = DateTime.Now
};
db.ActiveUsers.Add(activeUser);
}
try
{
db.SaveChanges();
}
catch(<Primary Key Violation>)
{
if(recursionCount < x)
{
Add(application, username, pageRequested, recursionCount++)
}
else
{
throw;
}
}
}
}
As for unit testing this, it will be very hard unless you insert an artificial delay or can force both threads to run at the same time. Sometimes the timing on the race conditions is in the millisecond range depending on the issue. Tasks may not work because they are not guaranteed to run at the same time, you throw them to the background thread pool and they run when they can. Old school threads may work but I don't know how to force it since the time between read and remove & create are most likely in the 5 ms range or less.
Assume I have an account_profile table, which has Score field that is similar to an account's money (the database type is BIGINT(20) and the EntityFramework type is long, because I don't need decimal). Now I have the following function:
public long ChangeScoreAmount(int userID, long amount)
{
var profile = this.Entities.account_profile.First(q => q.AccountID == userID);
profile.Score += amount;
this.Entities.SaveChanges();
return profile.Score;
}
However, I afraid that when ChangeScoreAmount are called multiple times concurrently, the final amount won't be correct.
Here are my current solutions I am thinking of:
Adding a lock with a static locking variable in the ChangeScoreAmount function, since the class itself may be instantiated multiple times when needed. It looks like this:
public long ChangeScoreAmount(int userID, long amount)
{
lock (ProfileBusiness.scoreLock)
{
var profile = this.Entities.account_profile.First(q => q.AccountID == userID);
profile.Score += amount;
this.Entities.SaveChanges();
return profile.Score;
}
}
The problem is, I have never tried a lock on static variable, so I don't know if it is really safe and if any deadlock would occur. Moreover, it may be bad if somewhere else outside this function, a change to Score field is applied midway.
OK this is no longer an option, because my server application will be run on multiple sites, that means the locking variable cannot be used
Creating a Stored Procedure in the database and call that Stored procedure in the function. However, I don't know if there is an "atomic" way to create that Store Procedure, so that it can only be called once at a time, since I still need to retrieve the value, changing it then update it again?
I am using MySQL Community 5.6.24 and MySQL .NET Connector 6.9.6 in case it matters.
NOTE My server application may be runned on multiple server machines.
You can use sql transactions with repeatable read isolation level instead of locking on the application. For example you can write
public long ChangeScoreAmount(int userID, long amount)
{
using(var ts = new TransactionScope(TransactionScopeOption.RequiresNew,
new TransactionOptions { IsolationLevel = IsolationLevel.RepeatableRead })
{
var profile = this.Entities.account_profile.First(q => q.AccountID == userID);
profile.Score += amount;
this.Entities.SaveChanges();
ts.Complete();
return profile.Score;
}
}
Transaction garantees that accountprofile record will not changed in db while you aren't commit or rollback.
I have to query in my company's CRM Solution(Oracle's Right Now) for our 600k users, and update them there if they exist or create them in case they don't. To know if the user already exists in Right Now, I consume a third party WS. And with 600k users this can be a real pain due to the time it takes each time to get a response(around 1 second). So I managed to change my code to use Parallel.ForEach, querying each record in just 0,35 seconds, and adding it to a List<User> of records to be created or to be updated (Right Now is kinda dumb so I need to separate them in 2 lists and call 2 distinct WS methods).
My code used to run perfectly before multithread, but took too long. The problem is that I can't make a batch too large or I get a timeout when I try to update or create via Web Service. So I'm sending them around 500 records at once, and when it runs the critical code part, it executes many times.
Parallel.ForEach(boDS.USERS.AsEnumerable(), new ParallelOptions { MaxDegreeOfParallelism = -1 }, row =>
{
...
user = null;
user = QueryUserById(row["USER_ID"].Trim());
if (user == null)
{
isUpdate = false;
gObject.ID = new ID();
}
else
{
isUpdate = true;
gObject.ID = user.ID;
}
... fill user attributes as generic fields ...
gObject.GenericFields = listGenericFields.ToArray();
if (isUpdate)
listUserUpdate.Add(gObject);
else
listUserCreate.Add(gObject);
if (i == batchSize - 1 || i == (boDS.USERS.Rows.Count - 1))
{
UpdateProcessingOptions upo = new UpdateProcessingOptions();
CreateProcessingOptions cpo = new CreateProcessingOptions();
upo.SuppressExternalEvents = false;
upo.SuppressRules = false;
cpo.SuppressExternalEvents = false;
cpo.SuppressRules = false;
RNObject[] results = null;
// <Critical_code>
if (listUserCreate.Count > 0)
{
results = _service.Create(_clientInfoHeader, listUserCreate.ToArray(), cpo);
}
if (listUserUpdate.Count > 0)
{
_service.Update(_clientInfoHeader, listUserUpdate.ToArray(), upo);
}
// </Critical_code>
listUserUpdate = new List<RNObject>();
listUserCreate = new List<RNObject>();
}
i++;
});
I thought about using lock or mutex, but it isn't gonna help me, since they will just wait to execute afterwards. I need some solution to execute only ONCE in only ONE thread that part of code. Is it possible? Can anyone share some light?
Thanks and kind regards,
Leandro
As you stated in the comments you're declaring the variables outside of the loop body. That's where your race conditions originate from.
Let's take variable listUserUpdate for example. It's accessed randomly by parallel executing threads. While one thread is still adding to it, e.g. in listUserUpdate.Add(gObject); another thread could already be resetting the lists in listUserUpdate = new List<RNObject>(); or enumerating it in listUserUpdate.ToArray().
You really need to refactor that code to
make each loop run as independent from each other as you can by moving variables inside the loop body and
access data in a synchronizing way using locks and/or concurrent collections
You can use the Double-checked locking pattern. This is usually used for singletons, but you're not making a singleton here so generic singletons like Lazy<T> do not apply.
It works like this:
Separate out your shared data into some sort of class:
class QuerySharedData {
// All the write-once-read-many fields that need to be shared between threads
public QuerySharedData() {
// Compute all the write-once-read-many fields. Or use a static Create method if that's handy.
}
}
In your outer class add the following:
object padlock;
volatile QuerySharedData data
In your thread's callback delegate, do this:
if (data == null)
{
lock (padlock)
{
if (data == null)
{
data = new QuerySharedData(); // this does all the work to initialize the shared fields
}
}
}
var localData = data
Then use the shared query data from localData By grouping the shared query data into a subordinate class you avoid the necessity of making its individual fields volatile.
More about volatile here: Part 4: Advanced Threading.
Update my assumption here is that all the classes and fields held by QuerySharedData are read-only once initialized. If this is not true, for instance if you initialize a list once but add to it in many threads, this pattern will not work for you. You will have to consider using things like Thread-Safe Collections.