We Keep Coding

Simple web application with ASP.NET MVC + EF Core sometimes creates several identical entries in the database

I have a web application where users can post offers for products.
Sometimes - less than 1 in 10 cases - saving an offer leads to multiple entries in the database with identical data (usually 2 entries, but 3 or even 4 identical entries already occurred).
The offer class is a simple entity class with some properties in it. It is also connected to two further entity classes OfferImage and OfferCategory, which store the associated images and the categories in which the offer should appear.
The code to save an item to the database is the following (part of a repository class):
public class OfferRepository {
...
public async Task InsertAsync(Offer offer)
{
Context.Offer.Add(offer);
await Context.SaveChangesAsync();
}
...
}
It is called within a service class:
public class OfferService
{
...
public async Task CheckinAsync(Offer offer)
{
await repository.InsertAsync(offer);
}
...
}
That method of this service class is called by an mvc controller:
public async Task<IActionResult> Create(CreateOfferViewModel createOfferViewModel)
{
if (ModelState.IsValid)
{
...
//conversion of the view model object to an Offer object
...
await offerService.CheckinAsync(offer);
}
...
}
As you can see, the structure is relatively simple. However, this error occurs regularly.
The lifecycle of the service classes is managed with dependency injection in startup.cs. OfferRepository and OfferService are both added scoped (services.AddScoped) The context class is added like this:
services.AddDbContext<Context>(options =>
{
options.UseSqlServer(
Configuration.GetConnectionString("DataConnection"),
sqlServerOptionsAction: sqlServerOptions =>
{
sqlServerOptions.EnableRetryOnFailure(maxRetryCount: 3, maxRetryDelay: TimeSpan.FromSeconds(3), errorNumbersToAdd: null);
}
);
});
To further narrow down the problem, I ran a profiler and got a recording of the INSERT statements (in chronological order):
SPID 59 - 2019-09-24 16:05:19.670 - INSERT INTO Offer ...
SPID 57 - 2019-09-24 16:05:19.673 - INSERT INTO Offer ...
SPID 59 - 2019-09-24 16:05:19.710 - INSERT INTO OfferImage ... / INSERT INTO OfferCategory ...
SPID 57 - 2019-09-24 16:05:19.760 - INSERT INTO OfferImage ... / INSERT INTO OfferCategory ...
What makes me suspicious is that there are two different process IDs that execute the INSERTs. Since the DbContext is scoped by default, there should only be one process ID under which all statemens are executed - or am I wrong? If I am not mistaken, this would mean that two requests are executed in parallel, which in turn raises further questions about how this can happen.
As you can see, I am a little confused and hope for help from someone who can explain this or has observed and solved something similar.
(SQL Server is version 13/2016, EF Core is version 2.2)
Thank you very much!

Every call to your API can use a different thread id. If PID really is a different process, then you have 2 different instances of your API running at once.
If you really don't want duplicates, then add a constraint to your database to prevent duplicates (product name or something). This means that SaveChanges will throw a DbUpdateException which you will need to catch and determine if it's a duplicate exception (which you can send an error back to the user via a HTTP response code, maybe 409 conflict), or something else (which is probably a 5xx error).

Task.WhenAll hangs out if there are no tasks to complete

I have method that inserts list of objects into Mongo DB.
public class StorageService : IStorageService
{
public Task<BulkWriteResult<Option>> SaveOptions(List<Option> contracts)
{
var context = new MongoContext<Option>();
return context.SaveCollection(contracts);
}
}
var optionIds = Task
.WhenAll(storageService.SaveOptions(optionDetails.Values.ToList()))
.Result;
If list of contracts is empty, then there is no objects to insert into DB, and no tasks to complete, so Task.WhenAll keeps running indefinitely creating a deadlock.
Question
Is there a way to return empty / completed task if list is empty, or maybe there is a better solution of how to get results of insert, but at the same time, correctly handle case when there are no results?
Update #1
Approximate structure.
WebApi - MVC project
[AcceptVerbs("POST")]
public Response<int> DownloadOptions([FromBody] ContractSelector data)
{
.. some controller code
var optionIds = Task
.WhenAll(storageService.SaveOptions(optionDetails.Values.ToList()))
.Result;
// this method should gather data from multiple APIs
// so I need Task.Result of all previous operations
// I could make this method async and use await, but it's not the case here
}
Class Library project, .NET 4.6.1
public class StorageService : IStorageService
{
public Task<BulkWriteResult<Option>> SaveOptions(List<Option> contracts)
{
var context = new MongoContext<Option>();
return context.SaveCollection(contracts);
}
}
Update #2
Why there is no use for async / await. There are 2 external API calls, one is to get general info about some asset, and the second one is to get prices for this asset, I can't change this. So, if I want to get all info in one method I must request general info, then Wait for Result, and, based on general info, request relevant prices. After this, I want to save gathered info into DB and return list of saved IDs in the response to my API.
Sequence of calls
1. UI
2. WebApi MVC Controller
3. Class Library
3.1 Request asset info - wait for the result
3.2 Get asset info - request prices for selected assets - wait for the result
3.3 Get asset info and prices info - save everything to DB - return response
var contracts = Task.WhenAll(optionService.GetContracts(params)).Result;
var prices = Task.WhenAll(optionService.GetOptionDetails(contracts)).Result;
var ids = Task.WhenAll(storageService.SaveOptions(prices.Values.ToList()));
So, response depends on 3.3, 3.3 depends on 3.2, 3.2 depends on 3.1. If you know how to turn it all to a non-blocking call, I'm all ears. For now I think that 3 blocking calls in 1 HTTP request are better than 3 separate async HTTP requests.

Can't explain in details, but looks like issue was caused by incompatibility between .NET framework and Mongo DB driver. Currently I'm using .NET 4.6.1 and Mongo 2.6.1 and it works fine.
Yesterday any try to save an empty list of objects into Mongo DB caused a deadlock.
Then I tried to refactor code and install the latest version of Mongo DB, which is 2.7.0. Now any CRUD operation with Mongo DB returns this error. There is no call stack and more details about this exception.
MethodAccessException: Attempt by method 'MongoDB.Driver.ClientSession..ctor(MongoDB.Driver.IMongoClient, MongoDB.Driver.ClientSessionOptions, MongoDB.Driver.IServerSession, Boolean)' to access method 'MongoDB.Driver.Core.Clusters.ClusterClock..ctor()' failed.
Then I tried to lower version of Mongo DB to 2.6.1 and when I try to insert empty list of objects I'm getting this exception, which is correct. When I insert at least one record it works as expected.
System.AggregateException: 'One or more errors occurred.'
ArgumentException: Must contain at least 1 request.
Parameter name: requests
Testing code
var item = new Demo
{
Symbol = "SomeSymbol",
Expiration = "2019-01-01"
};
var list = new List<Demo>();
list.Add(item);
list.Add(item); // if I comment these lines, then Mongo 2.6.1 returns correct exception
var optionIds = Task.WhenAll(storageService.Save(list)).Result;
Save method is implemented as a part of Mongo repository pattern
public Task<BulkWriteResult<T>> SaveCollection(List<T> items)
{
var records = new List<ReplaceOneModel<T>>();
var processes = new List<Task<ReplaceOneResult>>();
items.ForEach(contract =>
{
var record = new ReplaceOneModel<T>(Builders<T>
.Filter
.Where(o => o.Id == contract.Id), contract)
{
IsUpsert = true
};
records.Add(record);
});
return Collection.BulkWriteAsync(records);
}

How to obtain other Entities in C# WebCore Entity Framework

I have a C# WebCore REST service that talks to a front end and uses Entity Framework to CRUD into a database. This uses Dependency Injection to add the contexts upon startup:
services.AddDbContext<FileCacheContext>(options => options.UseSqlServer(Settings.ConnectionSetting));
services.AddDbContext<FileImportContext>(options => options.UseSqlServer(Settings.ConnectionSetting));
The functionality that I have allows a user to upload a file, which is manipulated by the server and some properties of that file are returned to the front end. The uploaded file is cached in the database (FileCacheContext).
After some time has passed, the user now wishes to confirm their action and "promote" the file from the Cache (FileCacheContext) to the Import (FileImportContext); this is done by an action in the front end that contains the id of the cached file.
This parameter is passed to a different REST Controller, which is being invoked using the FileImport context, rather than the FileCache context:
public PromoteController(FileImportContext context)
{
_context = context;
}
[HttpPost]
public void Post([FromBody] int fileCacheId)
{
...
What I now need to do is to "move" this cached file from one "context" to another, something along the lines of:
var cachedFile = _context.FileCache.Where(f => f.FileCacheId == cachedFileId).FirstOrSingle();
var importedFile = new FileImport() { FileData = cachedFile.FileData };
_context.FileImport.Add(importedFile);
_context.SaveChanges();
My issue is that I can only see the one context, so I cannot get hold of the other context to either read from or write into.
The two underlying tables have no relationship, so I cannot link them in any way to load one based upon the other.
How can I get access to another (unrelated) context in EF?

So to answer my own question, I found that I could of course include the additional contexts required when instantiating the controller:
public PromoteController(FileImportContext contextImport, FileCacheContext contextCache)
{
_contextImport = contextImport;
_contextCache = contextCache;
}
And then use these as required.
This doesn't feel like the right way to do it, but if it works, it works...

EF Core and big traffic leads to max pool size was reached error

We're using ASP.NET Entity Framework Core for querying our MSSQL database in our Web API app. Sometimes when we have big traffic, querying to DB ends with this error:
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
I wonder if our pattern of using DbContext and querying is correct or if I am missing some using/dispose pattern and error is caused by some memory leak (after some research I read then I should not use using because the lifetime is managed by the framework). I am following documentation...
My connectionString:
"myConnection": "Server=xxx;Database=xxx;user id=xxx;password=xxx;Max Pool Size=200;Timeout=200;"
My Startup.cs
public void ConfigureServices(IServiceCollection services)
{
.....
// scoped context
services.AddDbContext<MyDbContext>(
options => options.UseSqlServer(this.Configuration.GetConnectionString("myConnection")));
}
then in controllers I used dbcontext by dependency injection:
public class MyController : Controller
public MyController (MyDbContext context)
{
this.Context = context;
}
public ActionResult Get(int id)
{
// querying
return this.Context.tRealty.Where(x=>x.id == id).FirstOrDefault();
}
Should I use something like:
using (var context = this.Context)
{
return this.Context.tRealty.Where(x => x.id == id).FirstOrDefault();
}
But I think that this is bad pattern when I am using dependency injection of DbContext.

I think problem was caused by storing objects from database context queries to In memory cache. I had one big LINQ query to database context with some other subqueries inside. I called FirstOrDefault() on the end of main query but not inside subqueries. Controller was fine with it, it materialize queries by default.
return this.Context.tRealty.AsNoTracking().Where(
x => x.Id == id && x.RealtyProcess == RealtyProcess.Visible).Select(
s => new
{ .....
// subquery
videos = s.TVideo.Where(video => video.RealtyId == id && video.IsPublicOnYouTube).
Select(video => video.YouTubeId).ToList()), // missing ToList()
.....
}).FirstOrDefault();
And there was problem - subqueries were holding connection to database context when they where storing to In memory cache. When I implemented Redis distributed cache, it was first failing on some strange errors. It helps when I write ToList() or FirstOrDefault() to all my subqueries because distributed cache needs materialized objects.
Now I have all my queries materialized explicitly and I got no max pool size was reached error. So that one must be careful when stored objects from database context queries to In memory cache. It is need to materialize all queries to avoid to holding connection somewhere in memory.

You can set the lifetime of the DbContext in your startup.cs, see if this helps:
services.AddDbContext<MyDbContext>(options => options
.UseSqlServer(connection), ServiceLifetime.Scoped);
Also if your query is a simple read you can remove tracking by using .AsNoTracking().
Another way to improve your throughput is to prevent locks by using a transaction block with IsolationLevel.ReadUncommitted for simple reads.
You can also use the Snapshot isolation level - which is slightly more restrictive - if you do not want dirty reads.
TransactionOptions transactionOptions = new TransactionOptions() { IsolationLevel = IsolationLevel.ReadUncommitted};
using (TransactionScope transactionScope = new TransactionScope(TransactionScopeOption.Required, transactionOptions))
{
// insert magic here
}
Edit : As the author of the question mentioned, the above code is not (yet?) possible in EF Core.
A workaround can be found here using an explicit transaction:
using (var connection = new SqlConnection(connectionString))
{
connection.Open();
using (var transaction = connection.BeginTransaction())
{
// transaction.Commit();
// transaction.Rollback();
}
}
I have not tested this.
Edit 2: Another untested snippet where you can have executed commands to set isolation level:
using (var c1= new SqlConnection(connectionString))
{
c1.Open();
// set isolation level
Exec(c1, "SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;");
Exec(c1, "BEGIN TRANSACTION;");
// do your magic here
}
With Exec:
private static void Exec(SqlConnection c, string s)
{
using (var m = c.CreateCommand())
{
m.CommandText = s;
m.ExecuteNonQuery();
}
}
Edit 3: According to that thread, Transactions will be supported from .NET Core version 1.2 onwards.
#mukundabrt this is tracked by dotnet/corefx#2949. Note that
TransactionScope has already been ported to .NET Core but will only be
available in .NET Core 1.2.

I am adding an alternative answer, in case anyone lands here with a slightly different root cause, as was the case for my .NET Core MVC application.
In my scenario, the application was producing these "timeout expired... max pool size was reached" errors due to mixed use of async/await and Task.Result within the same controller.
I had done this in an attempt to reuse code by calling a certain asynchronous method in my constructor to set a property. Since constructors do not allow asynchronous calls, I was forced to use Task.Result. However, I was using async Task<IActionResult> methods to await database calls within the same controller. We engaged Microsoft Support, and an Engineer helped explain why this happens:
Looks like we are making a blocking call to an Async method inside
[...] constructor.
...
So, basically something is going wrong in the call to above
highlighted async method and because of which all the threads listed
above are blocked.
Looking at the threads which are doing same operation and blocked:
...
85.71% of threads blocked (174 threads)
We should avoid mixing async and blocking code. Mixed async and
blocking code can cause deadlocks, more-complex error handling and
unexpected blocking of context threads.
https://msdn.microsoft.com/en-us/magazine/jj991977.aspx
https://blogs.msdn.microsoft.com/jpsanders/2017/08/28/asp-net-do-not-use-task-result-in-main-context/
Action Plan
Please engage your application team to revisit the application code of above mentioned method to understand what is going
wrong.
Also, I would appreciate if you could update your application logic to
not mix async and blocking code. You could use await Task instead of
Task.Wait or Task.Result.
So in our case, I pulled the Task.Result out of the constructor and moved it into a private async method where we could await it. Then, since I only want it to run the task once per use of the controller, I store the result to that local property, and run the task from within that method only if the property value is null.
In my defense, I expected the compiler would at least throw a warning if mixing async and blocking code is so problematic. However, it seems obvious enough to me, in hindsight!
Hopefully, this helps someone...

WCF error when querying using LINQ to Objects

Please excuse me if this is a duplicate question - I have searched but have not found anything that explains my problem.
I created an ASP.NET website. I have 3 layers - Data Access, Business, Presentation. I used shared entities throughout using Entity Framework.
Presentation: UI
Business: WCF Service
Data Access: code to connect to DB (contains the EF context, connection string)
Entities: .edmx - EF Entities. These are shared throughout all 3 layers.
I have come upon some odd behavior in the WCF service which I cannot understand. My original code had a function in the DAL which queried the db for all customers, and returned a List. The service than further queried this list based on what the UI requested and returned a List to the Presentation Layer.
DAL:
public List<Customer> GetCustomers()
{
List<Customer> custList= new List<Customer>();
try
{
using (NorthWindsEntities context = new NorthWindsEntities(connectionString))
{
custList= context.Customers
.Include("Orders")
.Include("OrderDetails")
.ToList();
}
return custList;
}
catch (Exception ex)
{
//handle exception
return custList;
}
}
WCF:
public List<Customer> GetCustomersByState(string state)
{
contextManager contextDAL = new contextManager();
List<Customer> custList = contextDAL.GetCustomers()
.Where(c => c.State == state)
.ToList();
return custList;
}
When I debugged the code, eveything worked fine, but when the service tried to return the custList to the client, I got this error: An error occurred while receiving the HTTP response to localhost/WCF/MyService. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details.
I changed my DAL to inclued the entire query and my WCF Service to this:
public List<customer> GetCustomersByState(string state)
{
contextManager contextDAL = new contextManager();
List<Customer> custList = contextDAL.GetCustomers(state)
return custList;
}
This worked fine. I also tried to put the entire query in the service (the service connected directly to the context and got the list) and that worked fine as well.
So basically, my question is, why can't the WCF service return a list of objetcs that were queried using Linq to Objects (the DAL returns a List of customer so all further querying is linq to objects). It works fine using Linq to Entities (with all the code in the service) and it works fine with no querying at all - just getting a list and returning it.
I'm sorry for the lengthy post, I tried to include all necessary details...Thanks!

After much trial and error, I have found that it is not a querying issue, it is a size issue. Though the WCF service was filtering my data to only return a few records, I guess it retained some sort of reference to the rest of the data because it errored with too much data. If I do a partial filter on the DAL and then continue to filter on WCF layer, coming out the same amount of records as I originally tried to return, it returns the list without a problem.
I cannot explain why this happens, I don't know much about WCF, just explaining what I did to get around it in case anyone else has this problem.