We Keep Coding

Prevent synchronous API usage with EF Core

How can I prevent synchronous database access with Entity Framework Core? e.g. how can I make sure we are calling ToListAsync() instead of ToList()?
I've been trying to get an exception to throw when unit testing a method which calls the synchronous API. Are there configuration options or some methods we could override to make this work?
I have tried using a DbCommandInterceptor, but none of the interceptor methods are called when testing with an in-memory database.

The solution is to use a command interceptor.
public class AsyncOnlyInterceptor : DbCommandInterceptor
{
public bool AllowSynchronous { get; set; } = false;
public override InterceptionResult<int> NonQueryExecuting(DbCommand command, CommandEventData eventData, InterceptionResult<int> result)
{
ThrowIfNotAllowed();
return result;
}
public override InterceptionResult<DbDataReader> ReaderExecuting(DbCommand command, CommandEventData eventData, InterceptionResult<DbDataReader> result)
{
ThrowIfNotAllowed();
return result;
}
public override InterceptionResult<object> ScalarExecuting(DbCommand command, CommandEventData eventData, InterceptionResult<object> result)
{
ThrowIfNotAllowed();
return result;
}
private void ThrowIfNotAllowed()
{
if (!AllowSynchronous)
{
throw new NotAsyncException("Synchronous database access is not allowed. Use the asynchronous EF Core API instead.");
}
}
}
If you're wanting to write some tests for this, you can use a Sqlite in-memory database. The Database.EnsureCreatedAsync() method does use synchronous database access, so you will need an option to enable this for specific cases.
public partial class MyDbContext : DbContext
{
private readonly AsyncOnlyInterceptor _asyncOnlyInterceptor;
public MyDbContext(IOptionsBuilder optionsBuilder)
: base(optionsBuilder.BuildOptions())
{
_asyncOnlyInterceptor = new AsyncOnlyInterceptor();
}
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.AddInterceptors(_asyncOnlyInterceptor);
base.OnConfiguring(optionsBuilder);
}
public bool AllowSynchronous
{
get => _asyncOnlyInterceptor.AllowSynchronous;
set => _asyncOnlyInterceptor.AllowSynchronous = value;
}
}
Here are some helpers for testing. Ensure you aren't using sequences (modelBuilder.HasSequence) because this is not supported by Sqlite.
public class InMemoryOptionsBuilder<TContext> : IOptionsBuilder
where TContext : DbContext
{
public DbContextOptions BuildOptions()
{
var optionsBuilder = new DbContextOptionsBuilder<TContext>();
var connection = new SqliteConnection("Filename=:memory:");
connection.Open();
optionsBuilder = optionsBuilder.UseSqlite(connection);
return optionsBuilder.Options;
}
}
public class Helpers
{
public static async Task<MyDbContext> BuildTestDbContextAsync()
{
var optionBuilder = new InMemoryOptionsBuilder<MyDbContext>();
var context = new MyDbContext(optionBuilder)
{
AllowSynchronous = true
};
await context.Database.EnsureCreatedAsync();
context.AllowSynchronous = false;
return context;
}
}

How can I prevent synchronous database access with Entity Framework Core?
You can not. Period. THere is also no reason for this ever. You basically assume programmers using your API either are idiots or malicious - why else would you try to stop them from doing something that is legal in their language?
I have tried using a DbCommandInterceptor, but none of the interceptor methods are
called when testing with an in-memory database
There are a TON of problems with the in memory database. I would generally suggest not to use it - like at all. Unless you prefer a "works possibly" and "never actually use advanced features of the database at all". It is a dead end - we never do unit testing on API like this, all our unit tests actually are integration tests and test end to end (vs a real database).
In memory has serious no guarantee to work in anything non trivial at all. Details may be wrong - and you end up writing fake tests and looking for issues when the issue is that the behavior of the in memory database just is a little different than the real database. And let's not get into what you can do with the real database that in memory has no clue how to do to start with (and migrations also do not cover). Partial and filtered indices, indexed views are tremendous performance tools that can not be properly shown. And not get into detail differences for things like string comparisons.
But the general conclusion is that it is not your job to stop users from calling valid methods on EfCore etc. and you are not lucky to actually do that - not a scenario the team will ever support. There are REALLY good reasons at time to use synchronous calls - in SOME scenarios it seems the async handling is breaking down. I have some interceptors (in the http stack) where async calls just do not work. Like never return. Nothing I ever tried worked there - so I do sync calls when I have to (thank heaven I have a ton of caching in there).

You can prevent it at compile-time to some degree by using the Microsoft.CodeAnalysis.BannedApiAnalyzers NuGet package. More information about it here.
Methods that end up doing synchronous queries can then be added to BannedSymbols.txt, and you will get a compiler warning when attempting to use them. For example adding the following line to BannedSymbols.txt gives a warning when using First() on an IQueryable<T>:
M:System.Linq.Queryable.First`1(System.Linq.IQueryable{``0});Use async overload
These warnings can also be escalated to become compiler errors by treating warnings as errors as explained here:
https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/compiler-options/errors-warnings
Unfortunately not all synchronous methods can be covered by this approach. For example since ToList() is an extension on IEnumerable<T> (and not on IQueryable<T>), banning it will not allow any use of ToList() in the same project.

I can't really find a good Google answer for you. So my suggestion in the meantime is that you start doing peer-review, aka Code Reviews and any time you find a .Tolist(), you change it to await .ToListAsync().
It's not the most high tech solution, but it does keep everyone honest, but it also allows others to become familiar with your work should they ever need to maintain it while you're booked off sick.

EF core DbContext in a multithreaded API application

tl;dr How can I use Entity Framework in a multithreaded .NET Core API application even though DbContext is not threadsafe?
Context
I am working on a .NET Core API app exposing several RESTful interfaces that access the database and read data from it, while at the same time running several TimedHostedServices as background working threads that poll data regularly from other webservices and store them into the database.
I am aware of the fact that DbContext is not threadsafe. I read a lot of docs, blog Posts and answers here on Stackoverflow, and I could find a lot of (partly contradictory) answers for this but no real "best practice" when also working with DI.
Things I tried
Using the default ServiceLifetime.Scoped via the AddDbContext extension method results in exceptions due to race conditions.
I don't want to work with locks (e.g. Semaphore), as the obvious downsides are:
the code is polluted with locks and try/catch/finally for safely releasing the locks
it doesn't really seem 'robust', i.e. when I forget to lock a region that accesses the DbContext.
it seems redundant and 'unnatural' to artificially syncronize db access in the app when working with a database that also handles concurrent connections and access
Not injecting MyDbContext but DbContextOptions<MyDbContext> instead, building the context only when I need to access the db, using a using statement to immediatelly dispose it after the read/write seems like a lot of resource usage overhead and unnecessarily many connection opening/closings.
Question
I am really puzzled: how can this be achived?
I don't think my usecase is super special - populating the db from a Background worker and querying it from the web API layer - so there should be a meaningful way of doing this with ef core.
Thanks a lot!

You should create a scope whenever your TimedHostedServices triggers.
Inject the service provider in your constructor:
public MyServiceService(IServiceProvider services)
{
_services = services;
}
and then create a scope whenever the task triggers
using (var scope = _services.CreateScope())
{
var anotherService = scope.ServiceProvider.GetRequiredService<AnotherService>();
anotherService.Something();
}
A more complete example is available in the doc

Another approach to create own DbContextFactory and instantiate new instance for every query.
public class DbContextFactory
{
public YourDbContext Create()
{
var options = new DbContextOptionsBuilder<YourDbContext>()
.UseSqlServer(_connectionString)
.Options;
return new YourDbContext(options);
}
}
Usage
public class Service
{
private readonly DbContextFactory _dbContextFactory;
public Service(DbContextFactory dbContextFactory)
=> _dbContextFactory = dbContextFactory;
public void Execute()
{
using (var context = _dbContextFactory.Create())
{
// use context
}
}
}
With factory you don't need to worry about scopes anymore, and make your code free of ASP.NET Core dependencies.
You will be able to execute queries asynchronously, which not possible with scoped DbContext without workarounds.
You always be confident about what data saved when calling .SaveChanges(), where with scoped DbContext there are possibilities that some entity were changed in other class.

Azure mobile service: Define a Table only locally

I'm using offline sync in a Xamarin app. I have the following scenario:
I have a couple of tables which sync fine and one table called LocalOnlyTable which I don't want to sync. I just want to read/write it locally.
The problem appears when I pull one of my tables like so:
await exerciseTable.PullAsync(string.Format("{0}ItemByFK", typeof(Exercise).Name), exerciseTable.CreateQuery());
I get a MobileServicePushFailedException saying 404 LocalOnlyTable does not exist.
I'm wondering why Mobile Services tries to push/pull the LocalOnlyTable and
How can I prevent Mobile Services from trying to sync LocalOnlyTable?

Just came across your issue here and thought of sharing my solution.
1) Create a custom TableSyncHandler to block off local-only tables:
public class TableSyncHandler : IMobileServiceSyncHandler
{
private readonly IMobileServiceClient _client;
private readonly HashSet<string> _excludedTables = new HashSet<string>();
public TableSyncHandler(IMobileServiceClient client)
{
_client = client;
}
public void Exclude<T>()
{
_excludedTables.Add(_client.SerializerSettings.ContractResolver.ResolveTableName(typeof(T)));
}
public Task OnPushCompleteAsync(MobileServicePushCompletionResult result)
{
return Task.FromResult(0);
}
public Task<JObject> ExecuteTableOperationAsync(IMobileServiceTableOperation operation)
{
if (_excludedTables.Contains(operation.Table.TableName))
{
return Task.FromResult((JObject) null);
}
return operation.ExecuteAsync();
}
}
2) When you are initializing MobileServiceClient's SyncContext, register the tables you want to exclude to this syncHandler, and then initialize SyncContext using the syncHandler:
_store = new MobileServiceSQLiteStore("YourStore");
_store.DefineTable<User>();
_store.DefineTable<LocalOnlyTable>();
_syncHandler = new TableSyncHandler(client);
// LocalOnlyTable is excluded from sync operations
_syncHandler.Exclude<LocalOnlyTable>();
await client.SyncContext.InitializeAsync(_store, _syncHandler);
Disclaimer:
This has not gone to production yet, so I don't know if there will be performance impact, but seems to be working fine so far in testing.
This solution is based on Azure Mobile Services client v1.3.2 source code. It's not doing anything (pull/push) when the synchandler returns null result. This behaviour can possibly change in the future.

All actions take using the MSSyncTable APIs are tracked to be sent to the server. If you have a table you do not want to track you shouldn't use the MSSyncTable APIs to insert/update records.
You should be able to use either the SQLiteStore methods (like upsert) or execute SQL on your SQLite Db directly for your untracked tables.

What is the best method for making database connection (static, abstract, per request, ...)?

I used lot of model for connecting to db, in my last project that i worked with C# & entity framework, i created static class for db connecting but i had problem with opening and closing connection for that give me error when more than 10-15 requests come together, i solved it with changing method of connecting to db with i connect now per request and removed all static methods and classes.
Now i want to know,
What is best model for making connection?
Should i close it after every query and open it before using or ...?
A connection in static class is good model (that i don`t need to
create it, every time)?
Is there a good design pattern for this problem?
All of it is for the same question What is the best method for
making database connection (static, abstract, per request, ...)?
For example i working on a sms sender web panel, I should send 100K sms per second, these sms collect with others and make a package that every package have 1~20 sms then i need to send 5K~100K packages per one second and when i send a package i should do these steps:
Update single sms to delivered or not delivered
Update user balance if delivered decrease user balance in useraccounts table
Update number of sms send count in user table
Update number of sms send count in mobile number table
Update number of sms send count in sender number table
Update package for delivered and failed sms in package table
Update package for how thread send this package in package table
Update thread table for how many sms send it by this tread and how many failed
Add account document for this transactions in AccountDocument table
All steps and lot of other things like logs, user interface and monitoring widgets, that should doing and i need DB connection for doing every single of this transactions.
Now, What is best model for connecting to DB? By human request or by thread request or by every single transaction..

answers to your questions:
Close it. .NET does connection pooling for you under the hood.
Create it. use the using (Connection conn = new ....) each time - this way, you'll make the most out of the .NET pooling mechanism.
you can use the .NET ThreadPool (or your own custom one), define the ThreadPool to use solely 10 thread in parallel and Enqueue work items one after another. this way no more then 10 connections will be used in the same time + it'll probably work faster.
More about Custom ThreadPools: Custom ThreadPool Implementation
Per instance.
Here's my suggestion for an architecture:
Create a database table (queue) for pending SMS to be sent out.
each row will contain all the information needed for the sms + the current status.
create a worker process, perhaps a windows service which will sample this table constantly - let's say, each 5 seconds. it will select the TOP ~20 SMS with status = 'pending to be sent' (should be represented as int). and will update the status to 'sending'
each sms will be sent out using a custom threadpool on the windows service side.
in the end of the process, ALL the processed sms status will be updated to 'done' using a CTE (common table expression - you can send a cte with all the sms rows ids that have just been process to do a 'bulk update' to 'done' status).
you could make the status update stored procedure to be the same one as the 'getpending'. this way, you could select-for-update with no lock and make the database work faster.
this way, you can have more than just one processor service running (but then you'll have to loose the nolock).
remember to avoid as much locking as possible.
by the way, this is also good because you could send SMS from any place in your system by simply adding a row to the pending SMS table.
And one more thing, i would not recommend to use entity framework for this, as it has too much going on under the hood. All you need for this kind of task is to simply call 3-4 stored procedures, and that's it. Maybe take a look at Dapper-dot-NET - its a very lightweight MicroDal framework which in most cases works more than 10 times faster than EF (Entity Framework)

1. Should i close it after every query?
.Net does that for you so let it handle it, that's a garbage collector task. So don't bother disposing your objects manually, this is a good answer by Jon Skeet: https://stackoverflow.com/a/1998600/544283. However you could use the using(IDisposable){ } statement to force the GC to do it's work. Here is a nice article about resources reallocation: http://www.codeproject.com/Articles/29534/IDisposable-What-Your-Mother-Never-Told-You-About.
2. A connection in static class is good?
Never make a data context static! Data contexts are not thread safe or concurrent safe.
3. Is there a good design pattern for this problem?
As Belogix mentioned dependency injection and unit of work patterns are great, in fact entity framework is a unit of work itself. DI and UoW are a bit overrated though, it's not easy to implement if it's your first time handling an IoC container which if you're going that path I'd recommend Ninject. One other thing is you don't really need DI if you're not gonna run tests, the awesomeness of these patterns is to decouple, so you can test and mock without sweat.
In-short: If you're gonna run test against your code go for these patterns. If not, I'm providing you an example about how you could share your data context among the services you'd like. This is the answer to your fourth question.
4. What is the best method for making database connection (static, per request)?
Your context service:
public class FooContextService {
private readonly FooContext _ctx;
public FooContext Context { get { return _ctx; } }
public FooContextService() {
_ctx = new FooContext();
}
}
Other services:
public class UnicornService {
private readonly FooContext _ctx;
public UnicornService(FooContextService contextService) {
if (contextService == null)
throw new ArgumentNullException("contextService");
_ctx = contextService.Context;
}
public ICollection<Unicorn> GetList() {
return _ctx.Unicorns.ToList();
}
}
public class DragonService {
private readonly FooContext _ctx;
public DragonService(FooContextService contextService) {
if (contextService == null)
throw new ArgumentNullException("contextService");
_ctx = contextService.Context;
}
public ICollection<Dragon> GetList() {
return _ctx.Dragons.ToList();
}
}
Controller:
public class FantasyController : Controller {
private readonly FooContextService _contextService = new FooContextService();
private readonly UnicornService _unicornService;
private readonly DragonService _dragonService;
public FantasyController() {
_unicornService = new UnicornService(_contextService);
_dragonService = new DragonService(_contextService);
}
// Controller actions
}
Second thoughts (almost an edit):
If you need your context not to create the proxies for your entities therefore not having lazy loading either, you could overload your context service as follows:
public class FooContextService {
private readonly FooContext _ctx;
public FooContext Context { get { return _ctx; } }
public FooContextService() : this(true) { }
public FooContextService(bool proxyCreationEnabled) {
_ctx = new FooContext();
_ctx.Configuration.ProxyCreationEnabled = proxyCreationEnabled;
}
}
NOTE:
If you set the proxy creation enabled to false you will not have lazy loading out of the box.
If you have api controllers you don't want to deal with any full blown object graph.
EDIT:
Some reading first:
This link relates to a pre-release version of EF6: Entity Framework and Async.
Scott Allen posted about this in his blog: Async in Entity Framework 6.0.
If you're going to use Unit of Work I'd recommend to read this: Make the DbContext Ambient with UnitOfWorkScope.
Darin Dimitrov's answer on Do asynchronous operations in ASP.NET MVC use a thread from ThreadPool on .NET 4.
Get this done:
(_context as IObjectContextAdapter).ObjectContext.Connection.Open();
This is a great article about Managing Connections and Transactions.
Entity framework exposes EntityConnection through the Connection property. Read as: public sealed class EntityConnection : DbConnection.
Considerations for managing connections: (taken from previous link)
The object context will open the connection if it is not already open before an operation. If the object context opens the connection during an operation, it will always close the connection when the operation is complete.
If you manually open the connection, the object context will not close it. Calling Close or Dispose will close the connection.
If the object context creates the connection, the connection will always be disposed when the context is disposed.
In a long-running object context, you must ensure that the context is disposed when it is no longer required.
Hope it helps.

I think per request scales the best. Use a thread-safe connection pool and make the connection scope coincide with the unit of work. Let the service that's responsible for transactional behavior and units of work check out the connection, use it, and return it to the pool when the unit of work is either committed or rolled back.
UPDATE:
10-12 seconds to commit a status update? You've done something else wrong. Your question as written is not sufficient to provide a suitable answer.
Daily NASDAQ volume is 1.3B transactions, which on an 8 hour day works out to ~45K transactions per second. Your volume is 2X that of NASDAQ. If you're trying to do it with one machine, I'd say that NASDAQ is using more than one server.
I'd also wonder if you could do without that status being updated using ACID. After all, Starbucks doesn't use two-phase commit. Maybe a better solution would be to use a producer/consumer pattern with a blocking queue to update those statuses when you can after they're sent.

Making access to db static = bad?

In a couple of .NET C# webservice projects that i have done i have made access to db static with help of the singleton pattern. Then the other day my friend told me that this is a bad thing to do, because if a lot of request is made for the same db entity then the db would be locked because of the static instance. Is my friends assumptations right? I thought that every new request would make a new instance of the class?
The implementation of the singleton class looks like this:
public class WebService
{
private readonly IFactory _factory;
public WebService(IFactory factory)
{
_factory = factory;
}
public IDataRepository Data
{
get
{
return _factory.GetDatabase();
}
}
}
public static class WebServiceImpl
{
private static readonly WebService _webService = new WebShop(new WebserviceFactoryImpl());
public static WebService webService { get { return _webService; } }
}
_factory.GetDatabase() returns a new instace of the Database class.

Looking at WebServiceImpl, all calls will be sharing a single WebService instance. Now, this isn't necessarily a problem, depending on how that is implemented; for example, if _factory.GetDatabase(); ends up getting called per-request, then it might be that you are getting away with it. Depending further on what GetDatabase() does - i.e. does it get a new instance per call? or does it give you the same instance every time? Simply: we don't have enough information there to answer fully. But:
sharing a single database connection between requests is dangerous; either you need to lock / synchronize, or you risk lots of errors (database connections are not usually written to be thread-safe)
sharing an ORM between requests is even worse: in addition to everything above, you also get issues with data accumulating in the identity / object cache; ORM instances (data-context, etc) are intended to be short-lived and then discarded (and sometimes: disposed)
Having static access to the database is not necessarily a problem; it all comes down to how that is implemented - for example, a static-based API could still create (and dispose) a connection on every call.