We Keep Coding

Simultaneous requests slow down asp net core API

TLDR; I have an ASP.NET Core 5.0 API that's sitting at AWS. It makes a large call to MSSQL db to return ~1-4k rows of data. A single request is fine, taking ~500ms, but when multiple requests come in about the same time (4-5), the request slows to ~2000ms per call. What's going on?
There's not much more to state than what I have above. I open a connection to our DB then initialize a SqlCommand.
using (var connection = new SqlConnection(dbConnection))
connection.Open();
using (SqlCommand command = new SqlCommand(strSQLCommand))
I've tried both filling a datatable with SqlDataAdapter and using a SqlDataReader to fill up a custom object, I get similar slow downs either way. As stated above the query returns ~1-4k rows of data of varying types. And Postman says the returned Json data is about 1.95MB of size after decompression. The slowdown only occurs when multiple requests come in around the same time. I don't know if it's having trouble with multiple connections to the db, or if it's about the size of the data and available memory. Paging isn't an option, the request needs to return that much data.
This all occurs within a HttpGet function
[HttpGet]
[Route("Foo")]
[Consumes("application/json")]
[EnableCors("DefaultPolicy")]
public IActionResult Foo([FromHeader] FooRequest request)
{
///stuff
DataTable dt = new DataTable();
using (var connection = new SqlConnection(_dataDBConnection))
{
timer.Start();
connection.Open();
using (SqlCommand command = new SqlCommand(
"SELECT foo.name, bar.first, bar.second, bar.third, bar.fourth
FROM dbo.foo with(nolock)
JOIN dbo.bar with(nolock) ON bar.name = foo.name
WHERE bar.date = #date", connection))
{
command.Parameters.AddWithValue("#date", request.Date.ToString("yyyyMMdd"));
using (SqlDataAdapter adapter = new SqlDataAdapter(command))
{
adapter.Fill(dt);
}
}
timer.Stop();
long elapsed = timer.ElapsedMilliseconds;
}
///Parse the data from datatable into a List<object> and return
///I've also used a DataReader to put the data directly into the List<object> but experienced the same slowdown.
///response is a class containing an array of objects that returns all the data from the SQL request
return new JsonResult(response);
}
Any insights would be appreciated!
--EDIT AFTER ADDITOINAL TESTING---
[HttpGet]
[Route("Foo")]
[Consumes("application/json")]
[EnableCors("DefaultPolicy")]
public IActionResult Foo([FromHeader] FooRequest request)
{
///stuff
using (var connection = new SqlConnection(_dataDBConnection))
{
connection.Open();
///This runs significantly faster
using (SqlCommand command = new SqlCommand(#"dbo.spGetFoo", connection))
{
command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("#date", request.date.ToString("yyyyMMdd"));
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
///Add data to list to be returned
}
}
}
}
///Parse the data from datatable into a List<object> and return
///I've also used a DataReader to put the data directly into the List<object> but experienced the same slowdown.
///response is a class containing an array of objects that returns all the data from the SQL request
return new JsonResult(response);
}
--FINAL EDIT PLEASE READ--
People seem to be getting caught up on the DataAdapter and Fill portion instead of reading the full post. So, I'll include a final example here that provides the same issue above.
[HttpGet]
[Route("Foo")]
[Consumes("application/json")]
[EnableCors("DefaultPolicy")]
public async Task<IActionResult> Foo([FromHeader] FooRequest request)
{
///stuff
using (var connection = new SqlConnection(_dataDBConnection))
{
await connection.OpenAsync();
///This runs significantly faster
using (SqlCommand command = new SqlCommand(#"dbo.spGetFoo", connection))
{
command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("#date", request.date.ToString("yyyyMMdd"));
using (SqlDataReader reader = await command.ExecuteReaderAsync())
{
while (await reader.ReadAsync())
{
///Add data to list to be returned
}
}
}
}
///Parse the data from datatable into a List<object> and return
///response is a class containing an array of objects that returns all the data from the SQL request
return new JsonResult(response);
}

First thing to note here is that your action method is not asynchronous. Second thing to note here is that using adapters to fill datasets is something I hadn't seen for years now. Use Dapper! Finally, that call to the adapter's Fill() method is most likely synchronous. Move to Dapper and use asynchronous calls to maximize your ASP.net throughput.

I think your idea is correct, it shouldn't be a database problem.
I think that Session can be one suspect. If you use ASP.NET Core
Session in your application, requests are queued and processed one by
one. So, the last request can stay holt in the queue while the
previous requests are being processed.
Another can be bits of MVC running in your pipeline and that can bring
Session without asking you.
In addition, another possible reason is that all threads in the
ASP.NET Core Thread Pool are
busy.
In this case, a new thread will be created to process a new request
that takes additional time.
This is just my idea, any other cause is possible. Hope it can help you.

The reason this is slow is that the method is not async. This means that threads are blocked. Since Asp.Net has a limited thread pool, it will be exhausted after a while, and then additional requests will have to queue, which makes the system slow. All of this should be fixed by using async await pattern.
Since SQLDataAdapter does not provide any async methods, it could be easier to use a technology which provides such an async methods, e.g. EF Core. Otherwise you could start a new task for adapter.Fill, however, this is not a clean way of doing it.

How to execute RAW SQL Query with EF Core?

Basically the problem i have is that i want to run a query in a database that it's not a representation of my model.
This is my code to create the connection to another database:
public static OtherContext GetNewContextGeneric(string connectionString)
{
var builder = new DbContextOptionsBuilder();
builder.UseSqlServer(connectionString);
OtherContext db = new OtherContext(builder.Options);
return db;
}
And this is my code to execute the query:
public List<IQueryble> Query (string connectionString, string query)
{
try
{
using(var contextGeneric = ContextFactory.GetNewContextGeneric(connectionString))
{
//I want something like this
return contextGeneric.Query(query).ToList();
}
}
catch(System.Data.SqlClient.SqlException ex)
{
throw new SQLIncorrectException(ex);
}
catch(System.InvalidOperationException ex)
{
throw new NotImplementedException();
}
}
Can somebody help me?

You can use DbDataReader:
using (var command = context.Database.GetDbConnection().CreateCommand())
{
command.CommandText = "SELECT * From Make";
context.Database.OpenConnection();
using (var reader = command.ExecuteReader())
{
// Do something with result
reader.Read(); // Read first row
var firstColumnObject = reader.GetValue(0);
var secondColumnObject = reader.GetValue(1);
reader.Read(); // Read second row
firstColumnObject = reader.GetValue(0);
secondColumnObject = reader.GetValue(1);
}
}
Here you can learn more how to read values from DbDataReader.
Alternatively you could use FromSql() method, but that works only on predefined DbSet of some entity, which is not the solution you wanted.

In the question you say:
Basically the problem i have is that i want to run a query in a database that it's not a representation of my model.
and then in comments you add:
Because i don't know how is created the database, i don't know what tables are in the database i want to insert the sql query
Well, if you don't know the database, then you cannot use Entity Framework, as it requires you to have a detailed knowledge of the database you are connecting to.
You should use plain ADO.NET (or Dapper if you want to map results back to a known class) for this.

Your use case of an external database can still be achieved using EF Core, there is no need to resort to ADO.Net, this solution is based on this generic solution for EF Core by #ErikEj. (Note that some functions and namespaces changed for EF Core 3, so and remain in .Net 5+ but the same generic concept can still be applied)
public static IList<T> Query<T>(string connectionString, string query, params object[] parameters) where T : class
{
try
{
using (var contextGeneric = new ContextForQuery<T>(connectionString))
{
return contextGeneric.Query<T>().FromSql(query, parameters).ToList();
}
}
catch (System.Data.SqlClient.SqlException ex)
{
throw new SQLIncorrectException(ex);
}
catch (System.InvalidOperationException ex)
{
throw new NotImplementedException();
}
}
private class ContextForQuery<T> : DbContext where T : class
{
private readonly string connectionString;
public ContextForQuery(string connectionString)
{
this.connectionString = connectionString;
}
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseSqlServer(connectionString, options => options.EnableRetryOnFailure());
base.OnConfiguring(optionsBuilder);
}
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<T>().HasNoKey();
base.OnModelCreating(modelBuilder);
}
}
Then the usage of this requires a concrete type definition, to add support for anonymous types is a fair bit more effort, but creating a concrete type for this is not a bad thing, the whole point here is to try you towards more declarative code styles as they enhance the readability and inspection of the code as well as providing documentation and other extended configuration like related entities.
public class NamedObject
{
public int Id { get; set; }
public string Name { get; set; }
}
...
var connectionString = "Insert your connection string here...";
var data = Query<NamedObject>(connectionString, "SELECT TOP 10 Id, FullName as Name FROM Employee");
foreach (var emp in data)
{
Console.WriteLine(emp.Name);
}
Background
In EF 6 (.Net Framework) we could use DbContext.Database.FromSQL<T>() to execute ad-hoc SQL that would be automatically mapped to the specified type of T. This functionality was not replicated in EF Core because the result of FromSQL was inconsistent with the rest of EF, the result was a single use IEnumerable<T>. You could not further compose this query to Include() related entities nor could you add a filter to the underlying query.
In EF Core to Execute Raw SQL the type T that you want to return needs to be defined in the DbContext as a DbSet<T>. This set does not need to map to a table in the database at all, in fact since EF Core 2.1 we do not event need to specify a key for this type, it is simply a mechanism to pre-define the expected structure instead of executing Ad-Hoc requests on demand, it offers you the same functionality as the legacy FromSQL but also allows for you to define a rich set of navigation properties that would enable further composition of the query after your RawSQL is interpolated with the LINQ to SQL pipeline.
Once the type is defined in the context you simply call DbSet<T>.FromSqlRaw(). The difference is that now we have an IQueryable<T> that we can use to futher compose to include related entities or apply filters that will be evaluated within the database.
The solution posted in this response doesn't allow for composition, but uses the EF runtime in the expected sequence to give the same behaviours as the original EF 6 implementation.
In more recent versions of EF Core, and now in .Net 5+ the following subtle change need to be applied:
Core 2.1: return contextGeneric.Query<T>().FromSql(query, parameters).ToList();
Core 3+: return contextGeneric.Set<T>().FromSqlRaw(query, parameters).ToList();

You can use context.Database.ExecuteSqlRaw("select 1")
Don't forget to import the right namespace : using Microsoft.EntityFrameworkCore;

It worked like this:
private void SqlCommand (string connectionString, string query)
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
SqlCommand command = new SqlCommand(query, connection);
connection.Open();
SqlDataReader reader = command.ExecuteReader();
try
{
while (reader.Read())
{
var a = reader[0];
}
}
finally
{
// Always call Close when done reading.
reader.Close();
}
}
}
Or
using (var connection = ContextFactory.GetNewContextGeneric(connectionString).Database.GetDbConnection())
{
connection.Open();
DbCommand command = connection.CreateCommand();
command.CommandText = query;
using (var reader = command.ExecuteReader())
{
// Do something with result
reader.Read(); // Read first row
var firstColumnObject = reader.GetValue(0);
/*var secondColumnObject = reader.GetValue(1);
reader.Read(); // Read second row
firstColumnObject = reader.GetValue(0);
secondColumnObject = reader.GetValue(1);*/
connection.Close();
return firstColumnObject.ToString();
}
}

How to call Async stored procedure calls in C# in a generic way without Entity Framework

I would like to refactor many stored procedure calls that all have a very similar pattern to use a generic method. I'm still new to async programming so would like some advice on the impact of my intention.
Unfortunately I would not be able to use EF
Existing example:
var myModel = new Model();
using (var connection = new SqlConnection("Some Connection String"))
{
using (var command = new SqlCommand("Some Stored Procedure Name", connection)
{
CommandType = CommandType.StoredProcedure
})
{
await connection.OpenAsync();
//Code Building a list of params
using (var reader = await command.ExecuteReaderAsync())
{
while (await reader.ReadAsync())
{
//Code Building myModel
}
}
}
}
return myModel;
What I would like to achieve is a method in which you pass the stored procedure name and a list of parameters, and it would return an object that I can iterate though to retrieve the values I need.
Are there any considerations I need to be aware of regarding OpenAsync/ReadAsync? Also what object would be best to return data from the reader?

Use of SqlDataSource From Non-Control Situations

As part of my common utilities I used in all my line of business applications, I have this code...
using System.Web.UI.WebControls;
public class Database
{
/// <summary>
/// Creates a DataView object using the provided query and an SqlDataSource object.
/// </summary>
/// <param name="query">The select command to perform.</param>
/// <returns>A DataView with data results from executing the query.</returns>
public static DataView GetDataView(string query)
{
SqlDataSource ds = GetDBConnection();
ds.SelectCommand = query;
DataView dv = (DataView)ds.Select(DataSourceSelectArguments.Empty);
return dv;
}
/// <summary>
/// Creates a SqlDataSource object with initialized connection string and provider
/// </summary>
/// <returns>An SqlDataSource that has been initialized.</returns>
public static SqlDataSource GetDBConnection()
{
SqlDataSource db = new SqlDataSource();
db.ConnectionString = GetDefaultConnectionString(); //retrieves connection string from .config file
db.ProviderName = GetDefaultProviderName(); //retrieves provider name from .config file
return db;
}
}
Then, in my projects, to retrieve data from databases I'll have some code like..
DataView dv=Database.GetDataView("select mycolumn from my table");
//loop through data and make use of it
I have taken some heat from people for using SqlDataSource in this manner. People don't seem to like that I'm using a Web control purely from code instead of putting it on an ASPX page. It doesn't look right to them, but they haven't been able to tell me a downside. So, is there a downside? This is my main question. Because if there's a lot of downsides, I might have to change how I'm doing many internal applications I've developed.
My Database class even works from non-ASP.NET situations, so long as I add the System.Web assembly. I know it's a slight increase in package size, but I feel like it's worth it for the type of application I'm writing. Is there a downside to using SqlDataSource from say a WPF/Windows Forms/Console program?

Well, there are no hard rules stopping anyone from doing such implementation.
However, following are few questions that need to be answered before doing that implementation.
Is this usage thread safe? (because there is every possibility the same call can be made by multiple consuming applications.
Will there be a layered differentiation (UI.Control being used in a Data layer)?
What if that control becomes obsolete / restricted in the next framework releases?

Given how easy it is to replace this code, whilst removing the temptation to use dynamic SQL queries to pass parameters, I think the question should be: is there any benefit to keeping the code as-is?
For example:
public static class Database
{
private static readonly Func<DbCommandBuilder, int, string> getParameterName = CreateDelegate("GetParameterName");
private static readonly Func<DbCommandBuilder, int, string> getParameterPlaceholder = CreateDelegate("GetParameterPlaceholder");
private static Func<DbCommandBuilder, int, string> CreateDelegate(string methodName)
{
MethodInfo method = typeof(DbCommandBuilder).GetMethod(methodName, BindingFlags.Instance | BindingFlags.NonPublic, Type.DefaultBinder, new Type[] { typeof(Int32) }, null);
return (Func<DbCommandBuilder, int, string>)Delegate.CreateDelegate(typeof(Func<DbCommandBuilder, int, string>), method);
}
private static string GetDefaultProviderName()
{
...
}
private static string GetDefaultConnectionString()
{
...
}
public static DbProviderFactory GetProviderFactory()
{
string providerName = GetDefaultProviderName();
return DbProviderFactories.GetFactory(providerName);
}
private static DbConnection GetDBConnection(DbProviderFactory factory)
{
DbConnection connection = factory.CreateConnection();
connection.ConnectionString = GetDefaultConnectionString();
return connection;
}
public static DbConnection GetDBConnection()
{
DbProviderFactory factory = GetProviderFactory();
return GetDBConnection(factory);
}
private static void ProcessParameters(
DbProviderFactory factory,
DbCommand command,
string query,
object[] queryParameters)
{
if (queryParameters == null && queryParameters.Length == 0)
{
command.CommandText = query;
}
else
{
IFormatProvider formatProvider = CultureInfo.InvariantCulture;
DbCommandBuilder commandBuilder = factory.CreateCommandBuilder();
StringBuilder queryText = new StringBuilder(query);
for (int index = 0; index < queryParameters.Length; index++)
{
string name = getParameterName(commandBuilder, index);
string placeholder = getParameterPlaceholder(commandBuilder, index);
string i = index.ToString("D", formatProvider);
command.Parameters.AddWithValue(name, queryParameters[index]);
queryText = queryText.Replace("{" + i + "}", placeholder);
}
command.CommandText = queryText.ToString();
}
}
public static DataView GetDataView(string query, params object[] queryParameters)
{
DbProviderFactory factory = GetProviderFactory();
using (DbConnection connection = GetDBConnection(factory))
using (DbCommand command = connection.CreateCommand())
{
command.CommandType = CommandType.Text;
ProcessParameters(factory, command, query, queryParameters);
DbDataAdapter adapter = factory.CreateDataAdapter();
adapter.SelectCommand = command;
DataTable table = new DataTable();
adapter.Fill(table);
return table.DefaultView;
}
}
}
With this version, you can now pass in parameters simply and safely, without relying on custom code to try to block SQL injection:
DataView dv = Database.GetDataView(
"select mycolumn from my table where id = {0} and name = {1}",
1234, "Robert');DROP TABLE Students;--");
EDIT
Updated to support parameters for different providers, with help from this answer.

The only issues I see are
(1) this is like reinventing the wheel. There is Enterprise library v5 for FW3.5 and v6 for FW4.5, which has data access components. Use that.
With EL you can make a call and have 2,3,4 tables loaded in Dataset. With your method this is not possible, only one at the time.
Enterprise library is a complete Data Access suite provided by Microsoft. It takes care of all the little details and all you need is to call your data. This is complete data access layer. And if you look deeper, EL allows for integration of Data and Caching, and other things. But you don't have to use what you don't need. If you need data access you can use only that.
And (2) Generally, this is not a good idea to write low level assembly with high-level assembly in reference. Anything System.Web.... is UI and client related stuff. In a layered cake design this is like the top of it and Data Access is on the bottom. All references [save for "common"] should travel from bottom to the top and you have it in opposite direction.
Look at this picture:
This is from Microsoft. You see the layers of the "cake". All references are going up. What you've done - you took UI-related component and wrote Data Access in it.
You can call it opinion-based - but this opinion is standard practice and pattern in software development. Your question is also opinion based. Because you can code everything in single file, single class, and it will work. You can set references to System.Windows.Forms in Asp.net application, if you want to. Technically, it is possible but it is really bad practice.
Your application now have limited reusability. What if you write WPF component or service that need to use same Data Access. You have to drag all System.Web into it?

Cannot implicitly convert type DbDataReader to MySqlDataReader when using ExecuteReaderAsync

I've the following function that allows me to pass in a object and populate that object with the returning data, if any.
I've modified the function so that it can be called asynchronously.
public static async Task<MySqlDataReader> MySQLReturnReader(string sName, List<MySqlParameter> oParameters, Action<MySqlDataReader> fn)
{
using (MySqlConnection oConn = new MySqlConnection(MasterConn))
{
await oConn.OpenAsync();
using (MySqlCommand oMySqlCommand = new MySqlCommand(sName, oConn))
{
oMySqlCommand.CommandType = CommandType.StoredProcedure;
if (oParameters != null)
{
foreach (MySqlParameter oPar in oParameters)
{
oMySqlCommand.Parameters.Add(oPar);
}
}
oMySqlCommand.Connection.Open();
using (MySqlDataReader oReader = oMySqlCommand.ExecuteReaderAsync())
{
fn(oReader);
}
}
}
return;
}
My class object is something like;
public class MyClass
{
public int Id {get;set;}
public string Name {get;set;}
...
}
The function can be called like
List<MyClass> oMyClassList = new List<MyClass>();
List<MySqlParameter> oParams = new List<MySqlParameter>();
List<int> Ids = new List<int>(500);
Ids.Add(1);
Ids.Add(2);
...
Ids.Add(499);
foreach(int Id in Ids)
{
MySQLReturnReader("SPCheck", oParams, oRes =>
{
while (oRes.Read())
{
MyClass oMyClass = new MyClass();
oMyClass.Id = Convert.ToInt32(oRes["Id"]);
oMyClass.Name = oRes["Name"].ToString();
}
oMyClassList.Add(oMyClass);
}
);
}
The problem is I'm getting the compilation error of 'Cannot implicitly convert type 'System.Threading.Tasks.Task' to 'MySql.Data.MySqlClient.MySqlDataReader'. Where am I going wrong ?
I'm wanting to use ExecuteReaderAsync in this way, as the Stored procedure called is very complex and would prefer to run the requests in parallel.

In your code, you have:
using (MySqlDataReader oReader = oMySqlCommand.ExecuteReaderAsync())
The compiler error Cannot implicitly convert type 'System.Threading.Tasks.Task' to 'MySql.Data.MySqlClient.MySqlDataReader' means that you need to use the await keyword to "unwrap" the task returned by ExecuteReaderAsync:
using (MySqlDataReader oReader = await oMySqlCommand.ExecuteReaderAsync())
Note however that if you're using the official MySql.Data package, this call won't actually execute asynchronously. The Async methods in the MySql.Data connector aren't asynchronous; they block on network I/O and only return when the DB operation is complete. (For a much more detailed discussion, see this question and its top answer.) MySQL bug #70111 reports this problem in the MySQL connector.
To get truly asynchronous DB operations, you'll have to wait until that bug is fixed, or switch to a different connector. I've been developing a new, fully async connector that should be a drop-in replacement for MySql.Data; to try it out, install MySqlConnector from NuGet; its source is at GitHub.

This most likely indicates that the library you're using doesn't support ExecuteReaderAsync(), so you're just calling the default implementation inherited from DbCommand. This is why it returns the general DbDataReader (instead of the MySQL-specific one). And this also means that your code won't actually be asynchronous, the default version of ExecuteReaderAsync() is just a synchronous wrapper around ExecuteReader().
So, I think you should directly use the old ExecuteReader(), until your library adds support for ExecuteReaderAsync().