We Keep Coding

IAsyncEnumerable and database queries

I have three controller methods returning IAsyncEnumerable of WeatherForecast.
The first one #1 uses SqlConnection and yields results from an async reader.
The second one #2 uses EF Core with the ability to use AsAsyncEnumerable extension.
The third one #3 uses EF Core and ToListAsync method.
I think the downside of #1 and #2 is if I, for example, do something time-consuming inside while or for each then the database connection will be open till the end. In scenario #3 I'm able to iterate over the list with a closed connection and do something else.
But, I don't know if IAsyncEnumerable makes sense at all for database queries. Are there any memory and performance issues? If I use IAsyncEnumerable for returning let's say HTTP request from API, then once a response is returned it's not in memory and I'm able to return the next one and so on. But what about the database, where is the whole table if I select all rows (with IAsyncEnumerable or ToListAsync)?
Maybe it's not a question for StackOverflow and I'm missing something big here.
#1
[HttpGet("db", Name = "GetWeatherForecastAsyncEnumerableDatabase")]
public async IAsyncEnumerable<WeatherForecast> GetAsyncEnumerableDatabase()
{
var connectionString = "";
await using var connection = new SqlConnection(connectionString);
string sql = "SELECT * FROM [dbo].[Table]";
await using SqlCommand command = new SqlCommand(sql, connection);
connection.Open();
await using var dataReader = await command.ExecuteReaderAsync();
while (await dataReader.ReadAsync())
{
yield return new WeatherForecast
{
Date = Convert.ToDateTime(dataReader["Date"]),
Summary = Convert.ToString(dataReader["Summary"]),
TemperatureC = Convert.ToInt32(dataReader["TemperatureC"])
};
}
await connection.CloseAsync();
}
#2
[HttpGet("ef", Name = "GetWeatherForecastAsyncEnumerableEf")]
public async IAsyncEnumerable<WeatherForecast> GetAsyncEnumerableEf()
{
await using var dbContext = _dbContextFactory.CreateDbContext();
await foreach (var item in dbContext
.Tables
.AsNoTracking()
.AsAsyncEnumerable())
{
yield return new WeatherForecast
{
Date = item.Date,
Summary = item.Summary,
TemperatureC = item.TemperatureC
};
}
}
#3
[HttpGet("eflist", Name = "GetWeatherForecastAsyncEnumerableEfList")]
public async Task<IEnumerable<WeatherForecast>> GetAsyncEnumerableEfList()
{
await using var dbContext = _dbContextFactory.CreateDbContext();
var result = await dbContext
.Tables
.AsNoTracking()
.Select(item => new WeatherForecast
{
Date = item.Date,
Summary = item.Summary,
TemperatureC = item.TemperatureC
})
.ToListAsync();
return result;
}

Server-Side
If I only cared with the server I'd go with option 4 in .NET 6 :
Use an injected DbContext, write a LINQ query and return the results as AsAsyncEnumerable() instead of ToListAsync()
public class WeatherForecastsController:ControllerBase
{
WeatherDbContext _dbContext;
public WeatherForecastsController(WeatherDbContext dbContext)
{
_dbContext=dbContext;
}
public async IAsyncEnumerable<WeatherForecast> GetAsync()
{
return _dbContext.Forecasts.AsNoTracking()
.Select(item => new WeatherForecast
{
Date = item.Date,
Summary = item.Summary,
TemperatureC = item.TemperatureC
})
.AsAsyncEnumerable();
}
}
A new Controller instance is created for every request which mean the DbContext will be around for as long as the request is being processed.
The [FromServices] attribute can be used to inject a DbContext into the action method directly. The behavior doesn't really change, the DbContext is still scoped to the request :
public async IAsyncEnumerable<WeatherForecast> GetAsync([FromServices] WeatherContext dbContext)
{
...
}
ASP.NET Core will emit a JSON array but at least the elements will be sent to the caller as soon as they're available.
Client-Side
The client will still have to receive the entire JSON array before deserialization.
One way to handle this in .NET 6 is to use DeserializeAsyncEnumerable to parse the response stream and emit items as they come:
using var stream=await client.GetAsStreamAsync(...);
var forecasts= JsonSerializer.DeserializeAsyncEnumerable(stream, new JsonSerializerOptions
{
DefaultBufferSize = 128
});
await foreach(var forecast in forecasts)
{
...
}
The default buffer size is 16KB so a smaller one is needed if we want to receive objects as soon as possible.
This is a parser-specific solution though.
Use a streaming JSON response
A common workaround to this problem is to use streaming JSON aka JSON per line, aka Newline Delimited JSON aka JSON-NL or whatever. All names refer to the same thing - sending a stream of unindented JSON objects separated by a newline. It's an old technique that many tried to hijack and present as their own
{ "Date": "2022-10-18", Summary = "Blah", "TemperatureC"=18.5 }
{ "Date": "2022-10-18", Summary = "Blah", "TemperatureC"=18.5 }
{ "Date": "2022-10-18", Summary = "Blah", "TemperatureC"=18.5 }
That's not valid JSON but many parsers can handle it. Even if a parser can't, we can simply read one line of text at a time and parse it.
Use a different protocol
Even streaming JSON responses is a workaround. HTTP doesn't allow server-side streaming in the first place. The server has to send all data even if the client only reads the first 3 items since there's no way to cancel the response.
It's more efficient to use a protocol that does allow streaming. ASP.NET Core 6 offers two options:
Use ASP.NET Core SignalR with Server-side streaming to push data to clients over WebSockets
A gRPC Service that uses server-side streaming
In both cases the server sends objects to clients as soon as they're available. Clients can cancel the stream as needed.
In a SignalR hub, the code could return an IAsyncEnumerable or a Channel:
public class AsyncEnumerableHub : Hub
{
...
public async IAsyncEnumerable<WeatherForecast> GetForecasts()
{
return _dbContext.Forecasts.AsNoTracking()
.Select(item => new WeatherForecast
{
Date = item.Date,
Summary = item.Summary,
TemperatureC = item.TemperatureC
})
.AsAsyncEnumerable();
}
}
In gRPC, the server method writes objects to a response stream :
public override async Task StreamingFromServer(ForecastRequest request,
IServerStreamWriter<ForecastResponse> responseStream, ServerCallContext context)
{
...
await foreach (var item in queryResults)
{
if (context.CancellationToken.IsCancellationRequested)
{
return;
}
await responseStream.WriteAsync(new ForecastResponse{Forecast=item});
}
}

I want to pass a int "index" parameter to an method dialog and also ask user to input a string for specific value he is going to search

I am writing a chat bot that can ask a user for name and a requirement to search job with.
Name and Requirement will be stored in UserData and PrivateConversationData.
I am trying to send the requirement as an index, 1-5, to a method dialog and to specify a requirement like salary amount and then call an appropriate api. But the bot keep giving an error when passing the parameter. How can I fix the error? Is it the way my method receives it or I use the wrong way to call it?
I'm trying to combine the requirement and job stuff into one single to prevent [Community Edit: To prevent what?]
private async Task MessageReceivedAsync(IDialogContext context, IAwaitable<object> result)
{
if (string.IsNullOrEmpty(name))//ask for the name
{
//code for get name
}
else
{
context.PrivateConversationData.TryGetValue<int>("Index", out int index);
if (!Enumerable.Range(1, 5).Contains(index))
{
var getRequirement =
FormDialog.FromForm(Requirement.BuildForm,
FormOptions.PromptInStart);
context.Call(getRequirement, AfterGetRequirementAsync);//able to get requirement index as int 1~5. next going to ask what specific value
}
else
{
await context.PostAsync($"{name}:{index}: {activity.Text}");//testing: it is able to print the name and the index user choose
context.Wait(MessageReceivedAsync);
}
}
I am using context.Call(getRequirement, AfterGetRequirementAsync) to try to get both requirement and then ask for the specific value in AfterGetRequirementAsync.
private async Task AfterGetRequirementAsync(IDialogContext context, IAwaitable<Requirement> result)
{
//Requirement requirementindex = null;
var requirementindex = await result;
context.PrivateConversationData.SetValue<int>("Index", requirementindex.Index);
await context.PostAsync($"Result：{JsonConvert.SerializeObject(requirementindex)}");//for testing index after user's input about index
var jobs = await GetJobDialog.LoadJob(requirementindex.Index);//Im trying to pass the .Index
//await context.PostAsync($"Job Search Result : {Environment.NewLine}{JsonConvert.SerializeObject(jobs)}");//trying to print the list of result to user
context.Wait(MessageReceivedAsync);
}
In AfterGetRequirementAsync, it is able to get the requirementindex and I can store it in PrivateConversationData in MessageReceivedAsync as Index. But when I try to pass the requirementindex.Index to GetJobDialog.JobModel.LoadJob it give me error of [Community Edit: What's the error?]
public class GetJobDialog
{
public Task StartAsync(IDialogContext context)
{
context.Wait(LoadJob(context.UserData));
return Task.CompletedTask;
}
public static async Task<List<JobModel>> LoadJob(IDialog context, IAwaitable<JobResultModel> result, int index)//depends on the job searching index, using different method to search.
{//it should return list of jobs to user.
string url = "";
if (index == 1)//posting type
{ //code like index == 4 }
else if (index == 2)//level
{ //code like index == 4 }
else if (index == 3)//full time or part time
{ //code like index == 4 }
else if (index == 4)//salary from
{ //ask for internal or external and change the end= to match value
url = $"https://data.cityofnewyork.us/resource/kpav-sd4t.json?salary_range_from=40000";
}//the only thing different is after .json? the index = specific value
else if (index == 5)//salary to
{ //code like index == 4 }
else//if not getting any valid option, throw error message and ask for index again
{
}
using (HttpResponseMessage response = await ApiHelper.ApiHelper.ApiClient.GetAsync(url))
{
if (response.IsSuccessStatusCode)
{
JobResultModel job = await response.Content.ReadAsAsync<JobResultModel>();
return job.Results;
}
else
{
throw new Exception(response.ReasonPhrase);
}
}
}
}
I am also trying to get the user to input the specific amount in the GetJobDialog. That way, the user doesn't have to enter anything to trigger the chat bot again.
I'm just posting the API caller incase I have some mistake because I learn all these by myself and do not have a clear understanding of how C# and api work.
public static class ApiHelper
{
public static HttpClient ApiClient { get; set; }
public static void InitializeClient()
{
ApiClient = new HttpClient();
ApiClient.DefaultRequestHeaders.Accept.Clear();
ApiClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
}
}
I expect the chat bot be able to pass the index to LoadJob and ask for specific value before or after the call of LoadJob. Then response a list of job with different field to the user.

I see a few issues with your code that might be causing this. If you can link me to all of your code, I might be able to help debug. In the meantime, here's some things might be causing the problem:
You're using BotBuilder V3 Code. If you're writing a new bot, you should really be using Botframework V4. Here's a State Management Sample Bot that can help you get started. You really should switch to V4, especially if this is a newer bot. If you run into issues in the future, you'll receive better support.
It looks like you're saving Index to PrivateConversationData, but when you pass it LoadJob(), you're using UserData instead.
I'm not sure that you can pass information when using context.Wait(). None of the V3 Samples do that. I don't use V3 much, so I can't tell for sure. What you should instead do, is use something like context.PrivateConversationData.TryGetValue<int>("Index", out int index); to load the index instead of passing it.
It also looks like you didn't post the error message. If you can post that and all of your code, I can help debug further (if the above doesn't work).

Avoid fast post on webapi c#

I have problem in when user post the data. Some times the post run so fast and this make problem in my website.
The user want to register a form about 100$ and have 120$ balance.
When the post (save) button pressed sometimes two post come to server very fast like:
2018-01-31 19:34:43.660 Register Form 5760$
2018-01-31 19:34:43.663 Register Form 5760$
Therefore my client balance become negative.
I use If in my code to check balance but the code run many fast and I think both if happen together and I missed them.
Therefore I made Lock Controll class to avoid concurrency per user but not work well.
I made global Action Filter to control the users this is my code:
public void OnActionExecuting(ActionExecutingContext context)
{
try
{
var controller = (Controller)context.Controller;
if (controller.User.Identity.IsAuthenticated)
{
bool jobDone = false;
int delay = 0;
int counter = 0;
do
{
delay = LockControllers.IsRequested(controller.User.Identity.Name);
if (delay == 0)
{
LockControllers.AddUser(controller.User.Identity.Name);
jobDone = true;
}
else
{
counter++;
System.Threading.Thread.Sleep(delay);
}
if (counter >= 10000)
{
context.HttpContext.Response.StatusCode = 400;
jobDone = true;
context.Result = new ContentResult()
{
Content = "Attack Detected"
};
}
} while (!jobDone);
}
}
catch (System.Exception)
{
}
}
public void OnActionExecuted(ActionExecutedContext context)
{
try
{
var controller = (Controller)context.Controller;
if (controller.User.Identity.IsAuthenticated)
{
LockControllers.RemoveUser(controller.User.Identity.Name);
}
}
catch (System.Exception)
{
}
}
I made list static list of user and sleep their thread until previous task happen.
Is there any better way to manage this problem?

So the original question has been edited so this answer is invalid.
so the issue isn't that the code runs too fast. Fast is always good :) The issue is that the account is going into negative funds. If the client decides to post a form twice that is the clients fault. It maybe that you only want the client to pay only once which is an other problem.
So for the first problem, I would recommend a using transactions (https://en.wikipedia.org/wiki/Database_transaction) to lock your table. Which means that the add update/add a change (or set of changes) and you force other calls to that table to wait until those operations have been done. You can always begin your transaction and check that the account has the correct amount of funds.
If the case is that they are only ever meant to pay once then.. then have a separate table that records if the user has payed (again within a transaction), before processing the update/add.
http://www.entityframeworktutorial.net/entityframework6/transaction-in-entity-framework.aspx
(Edit: fixing link)

You have a few options here
You implement ETag functionality in your app which you can use for optimistic concurrency. This works well, when you are working with records, i.e. you have a database with a data record, return that to the user and then the user changes it.
You could add an required field with a guid to your view model which you pass to your app and add it to in memory cache and check it on each request.
public class RegisterViewModel
{
[Required]
public Guid Id { get; set; }
/* other properties here */
...
}
and then use IMemoryCache or IDistributedMemoryCache (see ASP.NET Core Docs) to put this Id into the memory cache and validate it on request
public Task<IActioNResult> Register(RegisterViewModel register)
{
if(!ModelState.IsValid)
return BadRequest(ModelState);
var userId = ...; /* get userId */
if(_cache.TryGetValue($"Registration-{userId}", register.Id))
{
return BadRequest(new { ErrorMessage = "Command already recieved by this user" });
}
// Set cache options.
var cacheEntryOptions = new MemoryCacheEntryOptions()
// Keep in cache for 5 minutes, reset time if accessed.
.SetSlidingExpiration(TimeSpan.FromMinutes(5));
// when we're here, the command wasn't executed before, so we save the key in the cache
_cache.Set($"Registration-{userId}", register.Id, cacheEntryOptions );
// call your service here to process it
registrationService.Register(...);
}
When the second request arrives, the value will already be in the (distributed) memory cache and the operation will fail.
If the caller do not sets the Id, validation will fail.
Of course all that Jonathan Hickey listed in his answer below applies to, you should always validate that there is enough balance and use EF-Cores optimistic or pessimistic concurrency

ASP.Net core sometimes the action controller is called twice

I have an application is ASP.net core and have integrated spreedly payment gateway for processing the payments. In my log files I can see that sometimes the payment controller executes twice. I generate an ID based on the time the request was received and the ID's are sometimes apart by 1 sec or at sometimes they are at the exact same time. This is resulting in charging the card twice only for few cases when this is happening. I cant seem to figure out what could be triggering this.
Following is the code that I am using
The user fills the application form and on the pay button click I am using this code to trigger spreedly
$('#REG').click(function () {
var options = {
company_name: "abcd",
sidebar_top_description: "Fees",
sidebar_bottom_description: "Only Visa and Mastercard accepted",
amount: "#string.Format("{0:c}",Convert.ToDecimal(Model.FeeOutstanding))"
}
document.getElementById('payment').value = 'App'
SpreedlyExpress.init(environmentKey, options);
SpreedlyExpress.openView();
$('#spreedly-modal-overlay').css({ "position": "fixed", "z-index": "9999", "bottom": "0", "top": "0", "right": "0", "left": "0" });
});
This opens the spreedly payment form as a popup where the user enters all the card information and hits the pay button. Which executes the payment controller
public async Task<IActionResult> Index(DynamicViewModel model)
{
if (ModelState.IsValid)
{
try
{
if (TempData.ContainsKey("PaymentFlag") && !String.IsNullOrEmpty(TempData["PaymentFlag"].ToString()))
{
// Some code logic that calls few async methods
//generate a id based on the time of current request
"APP-" + DateTime.Now.ToString("yyyyMMddHmmss-") + model.UserID;
// ... Other code here
}
The id that I generate is logged and I can see that some times in the log file that it ran twice for a customer with the ID having either the exact same time or there was a 1 sec difference. I have tested the double click scenario and also have put in some code to prevent double clicks. But still I cant seem to understand why sometimes this happens. It is not frequent. Its like 1 case that happens in 100 payments.
I have an action attribute to handle the duplicate requests. After putting in this code it did stopped the number of duplicate requests but not completely. Still in few cases some how the controllers gets called twice.
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
public class NoDuplicateRequestAttribute : ActionFilterAttribute
{
public int DelayRequest = 10;
// The Error Message that will be displayed in case of
// excessive Requests
public string ErrorMessage = "Excessive Request Attempts Detected.";
// This will store the URL to Redirect errors to
public string RedirectURL;
public override void OnActionExecuting(ActionExecutingContext filterContext)
{
// Store our HttpContext (for easier reference and code brevity)
var request = filterContext.HttpContext.Request;
// Store our HttpContext.Cache (for easier reference and code brevity)
var cache = filterContext.HttpContext.RequestServices.GetService<IMemoryCache>();
// Grab the IP Address from the originating Request (example)
var originationInfo = request.HttpContext.Connection.RemoteIpAddress.ToString() ?? request.HttpContext.Features.Get<IHttpConnectionFeature>()?.RemoteIpAddress.ToString();
// Append the User Agent
originationInfo += request.Headers["User-Agent"].ToString();
// Now we just need the target URL Information
var targetInfo = request.HttpContext.Request.GetDisplayUrl() + request.QueryString;
// Generate a hash for your strings (appends each of the bytes of
// the value into a single hashed string
var hashValue = string.Join("", MD5.Create().ComputeHash(Encoding.ASCII.GetBytes(originationInfo + targetInfo)).Select(s => s.ToString("x2")));
string cachedHash;
// Checks if the hashed value is contained in the Cache (indicating a repeat request)
if (cache.TryGetValue(hashValue,out cachedHash))
{
// Adds the Error Message to the Model and Redirect
}
else
{
// Adds an empty object to the cache using the hashValue
// to a key (This sets the expiration that will determine
// if the Request is valid or not)
var opts = new MemoryCacheEntryOptions()
{
SlidingExpiration = TimeSpan.FromSeconds(DelayRequest)
};
cache.Set(hashValue,cachedHash,opts);
}
base.OnActionExecuting(filterContext);
}

This isn't an ASP.NET Core issue. I'm 99% certain there are in fact multiple requests coming from the client and ASP.NET Core is simply handling them as it is meant to.
An option for you would be to put a guid or other identifier on the page and send it with the request. In your Controller, check your cache or session to see if that identifier already exists. If it does, throw an exception or return Ok() or log the occurrence or whatever you want to do in that case but don't charge the card.

Set Timeout For Controller Action

I have come across this thread already, but I might need something else for my situation.
I have an action that returns a ViewResult, which is called by the client's $.post()
JavaScript:
var link = 'GetFoo?fooBar=' + fooBar;
var jqxhr = $.post(link, function (response) {
$('#myDiv').replaceWith(response);
});
Controller:
public ViewResult GetFoo(String fooBar)
{
if (Request.IsAjaxRequest())
{
// perform a ridiculously long task (~12 minutes)
// algorithm: 1) download files from the Azure blob storage
// 2) update each file
// 3) reupload to blob storage
// 4) return a list of URIs to be displayed to the UI
return View("MyFooView", data);
}
throw new InvalidOperationException();
}
As the comment implies, there is long task running inside the Controller. (This is a document generation module that uploads PDFs to the Azure blob storage and returns a link to it to the View.)
This is working fine in my dev machine but when it goes live in a (secure) Azure production environment, it times out. I have put in lots of logging entries everywhere and as it turns out, it is able to upload the documents and return to the controller (i.e. it reaches the controller return statement above). However, when it is time to return the model data to the View, the client script doesn't called back (i.e. the div content doesn't get replaced with the results).
Is there a way to somehow prolong the timeout of the call? It is difficult to reproduce in my (unsecure) local environment so a definitive fix will help.
If I use the attribute [AsyncTimeout(3600)] on my GetFoo() method, then this action never gets called from the UI.
Any suggestions will be appreciated.

The problem is that the Azure load balancer has it's own timeout which is set to one minute. Any request that takes longer than a minute gets terminated. There is no way to change this.
The way around this in the Azure environment is to have one ajax call start the process and return some sort of process ID then have the client poll another ajax call to passing in this process ID to see if it's complete. It might looks something like this uncompiled and untested code. In javascript:
var link = 'BeginFooProcessing?fooBar=' + fooBar;
var jqxhr = $.post(link, function (response) {
var finishedlink = 'CheckFooFinished?fooId=' + response;
// Check to see if we're finished in 1 second
setTimeout("CheckIfFinishedYet('" + finishedlink + "')", 1000);
});
function CheckIfFinishedYet(finishedlink) {
var response = $.post(finishedlink, function (response) {
if (response == null) {
// if we didn't get a result, then check in another second
setTimeout("CheckIfFinishedYet('" + finishedlink + "')", 1000);
}
else {
// Yay, we've got a result so we'll just write it out
$('#myDiv').replaceWith(response);
}
});
}
And in your controller:
public ViewResult BeginFooProcessing(String fooBar)
{
if (Request.IsAjaxRequest())
{
Guid fooId = Guid.NewGuid();
var result = new FooResult
{
FooId = fooId,
HasFinishedProcessing = false,
Uris = new List<string>()
};
// This needs to go to persistent storage somewhere
// as subsequent requests may not come back to this
// webserver
result.SaveToADatabaseSomewhere();
System.Threading.Tasks.Task.Factory.StartNew(() => ProcessFoo(fooId));
return View("MyFooStartView", fooId);
}
throw new InvalidOperationException();
}
private void ProcessFoo(Guid fooId)
{
// Perform your long running task here
FooResult result = GetFooResultFromDataBase(fooId);
result.HasFinishedProcessing = true;
result.Uris = uriListThatWasCalculatedAbove;
result.SaveToADatabaseSomewhere();
}
public ViewResult CheckFooFinished(Guid fooId)
{
if (Request.IsAjaxRequest())
{
FooResult result = GetFooResultFromDataBase(fooId);
if (result.HasFinishedProcessing)
{
// Clean up after ourselves
result.DeleteFromDatabase();
return View("MyFooFinishedView", result.Uris);
}
return View("MyFooFinishedView", null);
}
throw new InvalidOperationException();
}
private class FooResult
{
public Guid FooId { get; set; }
public bool HasFinishedProcessing { get; set; }
public List<string> Uris;
}
Hopefully that will give you a starting point.

you want to look at this
answer to your question is: [AsyncTimeout(3600000)]
see more here
Controller Timeout MVC 3.0

To use Async controllers your controller has to inherit from AsyncController:
public class WebsiteController : AsyncController
And then any Action using Asynchronous methods has to use the format of
public void ActionNameAsync(int param)
public ActionResult ActionNameCompleted(int param)
Where ActionName is the name of your action, and instead of the Async function use
AsyncManager.OutstandingOperations.Increment();
each time you start a new aysnchronous method and
AsyncManager.OutstandingOperations.Decrement();
when the method finishes, after all outstanding operations have completed it'll move along to the Completed function (make sure to specify the parameters you need for the completed function in the async function so it knows what to pass along)
AsyncManager.Parameters["param"] = "my name";
Then using the AsyncTimeout attribute would actually affect the function. I'm not sure what happens if you try to apply that attribute to a action that isn't in an async controller.
These changes wouldn't require changing any of the references to the action in the javascript and what not as you'd still just request 'ActionName' and it would look to see if there was the Async/Completed setup version and if not it'd look for just that normal action and use whichever it finds.