Error when writing to response stream in AuthorizationHandler - c#

We're currently building an ASP.NET Web API and have implemented some custom authorization policies.
One of these policies checks whether the user has the 'Supervisor'-role within his team.
If not, we want to return a 403 error with a custom message.
This is how we have it now:
internal sealed class ValidTeamHandler : AuthorizationHandler<ValidTeamRequirement>
{
protected override async Task HandleRequirementAsync(AuthorizationHandlerContext context, ValidTeamRequirement requirement)
{
if (context.Resource is not DefaultHttpContext filterContext)
{
return;
}
var response = filterContext.HttpContext.Response;
// [...] removed for brevity
if (requirement.RequireTeamSupervisor)
{
var user = team.TeamUsers!.Single(tu => tu!.UserId.ToString() == userId.Value);
if (user!.Role != DDD.Enums.TeamRole.Supervisor)
{
context.Fail();
byte[] message = Encoding.UTF8.GetBytes("Only team supervisors are allowed to perform this action.");
response.OnStarting(async () =>
{
filterContext.HttpContext.Response.StatusCode = 403;
filterContext.HttpContext.Response.ContentType = "application/json";
await response.Body.WriteAsync(message);
});
}
}
}
}
We have some integration tests implemented that test this policy. When running these tests locally, we have no issues and everything works how it's supposed to work.
However, when we deploy the API to Azure (App Service), we get the following error whilst running these same tests:
Cannot write to the response body, the response has completed. Object name: 'HttpResponseStream'.
Stacktrace:
System.ObjectDisposedException:
at Microsoft.AspNetCore.Server.IIS.Core.HttpResponseStream.ValidateState (Microsoft.AspNetCore.Server.IIS, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Server.IIS.Core.HttpResponseStream.WriteAsync (Microsoft.AspNetCore.Server.IIS, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Server.IIS.Core.WrappingStream.WriteAsync (Microsoft.AspNetCore.Server.IIS, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at KeypointConnect.Api.Policies.ValidTeamHandler+<>c__DisplayClass2_3+<<HandleRequirementAsync>b__5>d.MoveNext (KeypointConnect.Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\KeypointConnect.Api\Policies\ValidTeamHandler.cs:111)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext+<FireOnStarting>d__170.MoveNext (Microsoft.AspNetCore.Server.IIS, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
The stacktrace shows that it fails on this line:
await response.Body.WriteAsync(message);
The strange thing is that it works perfectly fine locally. Any ideas on how to fix this?
Thanks in advance!

Related

Various System.OperationCanceledException exceptions happening on my Azure Web API

I have a .Net 6 Web API hosted on Azure App Service.
When I look at the Application Insight logs, I see a number of 500 Exceptions of this type:
System.OperationCanceledException: The operation was canceled.
This happens on various API endpoints. The stack trace looks like this:
System.OperationCanceledException:
at System.Threading.CancellationToken.ThrowOperationCanceledException (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Threading.CancellationToken.ThrowIfCancellationRequested (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Threading.SemaphoreSlim+<WaitUntilCountOrTimeoutAsync>d__31.MoveNext (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.IdentityModel.Protocols.ConfigurationManager`1+<GetConfigurationAsync>d__24.MoveNext (Microsoft.IdentityModel.Protocols, Version=6.10.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler+<HandleAuthenticateAsync>d__6.MoveNext (Microsoft.AspNetCore.Authentication.JwtBearer, Version=6.0.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
I guess this is happening when the front-end (which is an Angular web app, but could also be Postman) makes a request to the API, and for whatever reason, cancels the request before it is complete.
Can I simply ignore these errors? Should I be handling them somehow? (And if I should, how should I do so). Is my API perhaps crashing (and restarting) every time this happens, as it is an unhandled error?
Thanks
EDIT: Here is the code in my Startup.cs regarding JwtBearer, which is implicated in the stack trace. You will notice there is a section that deals with a SignalR Core chat hub, which is used by my API:
public void ConfigureServices(IServiceCollection services)
{
services.AddAuthentication(options =>
{
options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme;
options.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme;
}).AddJwtBearer(options =>
{
options.Authority = $"https://{Configuration["Auth0:Domain"]}/";
options.Audience = Configuration["Auth0:Audience"];
options.Events = new JwtBearerEvents
{
OnMessageReceived = context =>
{
StringValues accessToken = context.Request.Query["access_token"];
// If the request is for our SignalR hub...
PathString path = context.HttpContext.Request.Path;
if (!string.IsNullOrEmpty(accessToken) &&
(path.StartsWithSegments("/chatHub")))
{
// Read the token out of the query string
context.Token = accessToken;
}
return Task.CompletedTask;
}
};
});

HttpClient, Azure and System.Net.Sockets.SocketException: what to do?

We are getting lots of problems on a Web App in Azure, which use HttpClient to make remote call to a endpoint.
I was aware it was a problem with outbounds connections limit in Azure, so I refactored the code using only one instance of HttpClient with singleton pattern.
Here's my helper:
public class HttpClientHelper
{
string ApiKey = "";
static HttpClient httpClient;
public HttpClientHelper(IConfiguration config)
{
var appsettings = config.GetSection("AppSettings");
ApiKey = appsettings.GetChildren().Where(x => x.Key == "URL").FirstOrDefault().Value;
if (httpClient == null)
{
httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.Add("Authorization", ApiKey);
}
}
public async Task<T> SingleRead<T>(string baseAddress, string url)
{
T returnValue = default(T);
try
{
var response = await httpClient.GetAsync($"{baseAddress}/{url}").ConfigureAwait(false);
if (response.IsSuccessStatusCode)
{
var r = await response.Content.ReadAsStringAsync();
returnValue = JsonConvert.DeserializeObject<T>(r);
}
else
{
Log($"SingleRead failed, error: {JsonConvert.SerializeObject(response)}");
}
}
catch (Exception ex)
{
Log(ex, $"SingleRead ex: {baseAddress}/{url}");
throw ex;
}
return returnValue;
}
public async Task<T> PostRead<T>(string baseAddress, string url, object entity = null)
{
T returnValue = default(T);
try
{
string json = JsonConvert.SerializeObject(entity);
var body = new StringContent(json, UnicodeEncoding.UTF8, "application/json");
var response = await httpClient.PostAsync($"{baseAddress}/{url}", body).ConfigureAwait(false);
if (response.IsSuccessStatusCode)
{
var r = await response.Content.ReadAsStringAsync();
returnValue = JsonConvert.DeserializeObject<T>(r);
}
else
{
Log($"PostRead failed, error: {JsonConvert.SerializeObject(response)}");
}
}
catch (Exception ex)
{
Log(ex, $"PostRead url: {baseAddress}/{url} entity: {JsonConvert.SerializeObject(entity)}");
throw ex;
}
return returnValue;
}
}
And than I do basic request calling SingleRead() (for GET) or PostRead() (for POST).
The problem is that still, sometimes, I get SocketException:
Here's the fullstack trace:
System.Net.Http.HttpRequestException:
at MobilityServer.Helpers.HttpClientHelper+<PostRead>d__4`1.MoveNext (MobilityServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: c:\MyProject\WebApp\Helpers\HttpClientHelper.cs:77)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at MobilityServer.Helpers.MigrationHelper+<GetLdcHeaders>d__8.MoveNext (MobilityServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: c:\MyProject\WebApp\Helpers\MigrationHelper.cs:54)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at MobilityServer.Controllers.AndroidController+<GetLdcHeaders>d__8.MoveNext (MobilityServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: c:\MyProject\WebApp\Controllers\BasicController.cs:104)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor+TaskOfIActionResultExecutor+<Execute>d__0.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeActionMethodAsync>d__12.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeNextActionFilterAsync>d__10.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeInnerFilterAsync>d__13.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeNextResourceFilter>d__23.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeFilterPipelineAsync>d__18.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeAsync>d__16.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Routing.EndpointMiddleware+<Invoke>d__3.MoveNext (Microsoft.AspNetCore.Routing, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Routing, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware+<Invoke>d__7.MoveNext (Microsoft.AspNetCore.Diagnostics, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Eccezione interna System.Net.Sockets.SocketException gestita in MobilityServer.Helpers.HttpClientHelper+<PostRead>d__4`1.MoveNext:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.ConnectHelper+<ConnectAsync>d__2.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
And this the Exception Stack Trace:
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at MobilityServer.Helpers.HttpClientHelper.PostRead[T](String baseAddress, String url, Object entity) in c:\MyProject\WebApp\Helpers\HttpClientHelper.cs:line 63
I don't get where the problem can be.
The outbound connections now are lower than the service plan (which is "S3 : 2").
SNAT seems way down the limit:
The same for the outbound connections:
(note: I've resized from 3 instances to 2 yesterday at 21:30, that's why those green "drop").
I really can't get what's going on. The environment sames correctly setup.
Any clues?
EDIT
Here's the version (i.e. the edits) to add the suggested IHttpClientFactory:
// Startup.cs
public void ConfigureServices(IServiceCollection services)
{
services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
services.AddApplicationInsightsTelemetry(Configuration["APPINSIGHTS_CONNECTIONSTRING"]);
services.AddHttpClient<MigrationHelper>();
}
public class MigrationHelper
{
HttpClientHelper httpc;
public MigrationHelper(IConfiguration config, HttpClient httpClient)
{
// ...
httpc = new HttpClientHelper(config, httpClient);
}
}
public class HttpClientHelper
{
string ApiKey = "";
HttpClient httpClient;
public HttpClientHelper(IConfiguration config, HttpClient _httpClient)
{
var appsettings = config.GetSection("AppSettings");
ApiKey = appsettings.GetChildren().Where(x => x.Key == "URL").FirstOrDefault().Value;
httpClient = _httpClient;
httpClient.DefaultRequestHeaders.Add("Authorization", ApiKey);
}
// ...
}
Still doesn't works, and I got the same error Socket messages. Here's a full example:
{
"Message": "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond",
"Data": {},
"InnerException": {
"ClassName": "System.Net.Sockets.SocketException",
"Message": "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond",
"Data": null,
"InnerException": null,
"HelpURL": null,
"StackTraceString": " at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)",
"RemoteStackTraceString": null,
"RemoteStackIndex": 0,
"ExceptionMethod": null,
"HResult": -2147467259,
"Source": "System.Private.CoreLib",
"WatsonBuckets": null,
"NativeErrorCode": 10060
},
"StackTrace": " at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)\r\n at System.Threading.Tasks.ValueTask`1.get_Result()\r\n at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Threading.Tasks.ValueTask`1.get_Result()\r\n at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)\r\n at System.Threading.Tasks.ValueTask`1.get_Result()\r\n at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)\r\n at MobilityServer.Helpers.HttpClientHelper.PostRead[T](String baseAddress, String url, Object entity) in c:\\MyProject\\WebApp\\Helpers\\HttpClientHelper.cs:line 60",
"HelpLink": null,
"Source": "System.Net.Http",
"HResult": -2147467259
}

Try 1: The maximum you can do in your app is to use IHttpClientFactory to implement resilient HTTP requests (see: https://learn.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests).
Try 2: When you scale out the endpoint (if it is in your control) there's a Load Balancer over all your instances, so, traffic is automatically load balanced between them. Maybe you need a better load balancing solution (see: https://learn.microsoft.com/en-us/azure/architecture/guide/technology-choices/load-balancing-overview).
Try 3: If your app works better on a single machine, you should scale up and use per-app scaling (see: https://learn.microsoft.com/en-us/azure/app-service/manage-scale-per-app)
Try 4: Troubleshooting intermittent outbound connection errors in Azure App Service (see: https://learn.microsoft.com/en-us/azure/app-service/troubleshoot-intermittent-outbound-connection-errors)
P.S.: a wild guess is that the endpoint could not handle the increasing requests from your 3 instances (an instance has a limit of 256 concurrent connections (per server endpoint) allowed when making requests using an HttpClient object set via HttpClientHandler.MaxConnectionsPerServer property); that's why when you decrease the number of instances it behaves better

Azure storage error: "An attempt was made to access a socket in a way forbidden by its access permissions."

The Problem:
I see there are a few questions with this error, but it doesn't solve my issue.
I have a Web API hosted in Azure web application (Dev/Test F1 Tier) that uploads files to Azure storage blob container and saves to Azure storage table, but when I do that, I get the following error:
"An attempt was made to access a socket in a way forbidden by its access permissions.". If anyone can help or has any idea where I can look, I'd appreciate it. Thank you.
I've Tried:
I have used Application Insights and attached debugger from VS 2019, full stack trace at the bottom.
Connection string (Note '...' to replace full AccountName and AccountKey):
"StorageConnectionString": "DefaultEndpointsProtocol=https;AccountName=s...001;AccountKey=TJi...nDkw==;EndpointSuffix=core.windows.net"
On the azure web application side I've done the following:
I've Allowed all CORS
I don't use any VNet's
I don't have any Access restrictions
I'm using the connection string taken from Access keys under Settings on Azure storage
On the Azure storage side, I've done the following:
Account kind: StorageV2 (general purpose v2)
I've Allowed all CORS
I don't use any VNet's
Allow access from is set to All Networks
Code used to upload files
BlobContainerClient container = new BlobContainerClient(_AppSettings.StorageConnectionString,
$"patient-notes-{patientid}");
container.CreateIfNotExists(PublicAccessType.Blob);
var blockBlob = container.GetBlobClient(fileName);
blockBlob.Upload(stream);
return blockBlob.Uri;
Code to table storage:
private readonly AppSettings _AppSettings;
private readonly CloudStorageAccount _storageAccount;
private readonly CloudTableClient _tableClient;
private readonly TelemetryClient _telemetry;
public ErrorLogTableStorage(IOptions<AppSettings> appSettings, TelemetryClient telemetry)
{
_telemetry = telemetry;
_AppSettings = appSettings?.Value;
_storageAccount = CloudStorageAccount.Parse(_AppSettings.AzureTableStorageConnectionString);
_tableClient = _storageAccount.CreateCloudTableClient();
CloudTable table = _tableClient.GetTableReference(AzureStorageTable.ErrorLog.ToString());
table.CreateIfNotExistsAsync();
}
public async Task CreateErrorLog(ErrorLogEntity errorLog)
{
try
{
CloudTable table = _tableClient.GetTableReference(AzureStorageTable.ErrorLog.ToString());
TableOperation insertOperation = TableOperation.InsertOrMerge(errorLog);
await table.ExecuteAsync(insertOperation);
}
catch (Exception ex)
{
_telemetry.TrackException(ex);
}
}
Stack Trace for the table storage:
Microsoft.Azure.Cosmos.Table.StorageException:
at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor+<ExecuteAsync>d__1`1.MoveNext (Microsoft.Azure.Cosmos.Table, Version=1.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Infrastructure.AzureTableStorage.ErrorLogTableStorage+<CreateErrorLog>d__5.MoveNext (Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullInfrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: AzureTableStorage\ErrorLogTableStorage.csInfrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 46)
Inner exception System.Net.Http.HttpRequestException handled at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor+<ExecuteAsync>d__1`1.MoveNext:
at System.Net.Http.ConnectHelper+<ConnectAsync>d__1.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<ConnectAsync>d__52.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<CreateHttp11ConnectionAsync>d__53.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<GetHttpConnectionAsync>d__45.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<SendWithRetryAsync>d__47.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.RedirectHandler+<SendAsync>d__4.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.DiagnosticsHandler+<SendAsync>d__2.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpClient+<FinishSendAsyncUnbuffered>d__71.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor+<ExecuteAsync>d__1`1.MoveNext (Microsoft.Azure.Cosmos.Table, Version=1.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
Inner exception System.Net.Sockets.SocketException handled at System.Net.Http.ConnectHelper+<ConnectAsync>d__1.MoveNext:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.ConnectHelper+<ConnectAsync>d__1.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)

Okay, I got this to work, I had to add AppSettings: to the application setting.
change from this application setting
to this
But why do I need to add AppSettings:?
What's the correct way of doing this?

canceled while creating documents in cosmosdb

We are using cosmosdb to log our application logs. Our logs have two one layer have general info and other have more detailed information of the general log. We also log a copy of these in other two collections for backup. So we have completely four collections. In our code we are using Async CreateDocumentAsync to create these logs. We are storing the Task in a list and using Task.WaitAll to complete all the tasks. In dev environment we did not see any errors but in production we are seeing canceled errors in our application insights when creating new documents. sample picture below.
Below is the code structure we are using
API Controller code
public void Log(LogModel logObject)
{
try
{
_context.Save(logObject);
}
catch(Exception ex)
{
_azureTelemtry.TrackTrace($"Exception occured: {ex.ToString()}");
}
}
Context save method
public void Save(LogModel logObject)
{
List<Task> saveTasks = new List<Task>();
LogInfo logInfo = logObject.logInfo;
LogDetails logDetails = logObject.logDetails;
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, primaryLogInfoCollectionID), logInfo));
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, archieveLogInfoCollectionID), logInfo));
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, primaryLogDetailsCollectionID), logDetails));
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, archieveLogDetailsCollectionID), logDetails));
Task.WaitAll(saveTasks.ToArrays());
}
Exception Details
System.AggregateException: at
MyApplication.Logging.Controllers.ApiController.Log
(MyApplication.Logging, Version=2.0.0.0, Culture=neutral,
PublicKeyToken=nullMyApplication.Logging, Version=2.0.0.0,
Culture=neutral, PublicKeyToken=null:
C:\TFS\CA\Logging\Release\Release
3.0-Hotfix\Logging\Controllers\ApiController.csMyApplication.Logging, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 57) at
lambda_method (Anonymously Hosted DynamicMethods Assembly,
Version=0.0.0.0, Culture=neutral, PublicKeyToken=null) at
Microsoft.Extensions.Internal.ObjectMethodExecutor+<>c__DisplayClass33_0.b__0
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+d__12.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+d__10.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+d__14.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+d__22.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+d__17.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+d__15.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Builder.RouterMiddleware+d__4.MoveNext
(Microsoft.AspNetCore.Routing, Version=2.0.1.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware+d__11.MoveNext
(Microsoft.AspNetCore.Server.IISIntegration, Version=2.0.1.0,
Culture=neutral, PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware+d__3.MoveNext
(Microsoft.AspNetCore.Hosting, Version=2.0.1.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Frame`1+d__2.MoveNext
(Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.0.1.0,
Culture=neutral, PublicKeyToken=adb9793829ddae60) Inner exception
System.Threading.Tasks.TaskCanceledException handled at
MyApplication.Logging.Controllers.ApiController.Log:

You do not await tasks. Besides Save function is synchronous thus Task.WaitAll(saveTasks.ToArrays()); runs synchronpusly.
At least I expect that public void Save(LogModel logObject) would be public Task Save(LogModel logObject) and Task.WaitAll(saveTasks.ToArrays()); would be return Task.WhenAll(saveTasks.ToArrays());
public Task Save(LogModel logObject)
{
List<Task> saveTasks = new List<Task>();
LogInfo logInfo = logObject.logInfo;
LogDetails logDetails = logObject.logDetails;
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, primaryLogInfoCollectionID), logInfo));
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, archieveLogInfoCollectionID), logInfo));
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, primaryLogDetailsCollectionID), logDetails));
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, archieveLogDetailsCollectionID), logDetails));
return Task.WhenAll(saveTasks.ToArrays());
}
if your intention is to fire task and forget, now you can use _context.Save(logObject); but otherwise you need to use Task
another problem is: above code opens 4 tasks and process it uncontrollably. The default Concurrent Execution Limit in .Net Core is 10. Now think that Save method called three times one after another and none is finished yet. This means that you need to have 12 concurrent tasks. But .Net Core has limit to 10. Which I believe the last two tasks will be cancelled.

Writing to CloudAppendBlob: The remote server returned an error: (409) Conflict & ProcessExpectedStatusCodeNoException

I am trying to understand what causes the The remote server returned an error: (409) Conflict Exception.
I am writing lines to an AppendBlob by acquiring a lease, appending the text, and breaking the lease, like so. The error only happens every now and then, so I am curious what scenario would cause it to happen. My suspicion would be something may happen if multiple users hit the endpoint where this code executes and somehow to a call to acquire a lease happens at the same time.
// Get the AppendBlob reference
CloudAppendBlob blob = ….
// Acquire a lease on the blob until a call to breaking the lease is made
string leaseId = Guid.NewGuid().ToString();
await blob.AcquireLeaseAsync(null, leaseId);
// Append a new entry to the blob
await blob.AppendTextAsync(
textToBeAppended,
Encoding.UTF8,
new AccessCondition { LeaseId = leaseId },
new BlobRequestOptions(),
new OperationContext());
// break the lease on the blob
await blob.BreakLeaseAsync(null);
The Failed Method is Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException.
I see from telemetry that a call to PUT on the blob initially fails with the 409 status code and the exception above, but immediately after a successful PUT call with a 202 result code occurs. I'm assume Azure Storage is doing a retry, however I do not see the line of text in the file it was supposed to be appended to.
Here is the full call stack if that is helpful, I am not sure what the meaning of ProcessExpectedStatusCodeNoException is from this stack trace.
Microsoft.WindowsAzure.Storage.StorageException: at
Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync
(Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35Microsoft.WindowsAzure.Storage,
Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35:
c:\Program Files
(x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.csMicrosoft.WindowsAzure.Storage,
Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 50)
at
Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions+<>c__DisplayClass2`1.b__0
(Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35Microsoft.WindowsAzure.Storage,
Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35:
c:\Program Files
(x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.csMicrosoft.WindowsAzure.Storage,
Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 69)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(mscorlib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(mscorlib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089) at
Microsoft.Community.Data.Providers.BlobStorageProvider+d__1.MoveNext
(Microsoft.Community.Data, Version=1.0.0.0, Culture=neutral,
PublicKeyToken=nullMicrosoft.Community.Data, Version=1.0.0.0,
Culture=neutral, PublicKeyToken=null:
C:\BA\813\s\Microsoft.Community\Microsoft.Community.Data\Providers\BlobStorageProvider.csMicrosoft.Community.Data,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 67) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(mscorlib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089) at
Microsoft.Community.Data.Providers.BlobStorageProvider+d__1.MoveNext
(Microsoft.Community.Data, Version=1.0.0.0, Culture=neutral,
PublicKeyToken=nullMicrosoft.Community.Data, Version=1.0.0.0,
Culture=neutral, PublicKeyToken=null:
C:\BA\813\s\Microsoft.Community\Microsoft.Community.Data\Providers\BlobStorageProvider.csMicrosoft.Community.Data,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 72) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(mscorlib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(mscorlib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089) at
Microsoft.Community.Web.Repositories.AzureMLRecommendationsRepository+d__6.MoveNext
(Microsoft.Community.Web, Version=1.0.0.0, Culture=neutral,
PublicKeyToken=nullMicrosoft.Community.Web, Version=1.0.0.0,
Culture=neutral, PublicKeyToken=null:
C:\BA\813\s\Microsoft.Community\Microsoft.Community.Web\Repositories\AzureMLRecommendationsRepository.csMicrosoft.Community.Web,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 95)
Inner exception System.Net.WebException handled at
Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync:
at
Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException
(Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35Microsoft.WindowsAzure.Storage,
Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35:
c:\Program Files
(x86)\Jenkins\workspace\release_dotnet_master\Lib\Common\Shared\Protocol\HttpResponseParsers.Common.csMicrosoft.WindowsAzure.Storage,
Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 54)
at
Microsoft.WindowsAzure.Storage.Blob.CloudBlob+<>c__DisplayClass39.b__38
(Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35Microsoft.WindowsAzure.Storage,
Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35:
c:\Program Files
(x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlob.csMicrosoft.WindowsAzure.Storage,
Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35:
3830) at
Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse
(Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35Microsoft.WindowsAzure.Storage,
Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35:
c:\Program Files
(x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.csMicrosoft.WindowsAzure.Storage,
Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35:
299)

I'm wondering why you need to lease the blob and then append the text to the blob. If there are multiple writers simultaneously, it's EXPECTED that you'll meet (409) Conflict in your code.
As you can see in the official doc, acquiring lease (B) when a blob is still being leased by lease (A) will encounter 409 error:
Basically, blob lease is to achieve exclusive write access on a blob, it's inappropriate to use blob lease if you want to append against a blob from multiple writers simultaneously.
What are you afraid of if the AcquireLease and BreakLease operations are removed from your code? To be honest, I don't see any harm.

Sounds like an issue with with concurrency possibly:
Breaks the lease, if the blob has an active lease. Once a lease is broken, it cannot be renewed. Any authorized request can break the lease; the request is not required to specify a matching lease ID. When a lease is broken, the lease break period is allowed to elapse, during which time no lease operation except break and release can be performed on the blob. When a lease is successfully broken, the response indicates the interval in seconds until a new lease can be acquired.
https://learn.microsoft.com/en-us/rest/api/storageservices/lease-blob#outcomes-of-use-attempts-on-blobs-by-lease-state
If you look at link above you will see a huge table on how 409s can occur.
Can you run a fiddler to get the trace on inner exception?

Categories