Related
We're currently building an ASP.NET Web API and have implemented some custom authorization policies.
One of these policies checks whether the user has the 'Supervisor'-role within his team.
If not, we want to return a 403 error with a custom message.
This is how we have it now:
internal sealed class ValidTeamHandler : AuthorizationHandler<ValidTeamRequirement>
{
protected override async Task HandleRequirementAsync(AuthorizationHandlerContext context, ValidTeamRequirement requirement)
{
if (context.Resource is not DefaultHttpContext filterContext)
{
return;
}
var response = filterContext.HttpContext.Response;
// [...] removed for brevity
if (requirement.RequireTeamSupervisor)
{
var user = team.TeamUsers!.Single(tu => tu!.UserId.ToString() == userId.Value);
if (user!.Role != DDD.Enums.TeamRole.Supervisor)
{
context.Fail();
byte[] message = Encoding.UTF8.GetBytes("Only team supervisors are allowed to perform this action.");
response.OnStarting(async () =>
{
filterContext.HttpContext.Response.StatusCode = 403;
filterContext.HttpContext.Response.ContentType = "application/json";
await response.Body.WriteAsync(message);
});
}
}
}
}
We have some integration tests implemented that test this policy. When running these tests locally, we have no issues and everything works how it's supposed to work.
However, when we deploy the API to Azure (App Service), we get the following error whilst running these same tests:
Cannot write to the response body, the response has completed. Object name: 'HttpResponseStream'.
Stacktrace:
System.ObjectDisposedException:
at Microsoft.AspNetCore.Server.IIS.Core.HttpResponseStream.ValidateState (Microsoft.AspNetCore.Server.IIS, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Server.IIS.Core.HttpResponseStream.WriteAsync (Microsoft.AspNetCore.Server.IIS, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Server.IIS.Core.WrappingStream.WriteAsync (Microsoft.AspNetCore.Server.IIS, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at KeypointConnect.Api.Policies.ValidTeamHandler+<>c__DisplayClass2_3+<<HandleRequirementAsync>b__5>d.MoveNext (KeypointConnect.Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\KeypointConnect.Api\Policies\ValidTeamHandler.cs:111)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext+<FireOnStarting>d__170.MoveNext (Microsoft.AspNetCore.Server.IIS, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
The stacktrace shows that it fails on this line:
await response.Body.WriteAsync(message);
The strange thing is that it works perfectly fine locally. Any ideas on how to fix this?
Thanks in advance!
We recently migrated one project from Azure Cloud Services (classic) to Service Fabric. It is a ASP.NET WebAPI. Since then we are getting a lot of SocketException in the Deserialize method . The code looks like:
using (var decompressionStream = new GZipStream(Request.Body, CompressionMode.Decompress))
{
using (var sr = new StreamReader(decompressionStream))
{
// https://www.newtonsoft.com/json/help/html/Performance.htm
using (var jsonReader = new JsonTextReader(sr))
{
model= Serializer.Deserialize<Model>(jsonReader);
}
}
}
The exception is:
Message:
An established connection was aborted by the software in your host machine
Sometime we get:
The read operation failed, see inner exception. One or more errors occurred. An existing connection was forcibly closed by the remote host An existing connection was forcibly closed by the remote host.
Exception Stack:
System.IO.IOException:
at System.Net.Security._SslStream.EndRead (System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Threading.Tasks.TaskFactory`1+FromAsyncTrimPromise`1.Complete (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.AdaptedPipeline+<ReadInputAsync>d__18.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe.GetReadResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe.GetReadAsyncResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody+<PumpAsync>d__4.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe.GetReadResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe.GetReadAsyncResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody+<ReadAsync>d__27.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream+<ReadAsyncInternal>d__21.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.Read (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.IO.Compression.DeflateStream.Read (System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.IO.StreamReader.ReadBuffer (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.IO.StreamReader.Read (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Newtonsoft.Json.JsonTextReader.ReadData (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.JsonTextReader.ParseValue (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.JsonWriter.WriteToken (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.JsonWriter.WriteToken (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateJToken (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.JsonSerializer.Deserialize (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Controller.Post (Service, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: F:\Controller.cs:80)
Inner exception System.AggregateException handled at System.Net.Security._SslStream.EndRead:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.RawStream.EndRead (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Net.FixedSizeReader.ReadCallback (System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
Inner exception Microsoft.AspNetCore.Connections.ConnectionResetException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe.GetReadResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe.GetReadAsyncResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.RawStream+<ReadAsyncInternal>d__22.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Inner exception System.Net.Sockets.SocketException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.<GetResult>g__ThrowSocketException|7_0 (Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.GetResult (Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection+<ProcessReceives>d__24.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection+<DoReceive>d__23.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Earlier the code in Classic Service used to be like this and used to work fine:
Model model;
try
{
using (var sr = await OpenDecompressedBodyAsync(request.Content))
{
using (var jsonReader = new JsonTextReader(sr))
{
model = Serializer.Deserialize<Model>(jsonReader);
}
}
}
private async Task<StreamReader> OpenDecompressedBodyAsync(HttpContent requestContent)
{
var body = await requestContent.ReadAsStreamAsync();
var gzip = new GZipStream(body, CompressionMode.Decompress);
var sr = new StreamReader(gzip);
return sr;
}
We have tried a lot of things, however, the exception still exists. Also, it is not too common. Happens once in a while. The data volume is very high and each Request.Body can have upto 2-3MB of data. Can someone help why we are getting SocketException?
Try to disable firewall/antivirus, and test.
If it works. it means you need to allow specific port or create a new rule for this.
We are getting lots of problems on a Web App in Azure, which use HttpClient to make remote call to a endpoint.
I was aware it was a problem with outbounds connections limit in Azure, so I refactored the code using only one instance of HttpClient with singleton pattern.
Here's my helper:
public class HttpClientHelper
{
string ApiKey = "";
static HttpClient httpClient;
public HttpClientHelper(IConfiguration config)
{
var appsettings = config.GetSection("AppSettings");
ApiKey = appsettings.GetChildren().Where(x => x.Key == "URL").FirstOrDefault().Value;
if (httpClient == null)
{
httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.Add("Authorization", ApiKey);
}
}
public async Task<T> SingleRead<T>(string baseAddress, string url)
{
T returnValue = default(T);
try
{
var response = await httpClient.GetAsync($"{baseAddress}/{url}").ConfigureAwait(false);
if (response.IsSuccessStatusCode)
{
var r = await response.Content.ReadAsStringAsync();
returnValue = JsonConvert.DeserializeObject<T>(r);
}
else
{
Log($"SingleRead failed, error: {JsonConvert.SerializeObject(response)}");
}
}
catch (Exception ex)
{
Log(ex, $"SingleRead ex: {baseAddress}/{url}");
throw ex;
}
return returnValue;
}
public async Task<T> PostRead<T>(string baseAddress, string url, object entity = null)
{
T returnValue = default(T);
try
{
string json = JsonConvert.SerializeObject(entity);
var body = new StringContent(json, UnicodeEncoding.UTF8, "application/json");
var response = await httpClient.PostAsync($"{baseAddress}/{url}", body).ConfigureAwait(false);
if (response.IsSuccessStatusCode)
{
var r = await response.Content.ReadAsStringAsync();
returnValue = JsonConvert.DeserializeObject<T>(r);
}
else
{
Log($"PostRead failed, error: {JsonConvert.SerializeObject(response)}");
}
}
catch (Exception ex)
{
Log(ex, $"PostRead url: {baseAddress}/{url} entity: {JsonConvert.SerializeObject(entity)}");
throw ex;
}
return returnValue;
}
}
And than I do basic request calling SingleRead() (for GET) or PostRead() (for POST).
The problem is that still, sometimes, I get SocketException:
Here's the fullstack trace:
System.Net.Http.HttpRequestException:
at MobilityServer.Helpers.HttpClientHelper+<PostRead>d__4`1.MoveNext (MobilityServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: c:\MyProject\WebApp\Helpers\HttpClientHelper.cs:77)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at MobilityServer.Helpers.MigrationHelper+<GetLdcHeaders>d__8.MoveNext (MobilityServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: c:\MyProject\WebApp\Helpers\MigrationHelper.cs:54)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at MobilityServer.Controllers.AndroidController+<GetLdcHeaders>d__8.MoveNext (MobilityServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: c:\MyProject\WebApp\Controllers\BasicController.cs:104)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor+TaskOfIActionResultExecutor+<Execute>d__0.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeActionMethodAsync>d__12.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeNextActionFilterAsync>d__10.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeInnerFilterAsync>d__13.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeNextResourceFilter>d__23.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeFilterPipelineAsync>d__18.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeAsync>d__16.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Routing.EndpointMiddleware+<Invoke>d__3.MoveNext (Microsoft.AspNetCore.Routing, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Routing, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware+<Invoke>d__7.MoveNext (Microsoft.AspNetCore.Diagnostics, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Eccezione interna System.Net.Sockets.SocketException gestita in MobilityServer.Helpers.HttpClientHelper+<PostRead>d__4`1.MoveNext:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.ConnectHelper+<ConnectAsync>d__2.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
And this the Exception Stack Trace:
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at MobilityServer.Helpers.HttpClientHelper.PostRead[T](String baseAddress, String url, Object entity) in c:\MyProject\WebApp\Helpers\HttpClientHelper.cs:line 63
I don't get where the problem can be.
The outbound connections now are lower than the service plan (which is "S3 : 2").
SNAT seems way down the limit:
The same for the outbound connections:
(note: I've resized from 3 instances to 2 yesterday at 21:30, that's why those green "drop").
I really can't get what's going on. The environment sames correctly setup.
Any clues?
EDIT
Here's the version (i.e. the edits) to add the suggested IHttpClientFactory:
// Startup.cs
public void ConfigureServices(IServiceCollection services)
{
services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
services.AddApplicationInsightsTelemetry(Configuration["APPINSIGHTS_CONNECTIONSTRING"]);
services.AddHttpClient<MigrationHelper>();
}
public class MigrationHelper
{
HttpClientHelper httpc;
public MigrationHelper(IConfiguration config, HttpClient httpClient)
{
// ...
httpc = new HttpClientHelper(config, httpClient);
}
}
public class HttpClientHelper
{
string ApiKey = "";
HttpClient httpClient;
public HttpClientHelper(IConfiguration config, HttpClient _httpClient)
{
var appsettings = config.GetSection("AppSettings");
ApiKey = appsettings.GetChildren().Where(x => x.Key == "URL").FirstOrDefault().Value;
httpClient = _httpClient;
httpClient.DefaultRequestHeaders.Add("Authorization", ApiKey);
}
// ...
}
Still doesn't works, and I got the same error Socket messages. Here's a full example:
{
"Message": "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond",
"Data": {},
"InnerException": {
"ClassName": "System.Net.Sockets.SocketException",
"Message": "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond",
"Data": null,
"InnerException": null,
"HelpURL": null,
"StackTraceString": " at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)",
"RemoteStackTraceString": null,
"RemoteStackIndex": 0,
"ExceptionMethod": null,
"HResult": -2147467259,
"Source": "System.Private.CoreLib",
"WatsonBuckets": null,
"NativeErrorCode": 10060
},
"StackTrace": " at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)\r\n at System.Threading.Tasks.ValueTask`1.get_Result()\r\n at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Threading.Tasks.ValueTask`1.get_Result()\r\n at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)\r\n at System.Threading.Tasks.ValueTask`1.get_Result()\r\n at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)\r\n at MobilityServer.Helpers.HttpClientHelper.PostRead[T](String baseAddress, String url, Object entity) in c:\\MyProject\\WebApp\\Helpers\\HttpClientHelper.cs:line 60",
"HelpLink": null,
"Source": "System.Net.Http",
"HResult": -2147467259
}
Try 1: The maximum you can do in your app is to use IHttpClientFactory to implement resilient HTTP requests (see: https://learn.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests).
Try 2: When you scale out the endpoint (if it is in your control) there's a Load Balancer over all your instances, so, traffic is automatically load balanced between them. Maybe you need a better load balancing solution (see: https://learn.microsoft.com/en-us/azure/architecture/guide/technology-choices/load-balancing-overview).
Try 3: If your app works better on a single machine, you should scale up and use per-app scaling (see: https://learn.microsoft.com/en-us/azure/app-service/manage-scale-per-app)
Try 4: Troubleshooting intermittent outbound connection errors in Azure App Service (see: https://learn.microsoft.com/en-us/azure/app-service/troubleshoot-intermittent-outbound-connection-errors)
P.S.: a wild guess is that the endpoint could not handle the increasing requests from your 3 instances (an instance has a limit of 256 concurrent connections (per server endpoint) allowed when making requests using an HttpClient object set via HttpClientHandler.MaxConnectionsPerServer property); that's why when you decrease the number of instances it behaves better
The Problem:
I see there are a few questions with this error, but it doesn't solve my issue.
I have a Web API hosted in Azure web application (Dev/Test F1 Tier) that uploads files to Azure storage blob container and saves to Azure storage table, but when I do that, I get the following error:
"An attempt was made to access a socket in a way forbidden by its access permissions.". If anyone can help or has any idea where I can look, I'd appreciate it. Thank you.
I've Tried:
I have used Application Insights and attached debugger from VS 2019, full stack trace at the bottom.
Connection string (Note '...' to replace full AccountName and AccountKey):
"StorageConnectionString": "DefaultEndpointsProtocol=https;AccountName=s...001;AccountKey=TJi...nDkw==;EndpointSuffix=core.windows.net"
On the azure web application side I've done the following:
I've Allowed all CORS
I don't use any VNet's
I don't have any Access restrictions
I'm using the connection string taken from Access keys under Settings on Azure storage
On the Azure storage side, I've done the following:
Account kind: StorageV2 (general purpose v2)
I've Allowed all CORS
I don't use any VNet's
Allow access from is set to All Networks
Code used to upload files
BlobContainerClient container = new BlobContainerClient(_AppSettings.StorageConnectionString,
$"patient-notes-{patientid}");
container.CreateIfNotExists(PublicAccessType.Blob);
var blockBlob = container.GetBlobClient(fileName);
blockBlob.Upload(stream);
return blockBlob.Uri;
Code to table storage:
private readonly AppSettings _AppSettings;
private readonly CloudStorageAccount _storageAccount;
private readonly CloudTableClient _tableClient;
private readonly TelemetryClient _telemetry;
public ErrorLogTableStorage(IOptions<AppSettings> appSettings, TelemetryClient telemetry)
{
_telemetry = telemetry;
_AppSettings = appSettings?.Value;
_storageAccount = CloudStorageAccount.Parse(_AppSettings.AzureTableStorageConnectionString);
_tableClient = _storageAccount.CreateCloudTableClient();
CloudTable table = _tableClient.GetTableReference(AzureStorageTable.ErrorLog.ToString());
table.CreateIfNotExistsAsync();
}
public async Task CreateErrorLog(ErrorLogEntity errorLog)
{
try
{
CloudTable table = _tableClient.GetTableReference(AzureStorageTable.ErrorLog.ToString());
TableOperation insertOperation = TableOperation.InsertOrMerge(errorLog);
await table.ExecuteAsync(insertOperation);
}
catch (Exception ex)
{
_telemetry.TrackException(ex);
}
}
Stack Trace for the table storage:
Microsoft.Azure.Cosmos.Table.StorageException:
at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor+<ExecuteAsync>d__1`1.MoveNext (Microsoft.Azure.Cosmos.Table, Version=1.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Infrastructure.AzureTableStorage.ErrorLogTableStorage+<CreateErrorLog>d__5.MoveNext (Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullInfrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: AzureTableStorage\ErrorLogTableStorage.csInfrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 46)
Inner exception System.Net.Http.HttpRequestException handled at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor+<ExecuteAsync>d__1`1.MoveNext:
at System.Net.Http.ConnectHelper+<ConnectAsync>d__1.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<ConnectAsync>d__52.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<CreateHttp11ConnectionAsync>d__53.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<GetHttpConnectionAsync>d__45.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<SendWithRetryAsync>d__47.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.RedirectHandler+<SendAsync>d__4.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.DiagnosticsHandler+<SendAsync>d__2.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpClient+<FinishSendAsyncUnbuffered>d__71.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor+<ExecuteAsync>d__1`1.MoveNext (Microsoft.Azure.Cosmos.Table, Version=1.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
Inner exception System.Net.Sockets.SocketException handled at System.Net.Http.ConnectHelper+<ConnectAsync>d__1.MoveNext:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.ConnectHelper+<ConnectAsync>d__1.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Okay, I got this to work, I had to add AppSettings: to the application setting.
change from this application setting
to this
But why do I need to add AppSettings:?
What's the correct way of doing this?
We are using cosmosdb to log our application logs. Our logs have two one layer have general info and other have more detailed information of the general log. We also log a copy of these in other two collections for backup. So we have completely four collections. In our code we are using Async CreateDocumentAsync to create these logs. We are storing the Task in a list and using Task.WaitAll to complete all the tasks. In dev environment we did not see any errors but in production we are seeing canceled errors in our application insights when creating new documents. sample picture below.
Below is the code structure we are using
API Controller code
public void Log(LogModel logObject)
{
try
{
_context.Save(logObject);
}
catch(Exception ex)
{
_azureTelemtry.TrackTrace($"Exception occured: {ex.ToString()}");
}
}
Context save method
public void Save(LogModel logObject)
{
List<Task> saveTasks = new List<Task>();
LogInfo logInfo = logObject.logInfo;
LogDetails logDetails = logObject.logDetails;
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, primaryLogInfoCollectionID), logInfo));
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, archieveLogInfoCollectionID), logInfo));
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, primaryLogDetailsCollectionID), logDetails));
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, archieveLogDetailsCollectionID), logDetails));
Task.WaitAll(saveTasks.ToArrays());
}
Exception Details
System.AggregateException: at
MyApplication.Logging.Controllers.ApiController.Log
(MyApplication.Logging, Version=2.0.0.0, Culture=neutral,
PublicKeyToken=nullMyApplication.Logging, Version=2.0.0.0,
Culture=neutral, PublicKeyToken=null:
C:\TFS\CA\Logging\Release\Release
3.0-Hotfix\Logging\Controllers\ApiController.csMyApplication.Logging, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 57) at
lambda_method (Anonymously Hosted DynamicMethods Assembly,
Version=0.0.0.0, Culture=neutral, PublicKeyToken=null) at
Microsoft.Extensions.Internal.ObjectMethodExecutor+<>c__DisplayClass33_0.b__0
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+d__12.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+d__10.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+d__14.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+d__22.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+d__17.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+d__15.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Builder.RouterMiddleware+d__4.MoveNext
(Microsoft.AspNetCore.Routing, Version=2.0.1.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware+d__11.MoveNext
(Microsoft.AspNetCore.Server.IISIntegration, Version=2.0.1.0,
Culture=neutral, PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware+d__3.MoveNext
(Microsoft.AspNetCore.Hosting, Version=2.0.1.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60) at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e) at
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Frame`1+d__2.MoveNext
(Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.0.1.0,
Culture=neutral, PublicKeyToken=adb9793829ddae60) Inner exception
System.Threading.Tasks.TaskCanceledException handled at
MyApplication.Logging.Controllers.ApiController.Log:
You do not await tasks. Besides Save function is synchronous thus Task.WaitAll(saveTasks.ToArrays()); runs synchronpusly.
At least I expect that public void Save(LogModel logObject) would be public Task Save(LogModel logObject) and Task.WaitAll(saveTasks.ToArrays()); would be return Task.WhenAll(saveTasks.ToArrays());
public Task Save(LogModel logObject)
{
List<Task> saveTasks = new List<Task>();
LogInfo logInfo = logObject.logInfo;
LogDetails logDetails = logObject.logDetails;
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, primaryLogInfoCollectionID), logInfo));
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, archieveLogInfoCollectionID), logInfo));
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, primaryLogDetailsCollectionID), logDetails));
saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, archieveLogDetailsCollectionID), logDetails));
return Task.WhenAll(saveTasks.ToArrays());
}
if your intention is to fire task and forget, now you can use _context.Save(logObject); but otherwise you need to use Task
another problem is: above code opens 4 tasks and process it uncontrollably. The default Concurrent Execution Limit in .Net Core is 10. Now think that Save method called three times one after another and none is finished yet. This means that you need to have 12 concurrent tasks. But .Net Core has limit to 10. Which I believe the last two tasks will be cancelled.