We Keep Coding

SocketException while reading data from Request.Body in ASP.NET Core

We recently migrated one project from Azure Cloud Services (classic) to Service Fabric. It is a ASP.NET WebAPI. Since then we are getting a lot of SocketException in the Deserialize method . The code looks like:
using (var decompressionStream = new GZipStream(Request.Body, CompressionMode.Decompress))
{
using (var sr = new StreamReader(decompressionStream))
{
// https://www.newtonsoft.com/json/help/html/Performance.htm
using (var jsonReader = new JsonTextReader(sr))
{
model= Serializer.Deserialize<Model>(jsonReader);
}
}
}
The exception is:
Message:
An established connection was aborted by the software in your host machine
Sometime we get:
The read operation failed, see inner exception. One or more errors occurred. An existing connection was forcibly closed by the remote host An existing connection was forcibly closed by the remote host.
Exception Stack:
System.IO.IOException:
at System.Net.Security._SslStream.EndRead (System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Threading.Tasks.TaskFactory`1+FromAsyncTrimPromise`1.Complete (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.AdaptedPipeline+<ReadInputAsync>d__18.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe.GetReadResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe.GetReadAsyncResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody+<PumpAsync>d__4.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe.GetReadResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe.GetReadAsyncResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody+<ReadAsync>d__27.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream+<ReadAsyncInternal>d__21.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.Read (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.IO.Compression.DeflateStream.Read (System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.IO.StreamReader.ReadBuffer (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.IO.StreamReader.Read (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Newtonsoft.Json.JsonTextReader.ReadData (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.JsonTextReader.ParseValue (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.JsonWriter.WriteToken (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.JsonWriter.WriteToken (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateJToken (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Newtonsoft.Json.JsonSerializer.Deserialize (Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
at Controller.Post (Service, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: F:\Controller.cs:80)
Inner exception System.AggregateException handled at System.Net.Security._SslStream.EndRead:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.RawStream.EndRead (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Net.FixedSizeReader.ReadCallback (System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
Inner exception Microsoft.AspNetCore.Connections.ConnectionResetException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe.GetReadResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe.GetReadAsyncResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.RawStream+<ReadAsyncInternal>d__22.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Inner exception System.Net.Sockets.SocketException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.<GetResult>g__ThrowSocketException|7_0 (Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.GetResult (Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection+<ProcessReceives>d__24.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection+<DoReceive>d__23.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Earlier the code in Classic Service used to be like this and used to work fine:
Model model;
try
{
using (var sr = await OpenDecompressedBodyAsync(request.Content))
{
using (var jsonReader = new JsonTextReader(sr))
{
model = Serializer.Deserialize<Model>(jsonReader);
}
}
}
private async Task<StreamReader> OpenDecompressedBodyAsync(HttpContent requestContent)
{
var body = await requestContent.ReadAsStreamAsync();
var gzip = new GZipStream(body, CompressionMode.Decompress);
var sr = new StreamReader(gzip);
return sr;
}
We have tried a lot of things, however, the exception still exists. Also, it is not too common. Happens once in a while. The data volume is very high and each Request.Body can have upto 2-3MB of data. Can someone help why we are getting SocketException?

Try to disable firewall/antivirus, and test.
If it works. it means you need to allow specific port or create a new rule for this.

HttpClient, Azure and System.Net.Sockets.SocketException: what to do?

We are getting lots of problems on a Web App in Azure, which use HttpClient to make remote call to a endpoint.
I was aware it was a problem with outbounds connections limit in Azure, so I refactored the code using only one instance of HttpClient with singleton pattern.
Here's my helper:
public class HttpClientHelper
{
string ApiKey = "";
static HttpClient httpClient;
public HttpClientHelper(IConfiguration config)
{
var appsettings = config.GetSection("AppSettings");
ApiKey = appsettings.GetChildren().Where(x => x.Key == "URL").FirstOrDefault().Value;
if (httpClient == null)
{
httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.Add("Authorization", ApiKey);
}
}
public async Task<T> SingleRead<T>(string baseAddress, string url)
{
T returnValue = default(T);
try
{
var response = await httpClient.GetAsync($"{baseAddress}/{url}").ConfigureAwait(false);
if (response.IsSuccessStatusCode)
{
var r = await response.Content.ReadAsStringAsync();
returnValue = JsonConvert.DeserializeObject<T>(r);
}
else
{
Log($"SingleRead failed, error: {JsonConvert.SerializeObject(response)}");
}
}
catch (Exception ex)
{
Log(ex, $"SingleRead ex: {baseAddress}/{url}");
throw ex;
}
return returnValue;
}
public async Task<T> PostRead<T>(string baseAddress, string url, object entity = null)
{
T returnValue = default(T);
try
{
string json = JsonConvert.SerializeObject(entity);
var body = new StringContent(json, UnicodeEncoding.UTF8, "application/json");
var response = await httpClient.PostAsync($"{baseAddress}/{url}", body).ConfigureAwait(false);
if (response.IsSuccessStatusCode)
{
var r = await response.Content.ReadAsStringAsync();
returnValue = JsonConvert.DeserializeObject<T>(r);
}
else
{
Log($"PostRead failed, error: {JsonConvert.SerializeObject(response)}");
}
}
catch (Exception ex)
{
Log(ex, $"PostRead url: {baseAddress}/{url} entity: {JsonConvert.SerializeObject(entity)}");
throw ex;
}
return returnValue;
}
}
And than I do basic request calling SingleRead() (for GET) or PostRead() (for POST).
The problem is that still, sometimes, I get SocketException:
Here's the fullstack trace:
System.Net.Http.HttpRequestException:
at MobilityServer.Helpers.HttpClientHelper+<PostRead>d__4`1.MoveNext (MobilityServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: c:\MyProject\WebApp\Helpers\HttpClientHelper.cs:77)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at MobilityServer.Helpers.MigrationHelper+<GetLdcHeaders>d__8.MoveNext (MobilityServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: c:\MyProject\WebApp\Helpers\MigrationHelper.cs:54)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at MobilityServer.Controllers.AndroidController+<GetLdcHeaders>d__8.MoveNext (MobilityServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: c:\MyProject\WebApp\Controllers\BasicController.cs:104)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor+TaskOfIActionResultExecutor+<Execute>d__0.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeActionMethodAsync>d__12.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeNextActionFilterAsync>d__10.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeInnerFilterAsync>d__13.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeNextResourceFilter>d__23.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeFilterPipelineAsync>d__18.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeAsync>d__16.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Routing.EndpointMiddleware+<Invoke>d__3.MoveNext (Microsoft.AspNetCore.Routing, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Routing, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware+<Invoke>d__7.MoveNext (Microsoft.AspNetCore.Diagnostics, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Eccezione interna System.Net.Sockets.SocketException gestita in MobilityServer.Helpers.HttpClientHelper+<PostRead>d__4`1.MoveNext:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.ConnectHelper+<ConnectAsync>d__2.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
And this the Exception Stack Trace:
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
at System.Threading.Tasks.ValueTask`1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at MobilityServer.Helpers.HttpClientHelper.PostRead[T](String baseAddress, String url, Object entity) in c:\MyProject\WebApp\Helpers\HttpClientHelper.cs:line 63
I don't get where the problem can be.
The outbound connections now are lower than the service plan (which is "S3 : 2").
SNAT seems way down the limit:
The same for the outbound connections:
(note: I've resized from 3 instances to 2 yesterday at 21:30, that's why those green "drop").
I really can't get what's going on. The environment sames correctly setup.
Any clues?
EDIT
Here's the version (i.e. the edits) to add the suggested IHttpClientFactory:
// Startup.cs
public void ConfigureServices(IServiceCollection services)
{
services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
services.AddApplicationInsightsTelemetry(Configuration["APPINSIGHTS_CONNECTIONSTRING"]);
services.AddHttpClient<MigrationHelper>();
}
public class MigrationHelper
{
HttpClientHelper httpc;
public MigrationHelper(IConfiguration config, HttpClient httpClient)
{
// ...
httpc = new HttpClientHelper(config, httpClient);
}
}
public class HttpClientHelper
{
string ApiKey = "";
HttpClient httpClient;
public HttpClientHelper(IConfiguration config, HttpClient _httpClient)
{
var appsettings = config.GetSection("AppSettings");
ApiKey = appsettings.GetChildren().Where(x => x.Key == "URL").FirstOrDefault().Value;
httpClient = _httpClient;
httpClient.DefaultRequestHeaders.Add("Authorization", ApiKey);
}
// ...
}
Still doesn't works, and I got the same error Socket messages. Here's a full example:
{
"Message": "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond",
"Data": {},
"InnerException": {
"ClassName": "System.Net.Sockets.SocketException",
"Message": "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond",
"Data": null,
"InnerException": null,
"HelpURL": null,
"StackTraceString": " at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)",
"RemoteStackTraceString": null,
"RemoteStackIndex": 0,
"ExceptionMethod": null,
"HResult": -2147467259,
"Source": "System.Private.CoreLib",
"WatsonBuckets": null,
"NativeErrorCode": 10060
},
"StackTrace": " at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)\r\n at System.Threading.Tasks.ValueTask`1.get_Result()\r\n at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Threading.Tasks.ValueTask`1.get_Result()\r\n at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)\r\n at System.Threading.Tasks.ValueTask`1.get_Result()\r\n at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)\r\n at MobilityServer.Helpers.HttpClientHelper.PostRead[T](String baseAddress, String url, Object entity) in c:\\MyProject\\WebApp\\Helpers\\HttpClientHelper.cs:line 60",
"HelpLink": null,
"Source": "System.Net.Http",
"HResult": -2147467259
}

Try 1: The maximum you can do in your app is to use IHttpClientFactory to implement resilient HTTP requests (see: https://learn.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests).
Try 2: When you scale out the endpoint (if it is in your control) there's a Load Balancer over all your instances, so, traffic is automatically load balanced between them. Maybe you need a better load balancing solution (see: https://learn.microsoft.com/en-us/azure/architecture/guide/technology-choices/load-balancing-overview).
Try 3: If your app works better on a single machine, you should scale up and use per-app scaling (see: https://learn.microsoft.com/en-us/azure/app-service/manage-scale-per-app)
Try 4: Troubleshooting intermittent outbound connection errors in Azure App Service (see: https://learn.microsoft.com/en-us/azure/app-service/troubleshoot-intermittent-outbound-connection-errors)
P.S.: a wild guess is that the endpoint could not handle the increasing requests from your 3 instances (an instance has a limit of 256 concurrent connections (per server endpoint) allowed when making requests using an HttpClient object set via HttpClientHandler.MaxConnectionsPerServer property); that's why when you decrease the number of instances it behaves better

Azure storage error: "An attempt was made to access a socket in a way forbidden by its access permissions."

The Problem:
I see there are a few questions with this error, but it doesn't solve my issue.
I have a Web API hosted in Azure web application (Dev/Test F1 Tier) that uploads files to Azure storage blob container and saves to Azure storage table, but when I do that, I get the following error:
"An attempt was made to access a socket in a way forbidden by its access permissions.". If anyone can help or has any idea where I can look, I'd appreciate it. Thank you.
I've Tried:
I have used Application Insights and attached debugger from VS 2019, full stack trace at the bottom.
Connection string (Note '...' to replace full AccountName and AccountKey):
"StorageConnectionString": "DefaultEndpointsProtocol=https;AccountName=s...001;AccountKey=TJi...nDkw==;EndpointSuffix=core.windows.net"
On the azure web application side I've done the following:
I've Allowed all CORS
I don't use any VNet's
I don't have any Access restrictions
I'm using the connection string taken from Access keys under Settings on Azure storage
On the Azure storage side, I've done the following:
Account kind: StorageV2 (general purpose v2)
I've Allowed all CORS
I don't use any VNet's
Allow access from is set to All Networks
Code used to upload files
BlobContainerClient container = new BlobContainerClient(_AppSettings.StorageConnectionString,
$"patient-notes-{patientid}");
container.CreateIfNotExists(PublicAccessType.Blob);
var blockBlob = container.GetBlobClient(fileName);
blockBlob.Upload(stream);
return blockBlob.Uri;
Code to table storage:
private readonly AppSettings _AppSettings;
private readonly CloudStorageAccount _storageAccount;
private readonly CloudTableClient _tableClient;
private readonly TelemetryClient _telemetry;
public ErrorLogTableStorage(IOptions<AppSettings> appSettings, TelemetryClient telemetry)
{
_telemetry = telemetry;
_AppSettings = appSettings?.Value;
_storageAccount = CloudStorageAccount.Parse(_AppSettings.AzureTableStorageConnectionString);
_tableClient = _storageAccount.CreateCloudTableClient();
CloudTable table = _tableClient.GetTableReference(AzureStorageTable.ErrorLog.ToString());
table.CreateIfNotExistsAsync();
}
public async Task CreateErrorLog(ErrorLogEntity errorLog)
{
try
{
CloudTable table = _tableClient.GetTableReference(AzureStorageTable.ErrorLog.ToString());
TableOperation insertOperation = TableOperation.InsertOrMerge(errorLog);
await table.ExecuteAsync(insertOperation);
}
catch (Exception ex)
{
_telemetry.TrackException(ex);
}
}
Stack Trace for the table storage:
Microsoft.Azure.Cosmos.Table.StorageException:
at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor+<ExecuteAsync>d__1`1.MoveNext (Microsoft.Azure.Cosmos.Table, Version=1.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Infrastructure.AzureTableStorage.ErrorLogTableStorage+<CreateErrorLog>d__5.MoveNext (Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullInfrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: AzureTableStorage\ErrorLogTableStorage.csInfrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 46)
Inner exception System.Net.Http.HttpRequestException handled at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor+<ExecuteAsync>d__1`1.MoveNext:
at System.Net.Http.ConnectHelper+<ConnectAsync>d__1.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<ConnectAsync>d__52.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<CreateHttp11ConnectionAsync>d__53.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<GetHttpConnectionAsync>d__45.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<SendWithRetryAsync>d__47.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.RedirectHandler+<SendAsync>d__4.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.DiagnosticsHandler+<SendAsync>d__2.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpClient+<FinishSendAsyncUnbuffered>d__71.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.Cosmos.Table.RestExecutor.TableCommand.Executor+<ExecuteAsync>d__1`1.MoveNext (Microsoft.Azure.Cosmos.Table, Version=1.0.8.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
Inner exception System.Net.Sockets.SocketException handled at System.Net.Http.ConnectHelper+<ConnectAsync>d__1.MoveNext:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.ConnectHelper+<ConnectAsync>d__1.MoveNext (System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)

Okay, I got this to work, I had to add AppSettings: to the application setting.
change from this application setting
to this
But why do I need to add AppSettings:?
What's the correct way of doing this?

SQL Server Connection timeout issue with application

We have an application built in .NET with SQL Server as database where we are doing INSERT operations using EF(Entity Framework) and GET Operations using ADO.NET.
We are facing an issue recently in app where sometimes we are unable to do INSERT operation with exception of Timeout. We are able to fetch the data from application but unable to insert any data.
To resolve this we need to restart the SQL Server instance every time. But after one or two days again same situation arises where we are unable to do INSERT operations
NOTE : In ConnectionString we have kept Connection TimeOut value= 180.
Please find the stack trace error :
MIRRA.CareManagement.UM.Authorization.Exceptions.BusinessException:
at MIRRA.CareManagement.UM.Authorization.Business.Authorization.AuthorizationManager.SaveAuthorization (MIRRA.CareManagement.UM.Authorization.Business, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
at MIRRA.CareManagement.UM.Authorization.Web.Controllers.UM.Authorization.AuthorizationController.SaveCalypsoLyteAuthorization (MIRRA.CareManagement.UM.Authorization.Web, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
Inner exception System.Data.Entity.Infrastructure.DbUpdateException handled at MIRRA.CareManagement.UM.Authorization.Business.Authorization.AuthorizationManager.SaveAuthorization:
at MIRRA.CareManagement.UM.Authorization.DataRepository.EFRepository.EFGenericRepository`1.Save (MIRRA.CareManagement.UM.Authorization.DataRepository, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMIRRA.CareManagement.UM.Authorization.DataRepository, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\Agents\UM\UM-Portal-01\_work\9\s\MIRRA.UM.Authorization.Data\EFRepository\EFGenericRepository.csMIRRA.CareManagement.UM.Authorization.DataRepository, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 435)
at MIRRA.CareManagement.UM.Authorization.DataRepository.EFRepository.Authorization.AuthorizationRepository.SaveAuthorization (MIRRA.CareManagement.UM.Authorization.DataRepository, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMIRRA.CareManagement.UM.Authorization.DataRepository, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\Agents\UM\UM-Portal-01\_work\9\s\MIRRA.UM.Authorization.Data\EFRepository\Authorization\AuthorizationRepository.csMIRRA.CareManagement.UM.Authorization.DataRepository, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 180)
Inner exception System.Data.Entity.Core.UpdateException handled at MIRRA.CareManagement.UM.Authorization.DataRepository.EFRepository.EFGenericRepository`1.Save:
at System.Data.Entity.Core.Mapping.Update.Internal.UpdateTranslator.Update (EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction (EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.Entity.Core.Objects.ObjectContext.SaveChangesToStore (EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute (EntityFramework.SqlServer, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.Entity.Core.Objects.ObjectContext.SaveChangesInternal (EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.Entity.Internal.InternalContext.SaveChanges (EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
Inner exception System.Data.SqlClient.SqlException handled at System.Data.Entity.Core.Mapping.Update.Internal.UpdateTranslator.Update:
at System.Data.SqlClient.SqlConnection.OnError (System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning (System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.SqlClient.TdsParser.TryRun (System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData (System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.SqlClient.SqlDataReader.get_MetaData (System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader (System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds (System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.SqlClient.SqlCommand.RunExecuteReader (System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.SqlClient.SqlCommand.RunExecuteReader (System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.SqlClient.SqlCommand.ExecuteReader (System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch (EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader (EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.Entity.Core.Mapping.Update.Internal.DynamicUpdateCommand.Execute (EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Data.Entity.Core.Mapping.Update.Internal.UpdateTranslator.Update (EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
Inner exception System.ComponentModel.Win32Exception handled at System.Data.SqlClient.SqlConnection.OnError:
Please suggest if anyone faced the same issue.

Transient System.Security.Cryptography.CryptographicException in TwilioRequestValidator

So we have our Twilio callbacks setup to validate incoming requests, via the TwilioRequestValidator documented here.
However, something we're seeing, is that the production server will run for weeks with no problems, and then suddenly start failing with a CryptographicException. This causes all incoming Twilio requests to fail.
We have a call-stack (thanks to Azure Application Insights):
System.Security.Cryptography.CryptographicException:
at System.Security.Cryptography.CryptographicException.ThrowCryptographicException (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Security.Cryptography.Utils.HashData (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Security.Cryptography.SHA1CryptoServiceProvider.HashCore (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Security.Cryptography.HashAlgorithm.TransformBlock (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Security.Cryptography.HMAC.HashCore (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Security.Cryptography.HashAlgorithm.ComputeHash (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Twilio.Security.RequestValidator.GetValidationSignature (Twilio, Version=5.14.1.0, Culture=neutral, PublicKeyToken=null)
at Misc.TwilioRequestValidator.ValidateTwilioRequestAttribute.IsValidRequest (RCHHRATool, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
at Misc.TwilioRequestValidator.ValidateTwilioRequestAttribute.OnActionExecuting (RCHHRATool, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
at System.Web.Mvc.Async.AsyncControllerActionInvoker+AsyncInvocationWithFilters.InvokeActionMethodFilterAsynchronouslyRecursive (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Async.AsyncControllerActionInvoker+AsyncInvocationWithFilters.InvokeActionMethodFilterAsynchronouslyRecursive (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass33.<BeginInvokeActionMethodWithFilters>b__31 (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.Begin (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Async.AsyncControllerActionInvoker.BeginInvokeActionMethodWithFilters (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass21.<BeginInvokeAction>b__19 (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.Begin (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Async.AsyncControllerActionInvoker.BeginInvokeAction (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Controller.<BeginExecuteCore>b__1c (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1.CallBeginDelegate (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.Begin (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Controller.BeginExecuteCore (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.Begin (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Controller.BeginExecute (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__4 (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1.CallBeginDelegate (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.Begin (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.Mvc.MvcHandler.BeginProcessRequest (System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute (System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Web.HttpApplication+<>c__DisplayClass285_0.<ExecuteStepImpl>b__0 (System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Web.HttpApplication.ExecuteStepImpl (System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Web.HttpApplication.ExecuteStep (System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
And we have some application code, here:
[AttributeUsage(AttributeTargets.Method)]
public class ValidateTwilioRequestAttribute : ActionFilterAttribute
{
private readonly RequestValidator _requestValidator;
public ValidateTwilioRequestAttribute()
{
var authToken = ConfigurationManager.AppSettings["TwilioAuthToken"];
_requestValidator = new RequestValidator(authToken);
}
public override void OnActionExecuting(ActionExecutingContext actionContext)
{
var context = actionContext.HttpContext;
if (!IsValidRequest(context.Request))
{
actionContext.Result = new HttpStatusCodeResult(HttpStatusCode.Forbidden);
}
base.OnActionExecuting(actionContext);
}
private bool IsValidRequest(HttpRequestBase request)
{
var signature = request.Headers["X-Twilio-Signature"];
Debug.WriteLine(request.Headers["X-Twilio-Signature"]);
//var requestUrl = request.RawUrl;
var requestUrl = rewriteUri(request.Url.AbsoluteUri);
Debug.WriteLine("URI is: " + rewriteUri(request.Url.AbsoluteUri));
return _requestValidator.Validate(requestUrl, request.Form, signature);
}
private string rewriteUri(string absoluteUri)
{
//check to make sure we're not replacing 'https' with 'httpss'
if (!absoluteUri.Contains("https"))
{
return Regex.Replace(absoluteUri, #"http", "https");
}
return absoluteUri;
}
}
Any ideas on what could be causing this? Ideally, I'd like to handle this exception state, but I'm not sure how to handle the exception without knowing what the cause is.
EDIT: I'm getting this error message when I drill into the exception in application insights:
"Hash not valid for use in specified state."
EDIT #2: A quick search on that error messages digs up this interesting discussion on the thread safety of the Cryptography object.

Looks like this is a bug in the Twilio-csharp library. I believe they're using their Sha1 member in a non-thread safe manner.
I've hacked up a solution to re-init the class in the event of a CryptographicException. Since a server restart is fixing the issue, I think a force re-init of the class is called for.
try
{
return _requestValidator.Validate(requestUrl, request.Form, signature);
}
catch (CryptographicException e)
{
//if the request fails, re-init the class and try again
var authToken = ConfigurationManager.AppSettings["TwilioAuthToken"];
_requestValidator = new RequestValidator(authToken);
return IsValidRequest(request);
}
EDIT: Worth noting that this fix is causing knock-on issues related to the non-thread safe manner in which the library is used.