We Keep Coding

MSAL .NET Device Code Authentication in ASP.NET API not working when app is running in docker container

I'm trying to use device code authentication in my containerized ASP.NET WebAPI project. Everything works fine when i run the app outside of docker but i'm getting the following error when the app runs in a docker container:
Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HMBPTCKK2U5R", Request id "0HMBPTCKK2U5R:00000002": An unhandled exception was thrown by the application.
MSAL.NetCore.4.36.0.0.MsalServiceException:
ErrorCode: invalid_client
Microsoft.Identity.Client.MsalServiceException: A configuration issue is preventing authentication - check the error message from the server for details. You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details. Original exception: AADSTS7000218: The request body must contain the following parameter: 'client_assertion' or 'client_secret'.
Trace ID: 79b92211-930d-41a2-9bf7-af89711cbe00
Correlation ID: 7e73cb95-1e9c-4a28-ab66-0672aa3e9e1c
Timestamp: 2021-09-17 14:56:50Z
at Microsoft.Identity.Client.OAuth2.OAuth2Client.ThrowServerException(HttpResponse response, RequestContext requestContext)
at Microsoft.Identity.Client.OAuth2.OAuth2Client.CreateResponse[T](HttpResponse response, RequestContext requestContext)
at Microsoft.Identity.Client.OAuth2.OAuth2Client.ExecuteRequestAsync[T](Uri endPoint, HttpMethod method, RequestContext requestContext, Boolean expectErrorsOn200OK, Boolean addCommonHeaders)
at Microsoft.Identity.Client.OAuth2.OAuth2Client.GetTokenAsync(Uri endPoint, RequestContext requestContext, Boolean addCommonHeaders)
at Microsoft.Identity.Client.OAuth2.TokenClient.SendHttpAndClearTelemetryAsync(String tokenEndpoint, ICoreLogger logger)
at Microsoft.Identity.Client.OAuth2.TokenClient.SendHttpAndClearTelemetryAsync(String tokenEndpoint, ICoreLogger logger)
at Microsoft.Identity.Client.OAuth2.TokenClient.SendTokenRequestAsync(IDictionary`2 additionalBodyParameters, String scopeOverride, String tokenEndpointOverride, CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.DeviceCodeRequest.WaitForTokenResponseAsync(DeviceCodeResult deviceCodeResult, CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.DeviceCodeRequest.WaitForTokenResponseAsync(DeviceCodeResult deviceCodeResult, CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.DeviceCodeRequest.ExecuteAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.ApiConfig.Executors.PublicClientExecutor.ExecuteAsync(AcquireTokenCommonParameters commonParameters, AcquireTokenWithDeviceCodeParameters deviceCodeParameters, CancellationToken cancellationToken)
at Elicity.Ampere.Logic.Authentication.DeviceCodeAuthenticationLogic.GetTokenForWebApiUsingDeviceCodeFlowAsync(IEnumerable`1 scopes, Action`1 deviceCodeCallback) in /app/Logic/Logic.Authentication/DeviceCodeAuthenticationLogic.cs:line 109
at Elicity.Ampere.Logic.Authentication.DeviceCodeAuthenticationLogic.AcquireATokenFromCacheOrDeviceCodeFlowAsync(IEnumerable`1 scopes, Boolean interactive, Action`1 deviceCodeCallback) in /app/Logic/Logic.Authentication/DeviceCodeAuthenticationLogic.cs:line 78
at Elicity.Ampere.Service.Authentication.AuthenticationController.Get() in /app/Services/Service.Authentication/Controllers/AuthenticationController.cs:line 65
at lambda_method7(Closure , Object )
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
StatusCode: 401
ResponseBody: {"error":"invalid_client","error_description":"AADSTS7000218: The request body must contain the following parameter: 'client_assertion' or 'client_secret'.\r\nTrace ID: 79b92211-930d-41a2-9bf7-af89711cbe00\r\nCorrelation ID: 7e73cb95-1e9c-4a28-ab66-0672aa3e9e1c\r\nTimestamp: 2021-09-17 14:56:50Z","error_codes":[7000218],"timestamp":"2021-09-17 14:56:50Z","trace_id":"79b92211-930d-41a2-9bf7-af89711cbe00","correlation_id":"7e73cb95-1e9c-4a28-ab66-0672aa3e9e1c","error_uri":"https://login.microsoftonline.com/error?code=7000218"}
Headers: Cache-Control: no-store, no-cache
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
P3P: CP="DSP CUR OTPi IND OTRi ONL FIN"
client-request-id: 7e73cb95-1e9c-4a28-ab66-0672aa3e9e1c
x-ms-request-id: 79b92211-930d-41a2-9bf7-af89711cbe00
x-ms-ests-server: 2.1.12025.15 - WEULR2 ProdSlices
x-ms-clitelem: 1,7000218,0,,
Set-Cookie: fpc=Ah_T82wZi7hJgqcpQ9HohJiJEGHQAQAAAFaj1tgOAAAAUSo9cgEAAACyo9bYDgAAAA; expires=Sun, 17-Oct-2021 14:56:50 GMT; path=/; secure; HttpOnly; SameSite=None, x-ms-gateway-slice=estsfd; path=/; secure; httponly, stsservicecookie=estsfd; path=/; secure; samesite=none; httponly
Date: Fri, 17 Sep 2021 14:56:50 GMT
I tested the code in a MacOS and Linux Environment and both times it worked as expected.
public DeviceCodeAuthenticationLogic(PublicClientApplicationOptions msalOptions, IPublishEndpoint publishEndpoint)
{
_publishEndpoint = publishEndpoint;
_app = PublicClientApplicationBuilder.CreateWithApplicationOptions(msalOptions).Build();
var storageProperties = new StorageCreationPropertiesBuilder("user", "~/cache", msalOptions.ClientId)
.WithLinuxUnprotectedFile()
// This makes development on mac easier.
.WithMacKeyChain("app_context", "user_token")
.Build();
var cacheHelper = MsalCacheHelper.CreateAsync(storageProperties).GetAwaiter().GetResult();
cacheHelper.RegisterCache(_app.UserTokenCache);
}
public async Task<AuthenticationResult> AcquireATokenFromCacheOrDeviceCodeFlowAsync(IEnumerable<string> scopes, bool interactive = true, Action<DeviceCodeResult> deviceCodeCallback = null)
{
AuthenticationResult result = null;
var accounts = await _app.GetAccountsAsync();
if (accounts.Any())
try
{
// Attempt to get a token from the cache (or refresh it silently if needed)
result = await _app.AcquireTokenSilent(scopes, accounts.FirstOrDefault())
.ExecuteAsync();
}
catch (MsalUiRequiredException)
{
}
// No token in cache, attempt device code flow
if (result == null && interactive)
result = await GetTokenForWebApiUsingDeviceCodeFlowAsync(scopes, deviceCodeCallback);
return result;
}
private async Task<AuthenticationResult> GetTokenForWebApiUsingDeviceCodeFlowAsync(IEnumerable<string> scopes,
Action<DeviceCodeResult> deviceCodeCallback = null)
{
AuthenticationResult result;
try
{
result = await _app.AcquireTokenWithDeviceCode(scopes,
callback =>
{
if (deviceCodeCallback is not null) deviceCodeCallback(callback);
return Task.FromResult(0);
}).ExecuteAsync();
}
catch (MsalServiceException ex)
{
throw;
}
catch (OperationCanceledException)
{
result = null;
}
catch (MsalClientException ex)
{
result = null;
}
return result;
}
The error indicates to me that it's a configuration error on my part but it why would it work outside of an docker environment.
I am a bit lost on this one and any help is greatly appreciated.

I found the problem. The app registration in my b2c was not configured to allow public client authentication.
The switch controlling that setting
The authentication code worked after changing that value.

.NET5 JsonPatchDocument.ApplyTo throws when add or replace

I got a problem with my JsonPatchDocument object, which can Remove value from the field, but cannot add/replace it.
My controller code is as following:
[HttpPatch]
[Route("{layersGroupId}")]
public async Task<ActionResult> Patch([FromServices] IMediator mediator,
[FromBody] List<Operation<LayerGroup>> operations,
Guid layersGroupId,
CancellationToken cancellationToken)
{
await mediator.Send(new PatchLayersGroupCommand
{
LayersGroupId = layersGroupId,
JsonPatch = new JsonPatchDocument<LayerGroup>(operations, new DefaultContractResolver())
}, cancellationToken);
return Ok();
}
Code in handler:
var layersGroup = await databaseContext.UsersLayersViews
.Where(x => x.UserId == identity.Current.UserId && x.LayerGroupId == request.LayersGroupId)
.Select(x => x.LayerGroup)
.SingleOrDefaultAsync(cancellationToken);
if (layersGroup == null)
throw new ValidationException(ErrorCode.InvalidId, "Invalid layers group ID");
request.JsonPatch.ApplyTo(layersGroup); // Code throws here
The exception is:
Unhandled JsonPatchException caught, none of known global handlers could apply
ExceptionDetail
{HResult: -2146233088, Message: 'The value ''"NewName"'' is invalid for target location.', Source: 'Microsoft.AspNetCore.JsonPatch', StackTrace: ' at Microsoft.AspNetCore.JsonPatch.Internal.ErrorReporter.<>c.<.cctor>b__1_0(JsonPatchError error)
at Microsoft.AspNetCore.JsonPatch.Adapters.ObjectAdapter.Replace(Operation operation, Object objectToApplyTo)
at Microsoft.AspNetCore.JsonPatch.Operations.Operation`1.Apply(TModel objectToApplyTo, IObjectAdapter adapter)
at Microsoft.AspNetCore.JsonPatch.JsonPatchDocument`1.ApplyTo(TModel objectToApplyTo, IObjectAdapter adapter)
at Microsoft.AspNetCore.JsonPatch.JsonPatchDocument`1.ApplyTo(TModel objectToApplyTo)
at DHIPL.UrbanTools.GIS.Api.UseCases.Groups.Put.RenameGroup.RenameLayersGroupCommandHandler.Handle(RenameLayersGroupCommand request, CancellationToken cancellationToken) in C:\Users\msty\source\repos\urban-tools-backend\src\Services\GIS\GIS.Api\UseCases\Groups\Put\RenameGroup\RenameLayersGroupCommandHandler.cs:line 36
at DHIPL.Core.Api.Behaviors.ValidatorBehavior`2.Handle(TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate`1 next) in C:\Users\msty\source\repos\urban-tools-backend\src\Packages\DHIPL.Core.Api\Behaviors\ValidatorBehavior.cs:line 31
at DHIPL.Core.Api.Behaviors.TransactionBehavior`2.<>c__DisplayClass0_0.<<Handle>b__0>d.MoveNext() in C:\Users\msty\source\repos\urban-tools-backend\src\Packages\DHIPL.Core.Api\Behaviors\TransactionBehavior.cs:line 14
--- End of stack trace from previous location ---
at DHIPL.Core.Transactions.TransactionHelper.Wrap[TResult](Func`1 action) in C:\Users\msty\source\repos\urban-tools-backend\src\Packages\DHIPL.Core\Transactions\TransactionHelper.cs:line 51
at DHIPL.Core.Api.Behaviors.TransactionBehavior`2.Handle(TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate`1 next) in C:\Users\msty\source\repos\urban-tools-backend\src\Packages\DHIPL.Core.Api\Behaviors\TransactionBehavior.cs:line 14
at DHIPL.Core.Api.Behaviors.LoggingBehavior`2.Handle(TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate`1 next) in C:\Users\msty\source\repos\urban-tools-backend\src\Packages\DHIPL.Core.Api\Behaviors\LoggingBehavior.cs:line 32
at MediatR.Pipeline.RequestExceptionProcessorBehavior`2.Handle(TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate`1 next)
at MediatR.Pipeline.RequestExceptionProcessorBehavior`2.Handle(TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate`1 next)
at MediatR.Pipeline.RequestExceptionActionProcessorBehavior`2.Handle(TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate`1 next)
at MediatR.Pipeline.RequestExceptionActionProcessorBehavior`2.Handle(TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate`1 next)
at MediatR.Pipeline.RequestPostProcessorBehavior`2.Handle(TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate`1 next)
at MediatR.Pipeline.RequestPreProcessorBehavior`2.Handle(TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate`1 next)
at DHIPL.UrbanTools.GIS.Api.Controllers.GroupController.PutRename(IMediator mediator, List`1 operations, Guid layersGroupId, CancellationToken cancellationToken) in C:\Users\msty\source\repos\urban-tools-backend\src\Services\GIS\GIS.Api\Controllers\GroupController.cs:line 41
at lambda_method668(Closure , Object )
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at DHIPL.Core.Api.Exceptions.CoreExceptionMiddleware.Invoke(HttpContext context) in C:\Users\msty\source\repos\urban-tools-backend\src\Packages\DHIPL.Core.Api\Exceptions\CoreExceptionMiddleware.cs:line 30', FailedOperation: {…}, AffectedObject: {…}, Type: 'Microsoft.AspNetCore.JsonPatch.Exceptions.JsonPatchException'}
There is no single post about this error in the Internet. I'm confused now. Did anyone have such a problem before? How to solve it?
Just in case, PatchLayersGroupCommand has these data inside:
{
LayersGroupId: 'c36bb6c1-4be2-44b9-9239-d7e41d10ebaa',
JsonPatch: {
Operations: [
{
value: {
ValueKind: 'String',
_typeTag: 'JsonElement'
},
OperationType: 'Replace',
path: '/GroupName',
op: 'replace',
from: null,
_typeTag: 'Operation`1'
}
],
ContractResolver: {
DynamicCodeGeneration: false,
DefaultMembersSearchFlags: 'Instance, Public',
SerializeCompilerGeneratedMembers: false,
IgnoreSerializableInterface: false,
IgnoreSerializableAttribute: true,
IgnoreIsSpecifiedMembers: false,
IgnoreShouldSerializeMembers: false,
NamingStrategy: null,
_typeTag: 'DefaultContractResolver'
},
_typeTag: 'JsonPatchDocument`1'
},
_typeTag: 'RenameLayersGroupCommand'
}

Support for JsonPatch is enabled using the Microsoft.AspNetCore.Mvc.NewtonsoftJson package. To enable this feature, apps must:
Install the Microsoft.AspNetCore.Mvc.NewtonsoftJson NuGet package.
Update the project's Startup.ConfigureServices method to include a call to AddNewtonsoftJson:
services
.AddControllers()
.AddNewtonsoftJson();
After adding it to the services take JsonPatchDocument<T> directly from body and don't create it yourself. JsonPatch should now work.

How do I add a scoped DbContext to my ASP NET Core API Controllers that uses EntityFrameworkCore and SSH tunneling into MySQL?

I have the following ConfigureServices() content in my Startup.cs
public void ConfigureServices(IServiceCollection services)
{
services.AddControllers();
services.AddDbContext<GMContext>(options =>
{
var myConnectionSettings = Configuration.GetSection("ConnectionSettings").Get<ConnectionSettings>();
var sshSettings = myConnectionSettings.SSH;
var databaseSettings = myConnectionSettings.Database;
string keyFilePath = Path.Combine(Directory.GetCurrentDirectory(), "test4");
var authMethods = new List<AuthenticationMethod>
{
new PrivateKeyAuthenticationMethod(sshSettings.UserName, new PrivateKeyFile(keyFilePath))
};
using (var sshClient = new SshClient(new ConnectionInfo(sshSettings.Server,
sshSettings.Port, sshSettings.UserName, authMethods.ToArray())))
{
sshClient.Connect();
var forwardedPort = new ForwardedPortLocal(databaseSettings.BoundHost, databaseSettings.Host,
databaseSettings.Port);
sshClient.AddForwardedPort(forwardedPort);
forwardedPort.Start();
MySqlConnectionStringBuilder csb;
csb = new MySqlConnectionStringBuilder
{
Server = databaseSettings.BoundHost,
Port = forwardedPort.BoundPort,
UserID = databaseSettings.UserName,
Password = databaseSettings.Password,
Database = databaseSettings.DatabaseName
};
options.UseMySQL(csb.ConnectionString);
}
});
I then pick up the context in a controller and attempt to query the context I get a connection denied error:
namespace CoreGMWebAPI.Controllers
{
[Produces("application/json")]
[Route("api/[controller]")]
public class CropController : Controller
{
private readonly ILogger<CropController> _logger;
private readonly GMContext _context;
public CropController(ILogger<CropController> logger, GMContext context)
{
_logger = logger;
_context = context;
}
/// <summary>
/// Gets a Sample Items
/// </summary>
/// <returns></returns>
[HttpGet]
public async Task<List<TCountries>> GetSampleItems()
{
var result = _context.TCountries.Take(10).ToList();
return result;
}
}
}
The Error:
System.InvalidOperationException: An exception has been raised that is
likely due to a transient failure. Consider enabling transient error
resiliency by adding 'EnableRetryOnFailure()' to the 'UseMySql' call.
---> MySql.Data.MySqlClient.MySqlException (0x80004005): Unable to connect to any of the specified MySQL hosts. --->
System.AggregateException: One or more errors occurred. (No connection
could be made because the target machine actively refused it.
127.0.31.1:54960) ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException
(10061): No connection could be made because the target machine
actively refused it. 127.0.31.1:54960 at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception
source) at System.Net.Sockets.Socket.EndConnect(IAsyncResult
asyncResult) at
System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.TcpClient.<>c.b__28_1(IAsyncResult
asyncResult) at
System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean
requiresSynchronization) --- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean
includeTaskCanceledExceptions) at
System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout,
CancellationToken cancellationToken) at
System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout) at
MySql.Data.Common.StreamCreator.GetTcpStream(MySqlConnectionStringBuilder
settings) at
MySql.Data.Common.StreamCreator.GetStream(MySqlConnectionStringBuilder
settings) at MySql.Data.MySqlClient.NativeDriver.Open() at
MySql.Data.MySqlClient.NativeDriver.Open() at
MySql.Data.MySqlClient.Driver.Open() at
MySql.Data.MySqlClient.Driver.Create(MySqlConnectionStringBuilder
settings) at
MySql.Data.MySqlClient.MySqlPool.CreateNewPooledConnection() at
MySql.Data.MySqlClient.MySqlPool.GetPooledConnection() at
MySql.Data.MySqlClient.MySqlPool.TryToGetDriver() at
MySql.Data.MySqlClient.MySqlPool.GetConnection() at
MySql.Data.MySqlClient.MySqlConnection.Open() at
Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean
errorsExpected) at
Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean
errorsExpected) at
Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject
parameterObject) at
Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable1.Enumerator.InitializeReader(DbContext _, Boolean result) at MySql.Data.EntityFrameworkCore.Storage.Internal.MySQLExecutionStrategy.Execute[TState,TResult](TState state, Func3 operation, Func3 verifySucceeded) --- End of inner exception stack trace --- at MySql.Data.EntityFrameworkCore.Storage.Internal.MySQLExecutionStrategy.Execute[TState,TResult](TState state, Func3 operation, Func3 verifySucceeded) at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable1.Enumerator.MoveNext()
at System.Collections.Generic.List1..ctor(IEnumerable1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) at
CoreGMWebAPI.Controllers.CropController.GetSampleItem() in
C:\Users\lobos\source\repos\CoreGMWebAPI\CoreGMWebAPI\Controllers\CropController.cs:line
31 at lambda_method(Closure , Object ) at
Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()
at
Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper
mapper, ObjectMethodExecutor executor, Object controller, Object[]
arguments) at
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Logged|12_1(ControllerActionInvoker
invoker) at
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker
invoker, Task lastTask, State next, Scope scope, Object state, Boolean
isCompleted) at
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed
context) at
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State&
next, Scope& scope, Object& state, Boolean& isCompleted) at
Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location where exception was thrown --- at
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|19_0(ResourceInvoker
invoker, Task lastTask, State next, Scope scope, Object state, Boolean
isCompleted) at
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker
invoker) at
Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint
endpoint, Task requestTask, ILogger logger) at
Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext
context) at
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext
context)
HEADERS
======= Accept: application/json Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9,ca;q=0.8,fr;q=0.7,nb;q=0.6 Connection:
close Host: localhost:44313 Referer:
https://localhost:44313/swagger/index.html User-Agent: Mozilla/5.0
(Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/86.0.4240.198 Safari/537.36 dnt: 1 sec-fetch-site: same-origin
sec-fetch-mode: cors sec-fetch-dest: empty sec-gpc: 1
When I run the following quick test inside the ConfigureServices() method in Startup.cs it connects and queries without any issues:
...
options.UseMySQL(csb.ConnectionString);
var otherOps = new DbContextOptionsBuilder<GMContext>();
otherOps.UseMySQL(csb.ConnectionString);
using (var context = new GMContext(otherOps.Options))
{
var testRun = context.TCountries.Take(3).ToList();
}
This leads me to suspect that the SSHClient is closing before the Controller can use the dbContext. Should I somehow be passing the SSHClient as well as the dbContext? Any help would be appreciated.

Seems like the global exception handler is rethrowing the **BadHttpRequestException: Unexpected end of request content** as unhandled exception

My ASP .NET Core WebAPI application is getting some unidentified requests resulting in the below error.
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content.
This error is getting logged only at the kestrel (stdout_logfile) and not in the application's error log. Seems like the global exception handler is rethrowing the exception as unhandled exception.
As the request keeps coming, the application becomes slow and gets crashed, ended up in restarting the supervisor to bring up the application.
Error log:
^[[41m^[[30mfail^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
An unhandled exception has occurred while executing the request.
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content.
at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value)
at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
Code
Exception Filtering code:
public void ConfigureServices(IServiceCollection services)
{
...
...
services.AddControllers()
.ConfigureApiBehaviorOptions(options =>
{
options.SuppressConsumesConstraintForFormFileParameters = true;
options.SuppressInferBindingSourcesForParameters = true;
options.SuppressModelStateInvalidFilter = true;
options.SuppressMapClientErrors = true;
options.ClientErrorMapping[StatusCodes.Status404NotFound].Link =
"https://httpstatuses.com/404";
options.InvalidModelStateResponseFactory = context =>
{
var result = new BadRequestObjectResult(context.ModelState);
// TODO: add `using System.Net.Mime;` to resolve MediaTypeNames
result.ContentTypes.Add(MediaTypeNames.Application.Json);
result.ContentTypes.Add(MediaTypeNames.Application.Xml);
return result;
};
});
services.AddControllers(options =>
options.Filters.Add(new HttpResponseExceptionFilter())
);
...
...
}
Why is error not caught at the below global handler, or why is the error getting rethrown as unhandled exception. What am I doing wrong here?
Exception handler code:
public void OnActionExecuted(ActionExecutedContext context)
{
if (context.Exception is HttpResponseException exception)
{
logger.Error(context.Exception);
context.Result = new ObjectResult(exception.Value)
{
StatusCode = exception.Status,
};
context.ExceptionHandled = true;
}
else if (context.Exception is BadHttpRequestException badException)
{
logger.Error(context.Exception);
context.Result = new ObjectResult(badException.Data.Values)
{
StatusCode = badException.StatusCode,
};
context.ExceptionHandled = true;
}
else if (context.Exception is Exception otherException)
{
logger.Error(context.Exception);
context.Result = new ObjectResult(otherException.Data.Values)
{
StatusCode = otherException.HResult,
};
context.ExceptionHandled = true;
}
}

'Stream was already consumed' error using Polly to retry requests in ASP.NET Core

I have an ASP.NET Core 3.1 Web API service that is receiving a web request, doing some manipulations to it, and then passing it on to a backend service and returning the response synchronously. It was working fine but I wanted to introduce some retry logic to those backend requests in case some blips occurred.
I'm using a typed HttpClient and attempting to use Polly to implement the retry logic:
https://github.com/App-vNext/Polly/wiki/Polly-and-HttpClientFactory#using-polly-with-ihttpclientfactory
When the backend service works, everything seems to be fine but unfortunately, whenever my backend returns an error like a 500 Internal Server Error, I get the following exception:
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware: Error: An unhandled exception has occurred while executing the request.
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.InvalidOperationException: The stream was already consumed. It cannot be read again.
at System.Net.Http.StreamContent.PrepareContent()
at System.Net.Http.StreamContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpContent.CopyToAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at MyProject.MyController.Routing.HttpMessageRouter.SendRequestAndTrackTiming(Func`1 action, String destinationID) in /mnt/c/Users/myCode/source/repos/MyProject-GitLab/MyController/src/MyController/Routing/HttpMessageRouter.cs:line 59
at MyProject.MyController.Routing.HttpMessageRouter.SendNewRequest(IMessageWrapper`1 message) in /mnt/c/Users/myCode/source/repos/MyProject-GitLab/MyController/src/MyController/Routing/HttpMessageRouter.cs:line 33
at MyProject.MyController.Controllers.MyControllerController.Resource(String destinationId) in /mnt/c/Users/myCode/source/repos/MyProject-GitLab/MyController/src/MyController/Controllers/MyControllerController.cs:line 151
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Prometheus.HttpMetrics.HttpRequestDurationMiddleware.Invoke(HttpContext context)
at Prometheus.HttpMetrics.HttpRequestCountMiddleware.Invoke(HttpContext context)
at Prometheus.HttpMetrics.HttpInProgressMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
The code I have to send the request is as follows. This first method actually just utilizes a delegate so I can transparently add some metrics gathering around the typed HttpClient call. The typed HttpClient is called RoutingClient in this code:
public async Task<IActionResult> SendNewRequest(IMessageWrapper<HttpRequestMessage> message)
{
HttpResponseMessage destinationResponse = await SendRequestAndTrackTiming(() => _client.SendAsync(message.Message), message.DestinationID);
return CreateSerializeableResponseMessage(destinationResponse);
}
private ResponseMessageResult CreateSerializeableResponseMessage(HttpResponseMessage httpResponse)
{
ResponseMessageResult responseMessage = new ResponseMessageResult(httpResponse);
IOutputFormatter[] formattersList = { new HttpResponseMessageOutputFormatter() };
FormatterCollection<IOutputFormatter> formattersCollection = new FormatterCollection<IOutputFormatter>(formattersList);
responseMessage.Formatters = formattersCollection;
return responseMessage;
}
private async Task<HttpResponseMessage> SendRequestAndTrackTiming(Func<Task<HttpResponseMessage>> action, string destinationID)
{
HttpResponseMessage response = null;
Stopwatch stopwatch = new Stopwatch();
try
{
stopwatch.Start();
response = await action();
return response;
}
finally
{
stopwatch.Stop();
HttpStatusCode statusCode = (response != null) ? response.StatusCode : HttpStatusCode.InternalServerError;
_routedMessageMetricTracker.Histogram.Observe(destinationID, statusCode, stopwatch.Elapsed.TotalSeconds);
}
}
This is my code in Startup.ConfigureServices() (actually this is in an extension method I defined):
public static IServiceCollection AddRoutingClient(this IServiceCollection services, RoutingSettings routingSettings)
{
List<TimeSpan> retryTimeSpans = new List<TimeSpan>();
// routingSettings.RetrySeconds is just an array of double values.
foreach (double retrySeconds in routingSettings.RetrySeconds)
{
if (retrySeconds >= 0) retryTimeSpans.Add(TimeSpan.FromSeconds(retrySeconds));
}
services.AddHttpClient<IRoutingClient, RoutingClient>()
.AddPolicyHandler((services, request) => HttpPolicyExtensions.HandleTransientHttpError()
.WaitAndRetryAsync(retryTimeSpans, onRetry: (outcome, timespan, retryAttempt, context) =>
{
services.GetService<ILogger<RoutingClient>>()?
.LogWarning($"Delaying for {timespan.TotalMilliseconds}ms, then making retry {retryAttempt}.");
}
));
return services;
}
I'd really like to use the Polly approach because it seems clean but I am not sure what I am doing wrong here. I must be doing something wrong because I would expect this to be a very common use-case for Polly that should be handled.

It turns out my problem was not caused by the code dealing with the response above. Instead, it was actually caused by my code that was manipulating the request.
I was reading the incoming request as an HttpRequestMessage object via the HttpContext.GetHttpRequestMessage() method and I attempted to re-use that same object to pass on to the backend service via a call to my typed HttpClient. However, the content stream for that request is read-once so I had to make a copy of that HttpRequestMessage as described in this answer to another post: https://stackoverflow.com/a/34049029/1221718
Here is a slightly more verbose version of that answer's code:
private static async Task<HttpRequestMessage> CloneHttpRequestMessageAsync(HttpRequestMessage request)
{
HttpRequestMessage copyOfRequest = new HttpRequestMessage(request.Method, request.RequestUri);
// Copy the request's content (via a MemoryStream) into the cloned object
var ms = new MemoryStream();
if (request.Content != null)
{
await request.Content.CopyToAsync(ms).ConfigureAwait(false);
ms.Position = 0;
copyOfRequest.Content = new StreamContent(ms);
// Copy the content headers
if (request.Content.Headers != null)
{
foreach (var h in request.Content.Headers)
{
copyOfRequest.Content.Headers.Add(h.Key, h.Value);
}
}
}
copyOfRequest.Version = request.Version;
foreach (KeyValuePair<string, object> prop in request.Properties)
{
copyOfRequest.Properties.Add(prop);
}
foreach (KeyValuePair<string, IEnumerable<string>> header in request.Headers)
{
copyOfRequest.Headers.TryAddWithoutValidation(header.Key, header.Value);
}
return copyOfRequest;
}

Another solution to this problem is to enable buffering on the request stream.
This is done by calling the extension method EnableBuffering() for the HttpRequest, e.g.
public async Task InvokeAsync(HttpContext context, RequestDelegate next)
{
context.Request.EnableBuffering();
...
see https://devblogs.microsoft.com/dotnet/re-reading-asp-net-core-request-bodies-with-enablebuffering/ for more detail.