We Keep Coding

C# MongoDB driver logging invalid certificate even though queries work

I'm using the MongoDB driver with C# to work with a MongoDB cluster on Digital Ocean. All MongoDB clusters on Digital Ocean use self-signed certs. I have working code for using the CA certificate they provide when connecting to MongoDB. These settings work. I can read and write to the database just fine from within the application.
For reference, here's the code handling the Mongo connection:
using System.Net.Security;
using System.Security.Cryptography.X509Certificates;
using MongoDB.Driver;
namespace App.src.Common
{
public class MongoClientFactory
{
private X509Certificate? ca;
private readonly ILogger logger;
// MongoDB driver uses the same connection pool for client connections created with the
// same settings. To ensure this, always pass in the same reference to the driver as the
// one that was first instantiated with this class.
private readonly MongoClientSettings settings;
public MongoClientFactory(IConfiguration configuration, ILogger<MongoClientFactory> logger)
{
this.logger = logger;
IConfigurationSection config = configuration.GetSection("MongoSettings");
// TODO: change this key to something more descriptive of what it actually is.
settings = MongoClientSettings.FromConnectionString(configuration.GetConnectionString("Bukkit"));
string path = config["CACertificatePath"];
if (!String.IsNullOrEmpty(path))
{
this.ca = X509Certificate.CreateFromCertFile(path);
this.logger.LogDebug($"CA certificate read successfully from {path}");
settings.SslSettings = new SslSettings
{
ServerCertificateValidationCallback = this.ValidateServerCertificate
};
this.logger.LogDebug("CA certificate detected, SSL settings set to handle validation");
}
}
public MongoClient NewClient()
{
this.logger.LogDebug($"connecting to mongo at {settings.Server}");
return new MongoClient(settings);
}
private bool ValidateServerCertificate(object sender, X509Certificate? cert, X509Chain? chain, SslPolicyErrors sslPolicyErrors)
{
if (sslPolicyErrors == SslPolicyErrors.None)
{
this.logger.LogDebug("found no SSL policy errors, cert valid");
return true;
}
if (this.ca == null)
{
this.logger.LogError("loaded CA certificate is null, cert invalid");
return false;
}
if (chain == null)
{
this.logger.LogError("certificate chain is null, cert invalid");
return false;
}
// I guess the root is the last certificate in the chain???
X509ChainElement root = chain.ChainElements[chain.ChainElements.Count - 1];
if (this.ca.GetRawCertDataString().Equals(root.Certificate.GetRawCertDataString()))
{
this.logger.LogDebug("certificate root matches loaded ca certificate, cert valid");
return true;
}
this.logger.LogWarning($"CA cert didn't match");
this.logger.LogWarning($"SSL policy error {sslPolicyErrors} cannot be handled");
return false;
}
}
}
In my logs, I see the messages confirming everything is working. I see this for every query:
certificate root matches loaded ca certificate, cert valid
However, my logs are also polluted with this:
fail: App.src.Filters.JsonExceptionFilter[0]
Unhandled Exception: System.TimeoutException: A timeout occurred after 30000ms selecting a server using CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 }, OperationsCountServerSelector }. Client view of cluster state is { ClusterId : "2", ConnectionMode : "ReplicaSet", Type : "ReplicaSet", State : "Disconnected", Servers : [{ ServerId: "{ ClusterId : 2, EndPoint : "Unspecified/REDACTED:27017" }", EndPoint: "Unspecified/REDACTED:27017", ReasonChanged: "Heartbeat", State: "Disconnected", ServerVersion: , TopologyVersion: , Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server.
---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: UntrustedRoot
at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
at MongoDB.Driver.Core.Connections.SslStreamFactory.CreateStreamAsync(EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Servers.ServerMonitor.InitializeConnectionAsync(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Servers.ServerMonitor.HeartbeatAsync(CancellationToken cancellationToken)", LastHeartbeatTimestamp: "2022-03-11T20:56:09.1136007Z", LastUpdateTimestamp: "2022-03-11T20:56:09.1136011Z" }] }.
at MongoDB.Driver.Core.Clusters.Cluster.ThrowTimeoutException(IServerSelector selector, ClusterDescription description)
at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedHelper.HandleCompletedTask(Task completedTask)
at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedAsync(IServerSelector selector, ClusterDescription description, Task descriptionChangedTask, TimeSpan timeout, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Clusters.Cluster.SelectServerAsync(IServerSelector selector, CancellationToken cancellationToken)
at MongoDB.Driver.MongoClient.AreSessionsSupportedAfterServerSelectionAsync(CancellationToken cancellationToken)
at MongoDB.Driver.MongoClient.AreSessionsSupportedAsync(CancellationToken cancellationToken)
at MongoDB.Driver.MongoClient.StartImplicitSessionAsync(CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
at MongoDB.Driver.IAsyncCursorSourceExtensions.ToListAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)
at Bukkit.src.Repositories.FeatureToggleRepository.FindAllAsync(Expression`1 filter, Nullable`1 page, Nullable`1 size, Expression`1 sort, SortOrder sortOrder) in /app/src/Repositories/FeatureToggleRepository.cs:line 70
at Bukkit.src.Services.FeatureToggleService.GetEnabledFeatureToggles() in /app/src/Services/FeatureToggleService.cs:line 61
at Bukkit.src.Controllers.FeatureToggleController.GetEnabledToggles() in /app/src/Controllers/FeatureToggleController.cs:line 37
at lambda_method29(Closure , Object )
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
What could be causing this error?

After going nuts over this for a while, I happened to open a file for a class that was creating Mongo clients without using the MongoClientFactory above. Changing that worked.

Seems like the global exception handler is rethrowing the **BadHttpRequestException: Unexpected end of request content** as unhandled exception

My ASP .NET Core WebAPI application is getting some unidentified requests resulting in the below error.
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content.
This error is getting logged only at the kestrel (stdout_logfile) and not in the application's error log. Seems like the global exception handler is rethrowing the exception as unhandled exception.
As the request keeps coming, the application becomes slow and gets crashed, ended up in restarting the supervisor to bring up the application.
Error log:
^[[41m^[[30mfail^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
An unhandled exception has occurred while executing the request.
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content.
at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value)
at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
Code
Exception Filtering code:
public void ConfigureServices(IServiceCollection services)
{
...
...
services.AddControllers()
.ConfigureApiBehaviorOptions(options =>
{
options.SuppressConsumesConstraintForFormFileParameters = true;
options.SuppressInferBindingSourcesForParameters = true;
options.SuppressModelStateInvalidFilter = true;
options.SuppressMapClientErrors = true;
options.ClientErrorMapping[StatusCodes.Status404NotFound].Link =
"https://httpstatuses.com/404";
options.InvalidModelStateResponseFactory = context =>
{
var result = new BadRequestObjectResult(context.ModelState);
// TODO: add `using System.Net.Mime;` to resolve MediaTypeNames
result.ContentTypes.Add(MediaTypeNames.Application.Json);
result.ContentTypes.Add(MediaTypeNames.Application.Xml);
return result;
};
});
services.AddControllers(options =>
options.Filters.Add(new HttpResponseExceptionFilter())
);
...
...
}
Why is error not caught at the below global handler, or why is the error getting rethrown as unhandled exception. What am I doing wrong here?
Exception handler code:
public void OnActionExecuted(ActionExecutedContext context)
{
if (context.Exception is HttpResponseException exception)
{
logger.Error(context.Exception);
context.Result = new ObjectResult(exception.Value)
{
StatusCode = exception.Status,
};
context.ExceptionHandled = true;
}
else if (context.Exception is BadHttpRequestException badException)
{
logger.Error(context.Exception);
context.Result = new ObjectResult(badException.Data.Values)
{
StatusCode = badException.StatusCode,
};
context.ExceptionHandled = true;
}
else if (context.Exception is Exception otherException)
{
logger.Error(context.Exception);
context.Result = new ObjectResult(otherException.Data.Values)
{
StatusCode = otherException.HResult,
};
context.ExceptionHandled = true;
}
}

'Stream was already consumed' error using Polly to retry requests in ASP.NET Core

I have an ASP.NET Core 3.1 Web API service that is receiving a web request, doing some manipulations to it, and then passing it on to a backend service and returning the response synchronously. It was working fine but I wanted to introduce some retry logic to those backend requests in case some blips occurred.
I'm using a typed HttpClient and attempting to use Polly to implement the retry logic:
https://github.com/App-vNext/Polly/wiki/Polly-and-HttpClientFactory#using-polly-with-ihttpclientfactory
When the backend service works, everything seems to be fine but unfortunately, whenever my backend returns an error like a 500 Internal Server Error, I get the following exception:
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware: Error: An unhandled exception has occurred while executing the request.
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.InvalidOperationException: The stream was already consumed. It cannot be read again.
at System.Net.Http.StreamContent.PrepareContent()
at System.Net.Http.StreamContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpContent.CopyToAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at MyProject.MyController.Routing.HttpMessageRouter.SendRequestAndTrackTiming(Func`1 action, String destinationID) in /mnt/c/Users/myCode/source/repos/MyProject-GitLab/MyController/src/MyController/Routing/HttpMessageRouter.cs:line 59
at MyProject.MyController.Routing.HttpMessageRouter.SendNewRequest(IMessageWrapper`1 message) in /mnt/c/Users/myCode/source/repos/MyProject-GitLab/MyController/src/MyController/Routing/HttpMessageRouter.cs:line 33
at MyProject.MyController.Controllers.MyControllerController.Resource(String destinationId) in /mnt/c/Users/myCode/source/repos/MyProject-GitLab/MyController/src/MyController/Controllers/MyControllerController.cs:line 151
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Prometheus.HttpMetrics.HttpRequestDurationMiddleware.Invoke(HttpContext context)
at Prometheus.HttpMetrics.HttpRequestCountMiddleware.Invoke(HttpContext context)
at Prometheus.HttpMetrics.HttpInProgressMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
The code I have to send the request is as follows. This first method actually just utilizes a delegate so I can transparently add some metrics gathering around the typed HttpClient call. The typed HttpClient is called RoutingClient in this code:
public async Task<IActionResult> SendNewRequest(IMessageWrapper<HttpRequestMessage> message)
{
HttpResponseMessage destinationResponse = await SendRequestAndTrackTiming(() => _client.SendAsync(message.Message), message.DestinationID);
return CreateSerializeableResponseMessage(destinationResponse);
}
private ResponseMessageResult CreateSerializeableResponseMessage(HttpResponseMessage httpResponse)
{
ResponseMessageResult responseMessage = new ResponseMessageResult(httpResponse);
IOutputFormatter[] formattersList = { new HttpResponseMessageOutputFormatter() };
FormatterCollection<IOutputFormatter> formattersCollection = new FormatterCollection<IOutputFormatter>(formattersList);
responseMessage.Formatters = formattersCollection;
return responseMessage;
}
private async Task<HttpResponseMessage> SendRequestAndTrackTiming(Func<Task<HttpResponseMessage>> action, string destinationID)
{
HttpResponseMessage response = null;
Stopwatch stopwatch = new Stopwatch();
try
{
stopwatch.Start();
response = await action();
return response;
}
finally
{
stopwatch.Stop();
HttpStatusCode statusCode = (response != null) ? response.StatusCode : HttpStatusCode.InternalServerError;
_routedMessageMetricTracker.Histogram.Observe(destinationID, statusCode, stopwatch.Elapsed.TotalSeconds);
}
}
This is my code in Startup.ConfigureServices() (actually this is in an extension method I defined):
public static IServiceCollection AddRoutingClient(this IServiceCollection services, RoutingSettings routingSettings)
{
List<TimeSpan> retryTimeSpans = new List<TimeSpan>();
// routingSettings.RetrySeconds is just an array of double values.
foreach (double retrySeconds in routingSettings.RetrySeconds)
{
if (retrySeconds >= 0) retryTimeSpans.Add(TimeSpan.FromSeconds(retrySeconds));
}
services.AddHttpClient<IRoutingClient, RoutingClient>()
.AddPolicyHandler((services, request) => HttpPolicyExtensions.HandleTransientHttpError()
.WaitAndRetryAsync(retryTimeSpans, onRetry: (outcome, timespan, retryAttempt, context) =>
{
services.GetService<ILogger<RoutingClient>>()?
.LogWarning($"Delaying for {timespan.TotalMilliseconds}ms, then making retry {retryAttempt}.");
}
));
return services;
}
I'd really like to use the Polly approach because it seems clean but I am not sure what I am doing wrong here. I must be doing something wrong because I would expect this to be a very common use-case for Polly that should be handled.

It turns out my problem was not caused by the code dealing with the response above. Instead, it was actually caused by my code that was manipulating the request.
I was reading the incoming request as an HttpRequestMessage object via the HttpContext.GetHttpRequestMessage() method and I attempted to re-use that same object to pass on to the backend service via a call to my typed HttpClient. However, the content stream for that request is read-once so I had to make a copy of that HttpRequestMessage as described in this answer to another post: https://stackoverflow.com/a/34049029/1221718
Here is a slightly more verbose version of that answer's code:
private static async Task<HttpRequestMessage> CloneHttpRequestMessageAsync(HttpRequestMessage request)
{
HttpRequestMessage copyOfRequest = new HttpRequestMessage(request.Method, request.RequestUri);
// Copy the request's content (via a MemoryStream) into the cloned object
var ms = new MemoryStream();
if (request.Content != null)
{
await request.Content.CopyToAsync(ms).ConfigureAwait(false);
ms.Position = 0;
copyOfRequest.Content = new StreamContent(ms);
// Copy the content headers
if (request.Content.Headers != null)
{
foreach (var h in request.Content.Headers)
{
copyOfRequest.Content.Headers.Add(h.Key, h.Value);
}
}
}
copyOfRequest.Version = request.Version;
foreach (KeyValuePair<string, object> prop in request.Properties)
{
copyOfRequest.Properties.Add(prop);
}
foreach (KeyValuePair<string, IEnumerable<string>> header in request.Headers)
{
copyOfRequest.Headers.TryAddWithoutValidation(header.Key, header.Value);
}
return copyOfRequest;
}

Another solution to this problem is to enable buffering on the request stream.
This is done by calling the extension method EnableBuffering() for the HttpRequest, e.g.
public async Task InvokeAsync(HttpContext context, RequestDelegate next)
{
context.Request.EnableBuffering();
...
see https://devblogs.microsoft.com/dotnet/re-reading-asp-net-core-request-bodies-with-enablebuffering/ for more detail.

Microsoft Bot v4 Skill Bot responds with 500 (Internal Server)

I am using Microsoft Bot framework v4 (CoreBot template) with Master Bot and a Child Bot.
I have modified the sample code and as able to establish the connection between the Child bot and the Master Bot. To be precise, the request from Master bot hits the Child bot successfully. But, Master bot receives 500 Internal Server error as response from the Child Bot. But the Child bot works as expected independently.
This is the error message I get at Master Bot:
An unhandled exception occurred while processing the request.
ErrorResponseException: Operation returned an invalid status code 'InternalServerError'
Microsoft.Bot.Connector.Conversations.ReplyToActivityWithHttpMessagesAsync(string conversationId, string activityId, Activity activity, Dictionary<string, List<string>> customHeaders, CancellationToken cancellationToken) in Conversations.cs, line 1121
Stack
Query
Cookies
Headers
ErrorResponseException: Operation returned an invalid status code 'InternalServerError'
Microsoft.Bot.Connector.Conversations.ReplyToActivityWithHttpMessagesAsync(string conversationId, string activityId, Activity activity, Dictionary<string, List<string>> customHeaders, CancellationToken cancellationToken) in Conversations.cs
Microsoft.Bot.Connector.ConversationsExtensions.ReplyToActivityAsync(IConversations operations, string conversationId, string activityId, Activity activity, CancellationToken cancellationToken) in ConversationsExtensions.cs
Microsoft.Bot.Builder.BotFrameworkAdapter.SendActivitiesAsync(ITurnContext turnContext, Activity[] activities, CancellationToken cancellationToken) in BotFrameworkAdapter.cs
Microsoft.Bot.Builder.TurnContext+<>c__DisplayClass22_0+<<SendActivitiesAsync>g__SendActivitiesThroughAdapter|1>d.MoveNext() in TurnContext.cs
Microsoft.Bot.Builder.TurnContext.SendActivityAsync(IActivity activity, CancellationToken cancellationToken) in TurnContext.cs
MySampleBot.AdapterWithErrorHandler+<>c__DisplayClass0_0+<<-ctor>b__0>d.MoveNext() in AdapterWithErrorHandler.cs
+
23. // Log any leaked exception from the application.
24. logger.LogError(exception, $"[OnTurnError] unhandled error : {exception.Message}");
25.
26. // Send a message to the user
27. var errorMessageText = ChildBot Bot encountered an error or bug.";
28. var errorMessage = MessageFactory.Text(errorMessageText, errorMessageText, InputHints.ExpectingInput);
29. await turnContext.SendActivityAsync(errorMessage);
30.
31. errorMessageText = "To continue to run this Helpdesk Bot , please fix the bot source code.";
32. errorMessage = MessageFactory.Text(errorMessageText, errorMessageText, InputHints.ExpectingInput);
33. await turnContext.SendActivityAsync(errorMessage);
34.
35. if (conversationState != null)
Microsoft.Bot.Builder.BotAdapter.RunPipelineAsync(ITurnContext turnContext, BotCallbackHandler callback, CancellationToken cancellationToken) in BotAdapter.cs
Microsoft.Bot.Builder.BotFrameworkAdapter.ProcessActivityAsync(ClaimsIdentity claimsIdentity, Activity activity, BotCallbackHandler callback, CancellationToken cancellationToken) in BotFrameworkAdapter.cs
Microsoft.Bot.Builder.BotFrameworkAdapter.ProcessActivityAsync(string authHeader, Activity activity, BotCallbackHandler callback, CancellationToken cancellationToken) in BotFrameworkAdapter.cs
Microsoft.Bot.Builder.Integration.AspNet.Core.BotFrameworkHttpAdapter.ProcessAsync(HttpRequest httpRequest, HttpResponse httpResponse, IBot bot, CancellationToken cancellationToken) in BotFrameworkHttpAdapter.cs
MySampleBot.Controllers.BotController.PostAsync() in BotController.cs
+
28.
29. [HttpPost, HttpGet]
30. public async Task PostAsync()
31. {
32. // Delegate the processing of the HTTP POST to the adapter.
33. // The adapter will invoke the bot.
34. await Adapter.ProcessAsync(Request, Response, Bot);
35. }
lambda_method(Closure , object )
Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable+Awaiter.GetResult()
Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor+AwaitableResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, object controller, object[] arguments)
System.Threading.Tasks.ValueTask<TResult>.get_Result()
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
Show raw exception details
Microsoft.Bot.Schema.ErrorResponseException: Operation returned an invalid status code 'InternalServerError'
at Microsoft.Bot.Connector.Conversations.ReplyToActivityWithHttpMessagesAsync(String conversationId, String activityId, Activity activity, Dictionary`2 customHeaders, CancellationToken cancellationToken) in d:\a\1\s\libraries\Microsoft.Bot.Connector\Conversations.cs:line 1121
at Microsoft.Bot.Connector.ConversationsExtensions.ReplyToActivityAsync(IConversations operations, String conversationId, String activityId, Activity activity, CancellationToken cancellationToken) in d:\a\1\s\libraries\Microsoft.Bot.Connector\ConversationsExtensions.cs:line 241
at Microsoft.Bot.Builder.BotFrameworkAdapter.SendActivitiesAsync(ITurnContext turnContext, Activity[] activities, CancellationToken cancellationToken) in d:\a\1\s\libraries\Microsoft.Bot.Builder\BotFrameworkAdapter.cs:line 499
at Microsoft.Bot.Builder.TurnContext.<>c__DisplayClass22_0.<<SendActivitiesAsync>g__SendActivitiesThroughAdapter|1>d.MoveNext() in d:\a\1\s\libraries\Microsoft.Bot.Builder\TurnContext.cs:line 267
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Bot.Builder.TurnContext.SendActivityAsync(IActivity activity, CancellationToken cancellationToken) in d:\a\1\s\libraries\Microsoft.Bot.Builder\TurnContext.cs:line 196
at MySampleBot.AdapterWithErrorHandler.<>c__DisplayClass0_0.<<-ctor>b__0>d.MoveNext() in C:\Users\adminMCL.PL\source\repos\MySampleBot\AdapterWithErrorHandler.cs:line 29
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Bot.Builder.BotAdapter.RunPipelineAsync(ITurnContext turnContext, BotCallbackHandler callback, CancellationToken cancellationToken) in d:\a\1\s\libraries\Microsoft.Bot.Builder\BotAdapter.cs:line 210
at Microsoft.Bot.Builder.BotFrameworkAdapter.ProcessActivityAsync(ClaimsIdentity claimsIdentity, Activity activity, BotCallbackHandler callback, CancellationToken cancellationToken) in d:\a\1\s\libraries\Microsoft.Bot.Builder\BotFrameworkAdapter.cs:line 393
at Microsoft.Bot.Builder.BotFrameworkAdapter.ProcessActivityAsync(String authHeader, Activity activity, BotCallbackHandler callback, CancellationToken cancellationToken) in d:\a\1\s\libraries\Microsoft.Bot.Builder\BotFrameworkAdapter.cs:line 363
at Microsoft.Bot.Builder.Integration.AspNet.Core.BotFrameworkHttpAdapter.ProcessAsync(HttpRequest httpRequest, HttpResponse httpResponse, IBot bot, CancellationToken cancellationToken) in d:\a\1\s\libraries\integration\Microsoft.Bot.Builder.Integration.AspNet.Core\BotFrameworkHttpAdapter.cs:line 133
at MySampleBot.Controllers.BotController.PostAsync() in C:\Users\adminMCL.PL\source\repos\MySampleBot\Controllers\BotController.cs:line 34
at lambda_method(Closure , Object )
at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()
at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.AwaitableResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at System.Threading.Tasks.ValueTask`1.get_Result()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
What could be a possible reason for such issue ? I do not understand the root cause for this issue at the Child Bot while it works independently without any error.
Please help if you can observe if something is wrong in the code.
Thanks.

Occasional exception writing compressed content via WebApi to the client in streamed transfer mode

We have a WEPAPI service, which writes data in compressed form to the client using a message handler. Since we occasional receive very large messages, the system is configured using
HttpSelfHostConfiguration.TransferMode = TransferMode.Streamed;
The compress logic is:
public class CompressedHttpContent : CompressionHttpContent
{
public CompressedHttpContent(HttpContent content, ICompressor compressor)
: base(content, compressor)
{
}
protected async override Task SerializeToStreamAsync(System.IO.Stream stream, System.Net.TransportContext context)
{
try
{
if (this.OriginalContent != null)
using (OriginalContent)
{
using (var contentStream = await OriginalContent.ReadAsStreamAsync())
{
var compressedStream = Compressor.CreateCompressionStream(stream);
await contentStream.CopyToAsync(compressedStream);
compressedStream.Close();
compressedStream.Dispose();
}
}
}
catch (Exception e)
{
...
throw;
}
}
}
And occasionally we get an exception like:
System.InvalidOperationException: The AsyncEventArgs implementation 'System.ServiceModel.Channels.BufferedOutputAsyncStream+WriteAsyncState' tried to set the state to Pending multiple times without completing a pending operation. This could be caused by an incorrect application AsyncEventArgs implementation or other extensibility code that invokes Set() multiple times.
Server stack trace:
at System.Runtime.AsyncEventArgs.set_State(OperationState value)
at System.Runtime.AsyncEventArgs.SetAsyncState(AsyncEventArgsCallback callback, Object state)
at System.Runtime.AsyncEventArgs`1.Set(AsyncEventArgsCallback callback, TArgument arguments, Object state)
at System.ServiceModel.Channels.BufferedOutputAsyncStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncCallback callback, Object state)
at System.IO.Compression.DeflateStream.WriteDeflaterOutput(Boolean isAsync)
at System.IO.Compression.DeflateStream.InternalWrite(Byte[] array, Int32 offset, Int32 count, Boolean isAsync)
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)
Exception rethrown at [0]:
at System.IO.Compression.DeflateStream.EndWrite(IAsyncResult asyncResult)
at System.IO.Compression.GZipStream.EndWrite(IAsyncResult asyncResult)
at System.IO.Stream.<>c.<BeginEndWriteAsync>b__53_1(Stream stream, IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.IO.Stream.<CopyToAsyncInternal>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CompressedHttpContent.<SerializeToStreamAsync>d__2.MoveNext()
Unfortunately, when I search for this error, I don't get any hits.
We didn't get this when we were using the buffered transfer mode. We could potentially change to StreamedRequest mode, as it's only the incoming requests which can be very large.
Has anyone seen this or have any thoughts?
Thanks.

When experimenting with compressed responses, I encountered the same error.
What I found interesting was the following bit from the source code of BufferedOutputAsyncStream, which appears in the call stack:
" BufferedOutputAsyncStream is not a general-purpose stream wrapper, it requires that the base stream
1. allow concurrent IO (for multiple BeginWrite calls)
2. support the BeginWrite,BeginWrite,BeginWrite,.. Close() calling pattern.
Currently BufferedOutputAsyncStream only used to wrap the System.Net.HttpResponseStream, which satisfy both requirements."
(https://referencesource.microsoft.com/#System.ServiceModel/System/ServiceModel/Channels/BufferedOutputAsyncStream.cs)
This requirement does not appear to be fulfilled for the GZipStream, as can be tested with some simple code (generally throws an exception).
byte[] ba = File.ReadAllBytes(#"largerfile");
GZipStream gzs = new GZipStream(new MemoryStream(), CompressionLevel.Optimal);
gzs.WriteAsync(ba,0, 200000);
gzs.WriteAsync(ba, 0, 200000);
I suspect that ASP.NET WebAPI may be calling the GZipStream in an unsupported manner - although I am not yet convinced that this is actually what's happening.