We Keep Coding

Microsoft Exchange oAuth - System.OutOfMemoryException

I've written a console app thats running as a service (using TopShelf) and using a while loop it continuously polls an office 365 inbox every 30 seconds to check for new messages. I'm doing this using oAuth and the below Microsoft libary
Microsoft.Exchange.WebServices 2.2.0.
I'm using the reccomended approach from MS to get the Access Token silently AcquireTokenByUsernamePassword & AcquireTokenSilent.
After about 5 days of running perfectly I'm getting the below exception form Microsoft library:
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.String.Concat(String str0, String str1)
at Microsoft.Exchange.WebServices.Data.OAuthCredentials..ctor(String token, Boolean verbatim) in \\REDMOND\EXCHANGE\BUILD\E15\15.00.0913.015\SOURCES\sources\dev\EwsManagedApi\src\EwsManagedApi\Credentials\OAuthCredentials.cs:line 79
at Microsoft.Exchange.WebServices.Data.OAuthCredentials..ctor(String token) in \\REDMOND\EXCHANGE\BUILD\E15\15.00.0913.015\SOURCES\sources\dev\EwsManagedApi\src\EwsManagedApi\Credentials\OAuthCredentials.cs:line 36
When I've traced this through its being generated by this section in my code:
exchangeService = new ExchangeService
{
Credentials = new OAuthCredentials(authResult.AccessToken),
Url = new Uri(rdr.O365ServiceURL)
};
I've got my exchangeService object declared outside of my while loop and then I'm instantiating it inside the loop so I can make use of the AcquireTokenSilent call, otherwise Microsoft refuse the connection and issue the below message back:
Error: Your app has been throttled by AAD due to too many requests. To avoid this, cache your tokens see https://aka.ms/msal-net-throttling.
Is this a bug in Microsoft code or can I do something better to manage the memory here?
Also the service that crashed out was upto 3.6gb memory footprint which is about 3.5gb too high.

There was nothing wrong with the code in question. Whilst not ideal (I'm going to look I to the notification stream approach instead of polling thanks #paulsm4) it still worked and the Exchange Service was not disposable, so could not be instantiated with a using block.
The issue was actually in the log4net framework where I had the memory appender enabled. I removed this from my logger setup and the memory footprint stays where it should be.
This project uses three separate loggers and they were all building up in memory even after it was written to the file.
The instantiating of the exchange object was just the straw that broke the camels back and a red herring.
The app is sitting at a health 14MB memory footprint now.
Thanks!

Azure Service Bus MessageLockLostException when Completing Locked Message

I'm getting a MessageLockLostException when performing a complete operation on Azure Service Bus after performing a long operation of 30 minutes to over an hour. I want this process to scale and be resilient to failures so I keep hold of the Message lock and renew it well within the default lock duration of 1 minute. However when I try to complete the message at the end, even though I can see all the lock renewals have occurred at the correct time I get a MessageLockLostException. I want to scale this up in the future however there is currently only one instance of the application and I can confirm that the message still exists on the Service Bus Subscription after it errors so the problem is definitely around the lock.
Here are the steps I take.
Obtain a message and configure a lock
messages = await Receiver.ReceiveAsync(1, TimeSpan.FromSeconds(10)).ConfigureAwait(false);
var message = messages[0];
var messageBody = GetTypedMessageContent(message);
Messages.TryAdd(messageBody, message);
LockTimers.TryAdd(
messageBody,
new Timer(
async _ =>
{
if (Messages.TryGetValue(messageBody, out var msg))
{
await Receiver.RenewLockAsync(msg.SystemProperties.LockToken).ConfigureAwait(false);
}
},
null,
TimeSpan.FromSeconds(Config.ReceiverInfo.LockRenewalTimeThreshold),
TimeSpan.FromSeconds(Config.ReceiverInfo.LockRenewalTimeThreshold)));
Perform the long running process
Complete the message
internal async Task Complete(T message)
{
if (Messages.TryGetValue(message, out var msg))
{
await Receiver.RenewLockAsync(msg.SystemProperties.LockToken);
await Receiver.CompleteAsync(msg.SystemProperties.LockToken).ConfigureAwait(false);
}
}
The code above is a stripped down version of what's there, I removed some try catch error handling and logging we have but I can confirm that when debugging the issue I can see the timer execute on time. It's just the "CompleteAsync" that fails.
Additional Info;
Service Bus Topic has Partitioning Enabled
I have tried renewing it at 80% of the threshold (48 seconds), 30% of the Threshold (18 seconds) and 10% of the Threshold (6 seconds)
I've searched around for an answer and the closest thing I found was this article but it's from 2016.
I couldn't get it to fail in a standalone Console Application so I don't know if it's something I'm doing in my Application but I can confirm that the lock renewal occurs for the duration of the processing and returns the correct DateTime for the updated lock, I'd expect if the lock was truely lost that the CompleteAsync would fail
I'm using the Microsoft.Azure.ServiceBus nuget package Version="4.1.3"
My Application is Dotnet Core 3.1 and uses a Service Bus Wrapper Package which is written in Dotnet Standard 2.1
The message completes if you don't hold onto it for a long time and occasionally completes even when you do.
Any help or advice on how I could complete my Service Bus message successfully after an hour would be great

The issue here wasn't with my code. It was with Partitioning on the Service Bus topic. If you search around there are some issues on the Microsoft GitHub around completion of messages. That's not important anyway because the fix I used here was to use the Subscription forwarding feature to move the message to a new Topic with partitioning disabled and then read the message from that new topic and I was able to use the exact same code to keep the message locked for a long time and still complete it successfully

HttpClient concurrent behavior different when running in Powershell than in Visual Studio

I'm migrating millions of users from on-prem AD to Azure AD B2C using MS Graph API to create the users in B2C. I've written a .Net Core 3.1 console application to perform this migration. To speed things along I'm making concurrent calls to the Graph API. This is working great - sort of.
During development I experienced acceptable performance while running from Visual Studio 2019, but for test I'm running from the command line in Powershell 7. From Powershell the performance of concurrent calls to the HttpClient is very bad. It appears that there's a limit to the number of concurrent calls that HttpClient is allowing when running from Powershell, so calls in concurrent batches greater than 40 to 50 requests start to stack up. It seems to be running 40 to 50 concurrent requests while blocking the rest.
I'm not looking for assistance with async programming. I'm looking for a way to trouble shoot the difference between Visual Studio run-time behavior and Powershell command line run-time behavior. Running in release mode from Visual Studio's green arrow button behaves as expected. Running from the command line does not.
I fill a task list with async calls and then await Task.WhenAll(tasks). Each call takes between 300 and 400 milliseconds. When running from Visual Studio it works as expected. I make concurrent batches of 1000 calls and each individually completes within the expected time. The whole task block takes just a few milliseconds longer than the longest individual call.
The behavior changes when I run the same build from the Powershell command line. The first 40 to 50 calls take the expected 300 to 400 milliseconds but then the individual call times grow up to 20 seconds each. I think the calls are serializing, so only 40 to 50 are being executed at a time while the others wait.
After hours of trial and error I was able to narrow it down to the HttpClient. To isolate the problem I mocked the calls to HttpClient.SendAsync with a method that does Task.Delay(300) and returns a mock result. In this case running from the console behaves identically to running from Visual Studio.
I'm using IHttpClientFactory and I've even tried adjusting the connection limit on ServicePointManager.
Here's my registration code.
public static IServiceCollection RegisterHttpClient(this IServiceCollection services, int batchSize)
{
ServicePointManager.DefaultConnectionLimit = batchSize;
ServicePointManager.MaxServicePoints = batchSize;
ServicePointManager.SetTcpKeepAlive(true, 1000, 5000);
services.AddHttpClient(MSGraphRequestManager.HttpClientName, c =>
{
c.Timeout = TimeSpan.FromSeconds(360);
c.DefaultRequestHeaders.Add("User-Agent", "xxxxxxxxxxxx");
})
.ConfigurePrimaryHttpMessageHandler(() => new DefaultHttpClientHandler(batchSize));
return services;
}
Here's the DefaultHttpClientHandler.
internal class DefaultHttpClientHandler : HttpClientHandler
{
public DefaultHttpClientHandler(int maxConnections)
{
this.MaxConnectionsPerServer = maxConnections;
this.UseProxy = false;
this.AutomaticDecompression = System.Net.DecompressionMethods.GZip | System.Net.DecompressionMethods.Deflate;
}
}
Here's the code that sets up the tasks.
var timer = Stopwatch.StartNew();
var tasks = new Task<(UpsertUserResult, TimeSpan)>[users.Length];
for (var i = 0; i < users.Length; ++i)
{
tasks[i] = this.CreateUserAsync(users[i]);
}
var results = await Task.WhenAll(tasks);
timer.Stop();
Here's how I mocked out the HttpClient.
var httpClient = this.httpClientFactory.CreateClient(HttpClientName);
#if use_http
using var response = await httpClient.SendAsync(request);
#else
await Task.Delay(300);
var graphUser = new User { Id = "mockid" };
using var response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new StringContent(JsonConvert.SerializeObject(graphUser)) };
#endif
var responseContent = await response.Content.ReadAsStringAsync();
Here are metrics for 10k B2C users created via GraphAPI using 500 concurrent requests. The first 500 requests are longer than normal because the TCP connections are being created.
Here's a link to the console run metrics.
Here's a link to the Visual Studio run metrics.
The block times in the VS run metrics are different than what I said in this post because I moved all the synchronous file access to the end of the process in an effort to isolate the problematic code as much as possible for the test runs.
The project is compiled using .Net Core 3.1. I'm using Visual Studio 2019 16.4.5.

Two things come to mind. Most microsoft powershell was written in version 1 and 2. Version 1 and 2 have System.Threading.Thread.ApartmentState of MTA. In version 3 through 5 the apartment state changed to STA by default.
The second thought is it sounds like they are using System.Threading.ThreadPool to manage the threads. How big is your threadpool?
If those do not solve the issue start digging under System.Threading.
When I read your question I thought of this blog. https://devblogs.microsoft.com/oldnewthing/20170623-00/?p=96455
A colleague demonstrated with a sample program that creates a thousand work items, each of which simulates a network call that takes 500ms to complete. In the first demonstration, the network calls were blocking synchronous calls, and the sample program limited the thread pool to ten threads in order to make the effect more apparent. Under this configuration, the first few work items were quickly dispatched to threads, but then the latency started to build as there were no more threads available to service new work items, so the remaining work items had to wait longer and longer for a thread to become available to service it. The average latency to the start of the work item was over two minutes.
Update 1:
I ran PowerShell 7.0 from the start menu and the thread state was STA. Is the thread state different in the two versions?
PS C:\Program Files\PowerShell\7> [System.Threading.Thread]::CurrentThread
ManagedThreadId : 12
IsAlive : True
IsBackground : False
IsThreadPoolThread : False
Priority : Normal
ThreadState : Running
CurrentCulture : en-US
CurrentUICulture : en-US
ExecutionContext : System.Threading.ExecutionContext
Name : Pipeline Execution Thread
ApartmentState : STA
Update 2:
I wish better answer but, you will have compare the two environments till something stands out.
PS C:\Windows\system32> [System.Net.ServicePointManager].GetProperties() | select name
Name
----
SecurityProtocol
MaxServicePoints
DefaultConnectionLimit
MaxServicePointIdleTime
UseNagleAlgorithm
Expect100Continue
EnableDnsRoundRobin
DnsRefreshTimeout
CertificatePolicy
ServerCertificateValidationCallback
ReusePort
CheckCertificateRevocationList
EncryptionPolicy
Update 3:
https://learn.microsoft.com/en-us/uwp/api/windows.web.http.httpclient
In addition, every HttpClient instance uses its own connection pool,
isolating its requests from requests executed by other HttpClient
instances.
If an app using HttpClient and related classes in the Windows.Web.Http
namespace downloads large amounts of data (50 megabytes or more), then
the app should stream those downloads and not use the default
buffering. If the default buffering is used the client memory usage
will get very large, potentially resulting in reduced performance.
Just keep comparing the two environments and the issue should stand out
Add-Type -AssemblyName System.Net.Http
$client = New-Object -TypeName System.Net.Http.Httpclient
$client | format-list *
DefaultRequestHeaders : {}
BaseAddress :
Timeout : 00:01:40
MaxResponseContentBufferSize : 2147483647

How to programmatically mark an Azure WebJob as failed?

Is there a way to mark a WebJob (triggered, not continuous) as failed, without throwing an exception? I need to check that certain conditions are true to mark the job as successful.

According to Azure WebJob SDK, Code from TriggeredFunctionExecutor class.
public async Task<FunctionResult> TryExecuteAsync(TriggeredFunctionData input, CancellationToken cancellationToken)
{
IFunctionInstance instance = _instanceFactory.Create((TTriggerValue)input.TriggerValue, input.ParentId);
IDelayedException exception = await _executor.TryExecuteAsync(instance, cancellationToken);
FunctionResult result = exception != null ?
new FunctionResult(exception.Exception)
: new FunctionResult(true);
return result;
}
We know that the WebJobs status depends on whether your WebJob/Function is executed without any exceptions or not. We can't set the finial status of a running WebJob programmatically.
I need to check that certain conditions are true to mark the job as successful.
Throw an exception is the only way I found. Or you could store the webjob execute result in an additional place(For example, Azure Table Storage). We can get the current invocation id by ExecutionContext class. In your webjob, you could save the current invocation id and the status you wanted to an Azure Table Storage. You could query the status later if you needed from Azure Table Storage based on the invocation id.
public static void ProcessQueueMessage([QueueTrigger("myqueue")] string message, ExecutionContext context, TextWriter log)
{
log.WriteLine(message);
SaveStatusToTableStorage(context.InvocationId, "Fail/Success");
}
To use ExecutionContext as parameter, you need to install Azure WebJobs SDK Extensions using NuGet and invoke UserCore method before your run your WebJob.
var config = new JobHostConfiguration();
config.UseCore();
var host = new JobHost(config);
host.RunAndBlock();

Throwing an unmanaged exception will result in a Failed execution.
But i have noticed that it will also result with a bad management of your message: i.e. your message will be dequeued but not moved to your poison queue regarding your configuration (but maybe it was due to my SDK version).

#Jean NETR-VALERE the newer versions of the WebJobs packages do act as you say and if an exception is thrown the job will fail, and will continue to be run over and over and over until you finally clear your queue. This is absolutely horrible behavior and I have no clue why they changed this.
Yes they did change it to make it work this way, because I use an older version of the webjobs package just for this reason. About 3 months ago I upgraded to the newer version, and shortly after could not understand why the above behavior was happening . Once I reverted back to the older version, it started working correctly again and after failing 5 times is moved to poison queue and never ran again. My point is that if you want the correct (IMO) behavior, see if you can go back to using version 1.1.0 and you will be happy. Hope that helps.

To mark a triggered web job as failed you just need to set process exit code to non-zero.
System.Environment.ExitCode = 1;
When you throw an unhandled exception it also sets the exit code, that is how Azure determines failure.

Service Bus message abandoned despite WebJobs SDK handler completed successfully

I have implemented a long running process as a WebJob using the WebJobs SDK.
The long running process is awaited because I want the result.
public async Task ProcessMessage([ServiceBusTrigger("queuename")] MyMessage message)
{
await Run(message.SomeProperty); // takes several minutes
// I want to do something with the result here later..
}
What I can't figure out is why the message sometimes is abandoned which of course triggers the handler again. I've tried to debug (locally), setting breakpoints before ProcessMessage finishes and I can see that it appears to finish successfully.
The Sevice Bus part of the WebJobs SDK takes care of message lock renewal, so that shouldn't be a problem as far as I've understood.
What am I missing and how do I troubleshoot?

[Edited previously incorrect response]
The WebJobs SDK relies on the automatic lock renewals done by MessageReceiver.OnMessageAsync. These renewals are governed by the OnMessageOptions.AutoRenewTimeout setting, which can be configured like so in the v1.1.0 release of the WebJobs SDK:
JobHostConfiguration config = new JobHostConfiguration();
ServiceBusConfiguration sbConfig = new ServiceBusConfiguration();
sbConfig.OnMessageOptions = new OnMessageOptions
{
MaxConcurrentCalls = 16,
AutoRenewTimeout = TimeSpan.FromMinutes(10)
};
config.UseServiceBus(sbConfig);
You can also customize these values via a custom MessageProcessor. See the release notes here for more details on these new features.