Short Question:
Has anyone else encountered an issue in using a singleton .NET HttpClient where the application pegs the processor at 100% until it's restarted?
Details:
I'm running a Windows Service that does continuous, schedule-based ETL. One of the data-syncing threads occasionally either just dies, or starts running out of control and pegs the processor at 100%.
I was lucky enough to see this happening live before someone simply restarted the service (the standard fix), and was able to grab a dump-file.
Loading this in WinDbg (w/ SOS and SOSEX), I found that I have about 15 threads (sub-tasks of the main processing thread) all running with identical stack-traces. However, there don't appear to be any deadlocks. I.E. the high-utilization threads are running, but never finishing.
The relevant stack-trace segment follows (addresses omitted):
System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].FindEntry(System.__Canon)
System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].TryGetValue(System.__Canon, System.__Canon ByRef)
System.Net.Http.Headers.HttpHeaders.ContainsParsedValue(System.String, System.Object)
System.Net.Http.Headers.HttpGeneralHeaders.get_TransferEncodingChunked()
System.Net.Http.Headers.HttpGeneralHeaders.AddSpecialsFrom(System.Net.Http.Headers.HttpGeneralHeaders)
System.Net.Http.Headers.HttpRequestHeaders.AddHeaders(System.Net.Http.Headers.HttpHeaders)
System.Net.Http.HttpClient.SendAsync(System.Net.Http.HttpRequestMessage, System.Net.Http.HttpCompletionOption, System.Threading.CancellationToken)
...
[Our Application Code]
According to this article (and others I've found), the use of dictionaries is not thread-safe, and infinite loops are possible (as are straight-up crashes) if you access a dictionary in a multi-threaded manner.
BUT our application code is not using a dictionary explicitly. So where is the dictionary mentioned in the stack-trace?
Following through via .NET Reflector, it appears that the HttpClient uses a dictionary to store any values that have been configured in the "DefaultRequestHeaders" property. Any request the gets sent through the HttpClient, therefore, triggers an enumeration of a singleton, non-thread-safe dictionary (in order to add the default headers to the request), which could potentially infinitely spin (or kill) the threads involved if a corruption occurs.
Microsoft has stated bluntly that the HttpClient class is thread-safe. But it seems to me like this is no longer true if any headers have been added to the DefaultRequestHeaders of the HttpClient.
My analysis seems to indicate that this is the real root problem, and an easy workaround is to simply never use the DefaultRequestHeaders where the HttpClient could be used in a multi-threaded manner.
However, I'm looking for some confirmation that I'm not barking up the wrong tree. If this is correct, it seems like a bug in the .NET framework, which I automatically tend to doubt.
Sorry for the wordy question, but thanks for any input you may have.
Thanks for all the comments; they got me thinking along different lines, and helped me find the ultimate root cause of the issue.
Although the issue was a result of corruption in the backing dictionary of the DefaultRequestHeaders, the real culprit was the initialization code for the HttpClient object:
private HttpClient InitializeClient()
{
if (_client == null)
{
_client = GetHttpClient();
_client.DefaultRequestHeaders.Accept.Clear();
_client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
SetBaseAddress(BaseAddress);
}
return _client;
}
I said that the HttpClient was a singleton, which is partially incorrect. It's created as a single-instance that is shared amongst multiple threads doing a unit of work, and is disposed when the work is complete. A new instance will be spun up the next time this particular task must be done.
The "InitializeClient" method above is called every time a request is to be sent, and should just short-circuit due to the "_client" field not being null after the first run-through.
(Note that this isn't being done in the object's constructor because it's an abstract class, and "GetHttpClient" is an abstract method -- BTW: don't ever call an abstract method in a base-class's constructor... that causes other nightmares)
Of course, it's fairly obvious that this isn't thread-safe, and the resultant behavior is non-deterministic.
The fix is to put this code behind a double-checked "lock" statement (although I will be eliminating the use of the "DefaultRequestHeaders" property anyways, just because).
In a nutshell, my original question shouldn't ever be an issue if you're careful in how you initialize the HttpClient.
Thanks for the clarity of thought that you all provided!
Related
I've found something that feels like a (minor) bug in .NET library code, and I wanted to check whether I had mis-configured the usage in some manner?
I have an C# Web API, using OWIN and running on .NET 4.6.1 (sigh)
During IntegrationTests we spin up in-memory instances of the API and poke it.
For annoying reasons (which I'm looking into fixing independently) we spin up a new instance of the API per-test.
I've recently discovered that if I have an nlog trace record Tracing to the "VSConsole" target the writing a single line of Trace will actually write it N times, where N is the number of tests that have run so far.
Digging into it I found that the cause lay in Microsoft.Owin.Hosting.WebApp.Start():
A few levels down that method calls Microsoft.Owin.Hosting.Engine.HostingEngine.Start(), which calls EnableTracing(), which looks like this:
private void EnableTracing(StartContext context)
{
TextWriterTraceListener writerTraceListener = new TextWriterTraceListener(context.TraceOutput, "HostingTraceListener");
Trace.Listeners.Add((TraceListener) writerTraceListener);
TraceSource traceSource = new TraceSource("HostingTraceSource", SourceLevels.All);
traceSource.Listeners.Add((TraceListener) writerTraceListener);
context.Builder.Properties["host.TraceOutput"] = (object) context.TraceOutput;
context.Builder.Properties["host.TraceSource"] = (object) traceSource;
context.Builder.SetLoggerFactory(this._loggerFactory);
}
Notably, this is adding a Listener without checking for any existing listeners. So if you repeatedly start and close new WebApps then you add more and more listeners to the Trace.Listeners collection. Which then means when you Trace to it, it writes out to the VSConsole repeatedly.
I've checked and there's nothing in the IDisposable output of WebApp.Start() that looks like it attempts to remove the listener either.
This ... seems like a bug :D
Direct Questions
Am I mis-using/mis-configuring it in some way?
Is it a bug in the .NET/OWIN code?
If so, is it still present in latest .NET?
If so, where should I report it?
Is there a sensible way to fix the issue in my tests? Some way to cleanup the Listeners between tests?
Calling Trace.Listeners.Clear(); before each WebApp start fixes the symptoms of the problem; ensures that there's only one Listener active at a time.
The purpose of calling SqlDependency.Start multiple times is to ensure it's fine before some other action such as creating a new instance of SqlCacheDependency based on a Command. According to Microsoft's document about SqlDependency.Start at here https://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqldependency.start(v=vs.110).aspx (the Remarks section), looks like calling SqlDependency.Start multiple times is totally fine:
Multiple calls with identical parameters (the same connection string and Windows credentials in the calling thread) are valid.
But actually it can fail (and really it has never succeeded for me) for the second call, making all next attempts to call SqlDependency.Start fail (silently by returning false, no exception is thrown).
What I did should meet the first restriction (mentioned in the Remarks section in the above link), that is all the calls to SqlDependency.Start have the same parameters (in fact there was just 1 same parameter which is the connection string). It looks just like this:
//at initialization step (such as in `Application_Start()` in ASP.NET MVC)
SqlDependency.Start(myConnectionString);//this usually returns OK
//later at the time before creating an instance of SqlCacheDependency
//I tried to call the Start method again to ensure everything is ok
var ok = SqlDependency.Start(myConnectionString);//almost always false
if(ok){
//almost never reach here ...
}
So it's really hard to understand about what stated by Microsoft (in the first restriction in the Remarks section), the 2 calls are exactly the same. But with the second call failed, any that same call used after that will still fail (meaning there is not any chance to start it successfully once I attempted to call it more than once).
When I see the log in Sql Server I can see that there are a lot of messages saying something like Cannot find the remote service ... because it does not exist
I don't need a solution or work-around this problem, I just need some explanation to why it does not work expectedly like what Microsoft stated, or I misunderstood what stated by Microsoft?
As Jeroen Mostert mentioned in the comments and the docs for SqlCommand.Start() state:
Returns
Boolean
true if the listener initialized successfully; false if a compatible listener already exists.
As the remarks in the docs describe, SqlDependency.Start() and SqlDependency.Stop() will keep track of the number of calls to each one. It will ensure a background connection is running or being set up if the number of calls to SqlDependency.Start() exceeds the number of calls to SqlDependency.Stop() (though I think it loses track and resets its count if you call SqlDependency.Stop() more times than than you call SqlDependency.Start()).
Start() Errors
It may help to clarify that it is possible for SqlDependency.Start() to fail. One way to get it to fail is to call it multiple times from one AppDomain with different connection strings. Within a particular AppDomain, SqlDependency.Start() will throw an exception if you pass in a different connection string unless if at least one of the following properties in the connection string is different from a previously passed connection string:
Database name
Username
I.e., you are expected to normalize or cache the connection string you first pass to SqlDependency.Start() so that you never pass it a string that has, for example, a different value for Max Pool Size. I think it does this to try to avoid creating a lot of broker queues and connections for a single process. Additionally, when it tries to match up a command to a broker queue when you actually set up an SqlDependency later, it probably uses these distinguishing connection string properties to decide which queue to use.
ASP.NET Life Cycle
From the ASP.NET Application Life Cycle documentation under “Life Cycle Events and the Global.asax file”, note the following:
The Application_Start method, while an instance method, is called only when the application is starting which often occurs during the first HTTP request for your application. The documentation specifically states:
You should set only static data during application start. Do not set any instance data because it will be available only to the first instance of the HttpApplication class that is created.
The method you should use to clean up things which you initialized in Application_Start is Application_End. When a webapp is gracefully stopped, an instance of your application class will be created and Application_End called on it. Note that this might be a different instance of the application class than Application_Start was called on.
Because of ASP.NET’s architecture, a distinct HttpApplication class instance is required for each request that is processing. That means that multiple instances will be created to handle concurrent requests. The docs also state that, for performance reasons, application class instances may be cached by the framework and used for multiple requests. To give you an opportunity to initialize and cleanup your application class at an instance level, you may implement Init and Dispose methods. These methods should configure the application class’s instance variables that are not specific to a particular requests. The docs state:
Init
Called once for every instance of the HttpApplication class after all modules have been created.
Dispose
Called before the application instance is destroyed.
However, you mentioned that you were initializing global state (i.e., SqlDependency.Start()) in Application_Start and cleaning up global state (i.e., SqlDependency.Stop()) in Dispose(). Due to the fact that Application_Start will be called once and is intended for configuring statics/globals and Dispose() is called for each application class instance that the framework retires (which may happen multiple times before Application_End() is called), it is likely that you are stopping the dependency quickly.
Thus, it may be that SqlDependency.Stop() is called after the server runs out of requests, in which case it would clean up the HttpApplication instance by calling Dispose(). Any attempts to actually start monitoring for changes by attaching an SqlDependency to an SqlCommand should likely fail at after that. I am not sure what already-subscribed commands will do, but they may fail at that point which would trigger your code to resubscribe a new dependency which should then hit an error. This could be the explanation for your “Cannot find the remote service” errors—you called SqlDependency.Stop() too early and too often.
In my scenario, I have to send data from one web application to a webapi which is an effective data store. The requests are necessarily synchronous and I most definitely want an Exception thrown if something goes awry as it means a critical part of the application is unavailable.
This is a derivative of, though not duplication of an existing question; Why use HttpClient for Synchronous Connection.
Yet over and over, including in the article I see above, I see a consistent recommendation to use HttpClient, even in a synchronous scenario. The best reason I've seen is the accepted answer in the SO post above but it essentially boils down to;
Use this because "shiny".
Which I'm not liking as an acceptable answer for my scenario. I'd prefer to use the correct object for the task at hand and this seems to be the older HttpWebRequest. Even Ben Watson's excellent resource "Writing High-Performance .NET Code" states the following;
Another example is the System.Net.HttpWebRequest class, which will
throw an exception if it receives a non-200 response from a server.
This bizarre behavior is thankfully corrected in the
System.Net.Http.HttpClient class in .NET 4.5
But in my scenario, I actually do want that behavior. While there are a lot of good use cases for HttpClient, can anyone offer a good reason not to use HttpWebRequest in my scenario? Am I using the correct object or not? And more importantly, why?
HttpClient is designed to give more control over http protocol, where else doing same in HttpWebRequest or WebClient was not that straight forward. Apart from asynchronous, there are many benefits of HttpClient
Benefits of HttpClient
Biggest benefit of HttpClient is plugin architecture, that lets you change underlying behavior of HTTP protocol easily.
HttpClient is extensible, underlying HttpMessageHandler allows you to completely by pass underlying Microsoft's HttpClient implementation and you can plugin your own implementation. For example, in iOS and Android, instead of using .Net's HttpClient, we could use native Http stack.
It is easy to replace caching, cookies by customizing HttpMessageHandler
CancellationToken support is excellent when we want to cancel a long running Http request.
Not shiny, but important, Multi threaded, HttpClient is optimized to manage multiple requests with single instance. CPU time is utilized very efficiently without using too many locks (synchronous operations depend on locks, which is considerable overhead on CPU). Today we are living in world of micro services. In server with many clients to serve and mobile OS, CPU time is costly.
Drawbacks
Only drawback is async/await, you can't simply use async libraries easily in synchronous code without using a Task Runner or deadlocks. Though there are many libraries supporting how to synchronously use async code.
There is no great benefit of HttpClient on Desktop application with lots of CPU time as spare.
HttpClient's behavior is considered "cleaner" because a non-success response from the server doesn't necessarily mean something has gone awry. While it's not true of your situation, imagine a process that wants to check that a resource does not exist and expects that it typically does not. With HttpWebRequest, the normal execution flow throws an exception, which is kind of gross and can complicate things, whereas HttpClient does not.
For your specific scenario, the distinction is perhaps irrelevant. Other situations in your program might prefer the HttpClient behavior though, and it's nice to standardize on a single HTTP client instead of having to juggle two.
HttpClient is not a replacement of WebClient/HttpWebRequest. HttpWebRequest gives you more flexibility, but at the same time it makes your code bit more verbose. Where as HttpClient provides a simple interface. You can use HttpWebRequest over HttpClient if you really want the additional features.
As per non-200 response code exceptions are concerned, HttpClient provides a way to simulate that behavior. You have to invoke
response.EnsureSuccessStatusCode();
For more details please visit Usage of EnsureSuccessStatusCode and handling of HttpRequestException it throws
I'm currently investigating some code which has a cache layer which at the bottom level uses the MemoryCache class. This is a c# Windows Service app so not web/IIS. There is a section of the code which spawns off a number of threads in which it creates and executes some code in a POC to perform some calculations. These are then stored in the above mentioned cache layer. What is been seen is it looks like cached values seem to be getting stored per thread and not at apication level. I thought that MemoryCache was a Singleton that would sit out side of the individual threads.
Can anybody confirm this behaviour would be expected?
Many thanks for any comments.
A MemoryCache is thread-safe but there is no reason to assume it's a singleton. If you want different threads to access the same MemoryCache instance you need to give them all a reference to the same instance (either as a singleton (really bad) static (still bad) or through argument passing as dependency injection (good)).
The simple way to do it (which does use global state) is to access the default memory cache:
var cache = MemoryCache.Default; // not really a good idea and harder to test, works
You can find the specific docs here. Make sure to configure it in your app/web.config file.
I'm trying to write a producer/consumer system using Redis in C#. Each message produced must be consumed by only one consumer, and I want the consumers to wait for elements created by the consumer. My system must support many produced/consumer sets.
I am using StackExchange.Redis to communicate with Redis, and using lists where elements are added using ListLeftPush and removed with ListRightPop. What I am experiencing is that while the ListRightPop method should block until an element exists in the list (or after a defined timeout), it always returns automatically if there are no elements in the list. This is the test code I wrote to check this:
IDatabase cache = connection.GetDatabase();
Trace.TraceInformation("waiting "+DateTime.Now);
var res = cache.ListRightPop("test");
Trace.TraceInformation("Got "+res+", Ended" + DateTime.Now);
And I'm getting a nil result after less than 1 second.
The standard pop operations do not block: they return nil if the list is empty or does not exist.
SE.Redis is a multiplexer. Using a blocking pop is a very very bad idea. This is explained more, with workarounds discussed specifically for blocking pops, in the documentation: https://stackexchange.github.io/StackExchange.Redis/PipelinesMultiplexers
StackExchange.Redis is merely hitting the redis server's exposed API, the relevant method of which is BRPOP in your case. The documentation for that is:
http://redis.io/commands/blpop - blocking left pop
http://redis.io/commands/brpop - blocking right pop
While those methods do describe the blocking behavior you are looking for, I believe SE.Redis ListRightPop is calling
http://redis.io/commands/rpop - right pop
I may not be up to the latest SE.Redis package, but intellisense is not giving me an option to supply a timeout like you claim. Additionally, there does not appear to be any methods starting with .List in the IDatabase interface that has the word "block" in it, so I'm not sure SE.Redis exposes a Redis BRPOP API. You can either write your own or ask Marc Gravell nicely, but this is a pretty big request I think because of the blocking nature of the call and the way the multiplexer works.