In my scenario, I have to send data from one web application to a webapi which is an effective data store. The requests are necessarily synchronous and I most definitely want an Exception thrown if something goes awry as it means a critical part of the application is unavailable.
This is a derivative of, though not duplication of an existing question; Why use HttpClient for Synchronous Connection.
Yet over and over, including in the article I see above, I see a consistent recommendation to use HttpClient, even in a synchronous scenario. The best reason I've seen is the accepted answer in the SO post above but it essentially boils down to;
Use this because "shiny".
Which I'm not liking as an acceptable answer for my scenario. I'd prefer to use the correct object for the task at hand and this seems to be the older HttpWebRequest. Even Ben Watson's excellent resource "Writing High-Performance .NET Code" states the following;
Another example is the System.Net.HttpWebRequest class, which will
throw an exception if it receives a non-200 response from a server.
This bizarre behavior is thankfully corrected in the
System.Net.Http.HttpClient class in .NET 4.5
But in my scenario, I actually do want that behavior. While there are a lot of good use cases for HttpClient, can anyone offer a good reason not to use HttpWebRequest in my scenario? Am I using the correct object or not? And more importantly, why?
HttpClient is designed to give more control over http protocol, where else doing same in HttpWebRequest or WebClient was not that straight forward. Apart from asynchronous, there are many benefits of HttpClient
Benefits of HttpClient
Biggest benefit of HttpClient is plugin architecture, that lets you change underlying behavior of HTTP protocol easily.
HttpClient is extensible, underlying HttpMessageHandler allows you to completely by pass underlying Microsoft's HttpClient implementation and you can plugin your own implementation. For example, in iOS and Android, instead of using .Net's HttpClient, we could use native Http stack.
It is easy to replace caching, cookies by customizing HttpMessageHandler
CancellationToken support is excellent when we want to cancel a long running Http request.
Not shiny, but important, Multi threaded, HttpClient is optimized to manage multiple requests with single instance. CPU time is utilized very efficiently without using too many locks (synchronous operations depend on locks, which is considerable overhead on CPU). Today we are living in world of micro services. In server with many clients to serve and mobile OS, CPU time is costly.
Drawbacks
Only drawback is async/await, you can't simply use async libraries easily in synchronous code without using a Task Runner or deadlocks. Though there are many libraries supporting how to synchronously use async code.
There is no great benefit of HttpClient on Desktop application with lots of CPU time as spare.
HttpClient's behavior is considered "cleaner" because a non-success response from the server doesn't necessarily mean something has gone awry. While it's not true of your situation, imagine a process that wants to check that a resource does not exist and expects that it typically does not. With HttpWebRequest, the normal execution flow throws an exception, which is kind of gross and can complicate things, whereas HttpClient does not.
For your specific scenario, the distinction is perhaps irrelevant. Other situations in your program might prefer the HttpClient behavior though, and it's nice to standardize on a single HTTP client instead of having to juggle two.
HttpClient is not a replacement of WebClient/HttpWebRequest. HttpWebRequest gives you more flexibility, but at the same time it makes your code bit more verbose. Where as HttpClient provides a simple interface. You can use HttpWebRequest over HttpClient if you really want the additional features.
As per non-200 response code exceptions are concerned, HttpClient provides a way to simulate that behavior. You have to invoke
response.EnsureSuccessStatusCode();
For more details please visit Usage of EnsureSuccessStatusCode and handling of HttpRequestException it throws
Related
My .NET Core project is generating only async methods for the WCF service. Is there any way to use synchronous methods like in regular .NET projects?
In regular .NET there was both sync and async methods like:
WcfClient.MethodName and WcfClient.MethodNameAsync
In .NET Core there are only WcfClient.MethodNameAsync
This question helped me:
How to call asynchronous method from synchronous method in C#?
But this solution causes other problems. Any ideas how to generate synchronous methods/functions?
Check this GitHub issue; they discuss some facts that you may be interested in.
You are right. Our NetCore/Netstandard support does not yet include Sync implementation of the APIs.
Please note that this change is on our Roadmap as one of the parity items for the transition to .NetCore and we will update with more specific timelines as soon as it is available.
erezvani1529 commented on Nov 29, 2016
And
Can you clarify for us what your need is for the synchronous methods? Overall, sync is much less resource efficient than async and is not something that we recommend to our customers. You can always use .GetAwaiter.GetResult() if you need to block on the async calls.
We have had some conversations with the CLR team, who provided this guidance:
If you expose an asynchronous endpoint from your library, avoid exposing a synchronous method that just wraps the asynchronous implementation. Doing so hides from the consumer the true nature of the implementation, and it should be left up to the consumer how they want to consume the implementation. If the consumer chooses to block waiting for the asynchronous implementation to complete, that’s up to the caller, and they can do so with their eyes wide open.
This is even more important for the “sync over async” case than it is for the “async over sync” case, because for “sync over async,” it can lead to significant problems with the application, such as hangs.
.NET Core team has chosen to not support a true sync api for the reasons listed above (resource consumption, etc.) Even if we implement a true sync, it would end up being sync over async at some level. For this reason, we believe that adding fake sync apis would be a hindrance and not a help to our customers. Please let us know if you have any feedback regarding this.
pemari-msft commented on May 11, 2017
And
Let me explain the rationale. True sync is not an option for a library if it is consuming a library that does not support true sync, as is the case for us and .NET Core. (And anyway, true sync is something of a misnomer for APIs that make network calls, but that's a different discussion.) So we have two options: create a sync-over-async wrapper, or let the caller do it. But as we have seen, the official guidance and prevailing wisdom is that exposing a wrapper should be avoided. This is with good reason, and it's similar to avoiding an async API that wraps a synchronous invocation. Avoiding such APIs not only keeps the API surface clean, it also avoids confusion. Customers will think there is true support for sync or async and expose this in their own public interface only to discover later that the underlying implementation doesn't actually support it! So we have concluded that it is best to follow the prevailing wisdom in this case.
You are correct that deadlock avoidance is essential, especially in sync-over-async scenarios, so let me say a little about how this is done. The deadlock issue arises when a thread calls an async method and then blocks waiting for the result, while the async method chain is waiting for the thread to free up so it can continue. The solution is to have the async method do its work in a different context where it won't be constrained by a limited pool of threads. This is done internally in the async method using ConfigureAwait, so deadlocks are avoided within the library itself.
If you feel that true sync is important then you can bring this up at https://github.com/dotnet/corefx to get this supported in .NET Core.
mirobers commented on May 12, 2017
So you would either change your project to avoid Azure or fake synchronous calls.
However, in 2018, there was a change in view:
Well, after enough discussion, following the lead set by HttpWebRequest we have decided to offer sync methods as sync-over-async. This is a time buffer for our users before we deprecate the sync APIs in future versions of the library.
#copernicus365 , to keep it simpler for people, particularly those who are already using the library, our Netstandard2.0 support which includes the feature parity between desktop and Netcore, will leave the methods as-is, in the same namespace / binary of the split-per-service packages(split packages are in preview at the moment). Naturally, we’ll need to echo guidance in our docs by the .NET team about the benefits of converting to async, and try to make as clear as possible that it’s sync-over-async under the covers.
Many thanks to everyone who took the time to chime in on this thread and give us feedback!
One related note - for existing people who are already using the sync methods on .NET Desktop, these will also be moving to sync-over-async. We’re modifying the codebase to use HttpClient everywhere (both in desktop and in .NET Core), which doesn’t offer any sort of true sync. This is to get away from having separate implementations of everything (one with HttpWebRequest and one with HttpClient), which has led to many bugs and behavior differences between the different implementations.
erezvani1529 commented on Mar 16, 2018
And the changes landed in the .NET Standard 2.0 target in May 2018.
I'm not exactly sure how a WCF proxy class will handle sending requests through plinq. Does the following code snippit look ok, or does it look like it could cause problems with shared state across multiple threads? (also, I already understand that a using block is not ideal, this is just an example)
using (var proxy = new ServiceProxyOfSomeSort())
{
_aBunchOfMessagesToSend.AsParallel()
.WithDegreeOfParallelism(SomeDegree).ForAll(m =>
{
proxy.SomeOperation(m);
}
}
Should I be creating the proxy once per thread? Is it ok to share the proxy across threads? I don't want to create more proxies than I need to because that is a somewhat expensive operation.
Edit:
I don't really have any of the implementation details of the service on the server side. From a requirements standpoint they should have developed it so that multiple clients could call it at anytime (async). Assuming that they can handle async calls (which may be a big assumption) I'm just trying to figure out if this an acceptable approach from the client side. It is working, I just don't know if there are any gotchas with this approach.
In response to your question about the proxy: you should create a new one per thread. Reusing the proxy tends to work for a while and then throw a fault after tens of requests. As they love to say, its behavior is undefined. The performance overhead of creating new proxies is not huge (after the first one is created), so it shouldn't be a big deal.
As for the discussion of multiple concurrent requests swamping the server: a few, or even a dozen, should be fine - the server can probably handle as many requests as your processor can create threads. Then again, the sudden influx of requests from a single source might be interpreted as a Denial Of Service attack, so you should be wary, particularly if your control over the service is limited.
Problem: I need to download hundreds of images from different hosts. Each host has anywhere between 20-hundreds of images.
Solution: using a new WebClient every time a image needs to be downloaded through the WebClient's DownloadData method.
Or would be better to keep a pool of open socket connections and making the http request using lower level calls?
Is it expensive to open/close a tcp connection (I'm assuming that is what WebClient does), so that using a pools sounds more efficient?
I believe the underlying infrastructure which WebClient uses will already pool HTTP connections, so there's no need to do this. You may want to check using something like Wireshark of course, with some sample URLs.
Fundamentally, I'd take the same approach to this as with other programming tasks - write the code in the simplest way that works, and then check whether it performs well enough for your needs. If it does, you're done. If it doesn't, use appropriate tools (network analyzers etc) to work out why it's not performing well enough, and use more complicated code only if it fixes the problem.
My experience is that WebClient is fine if it doesn't what you need - but it doesn't give you quite as much fine-grained control as WebRequest. If you don't need that control, go with WebClient.
I use HttpWebRequest and HttpWebResponse to scrape anything I want. Unless, of course, there are services available for the requirement, but even though, sometimes, there are limitations (business limitations) and I often prefer to dig the html from pure http request. Sometimes just make feel more like developer, you know...
I have a server that I have no control over, it's JSON based and I've put together a simple proof of concept that calls the server using HTTPWebRequest etc and it works fine (if a little wordy since MS have removed all Synchronous I/O calls).
Is there a better way of doing this? I've been looking at WCF as an option but any stable and reasonably performant library should do the job. This is a new area for me so I'm a little unsure what the best practice is (or where to find it out)
Thanks in advance
Dave
Not sure whether it's the best method, but HttpWebRequest plus DataContractJsonSerializer are probably the best approach using classes from the Windows Phone library only -- plus HttpWebRequest's asynchronous methods ensure that your code won't block the UI thread when performing network requests.
Once you've written the http request code once, you can easily abstract it away for reuse. You've already done the hard part :-)
I am working on a class library that retrieves information from a third-party web site. The web site being accessed will stop responding if too many requests are made within a set time period (~0.5 seconds).
The public methods of my library directly relate to a resource an file on the web server. In other words, each time a method is called, an HttpWebRequest is created and sent to the server. If all goes well, an XML file is returned to the caller. However, if this is the second web request in less than 0.5s, the request will timeout.
My dilemma lies in how I should handle request throttling (if at all). Obviously, I don't want the caller sit around waiting for a response -- especially if I'm completely certain that their request will timeout.
Would it make more sense for my library to queue and throttle the webrequests I create, or should my library simply throw an exception if the a client does not wait long enough between API calls?
The concept of a library is to give its client code as little to worry about as possible. Therefore I would make it the libraries job to queue requests and return results in a timely manner. In an ideal world you would use a callback or delegate model so that the client code can operate in asynchronously, not blocking the UI. You could also offer the option for skipping the queue, (and failing if it operates too soon) and possibly even offer priorities within the queue model.
I also believe it is the responsibility of the library author to default to being a good citizen, and for the library's default operation to be to comply to the conditions of the data provider.
I'd say both - you're dealing with two independent systems and both should take measures to defend themselves from excessive load. The web server should refuse incoming connections, and the client library should take steps to reduce the requests it makes to a slow or unresponsive external service. A common pattern for dealing with this on the client is 'circuit breaker' which wraps calls to an external service, and fails fast for a certain period after failure.
That's the Web server's responsibility, imo. Because the critical load depends on hardware, network bandwidth, etc a lot of things that are outside of your application's control, it should not concern itself with trying the deal with it. IIS can throttle traffic based on various configuration options.
What kind of client is it? Is this an interactive client, for eg: GUI based app?
In that case, you can equate that to a webbrowser scenario, and let the timeout surface to the caller. Also, if you know for sure that this webserver is throttling requests, you can tell the client that he has to wait for a given time period before retrying. In that way, the client will not keep on re-issuing requests, and will know when the first timeout occurs that it is futile to issue requests too fast.