We Keep Coding

Avoid network throttling when making HTTP requests

I have a very specific problem when making HTTP webrequests. To be more specific, the application is making the requests 24/7 and updating my database table. Since other users are performing requests as well, I've came into a situation where when I make parallel web requests using parallel for loop with a combination of concurrent bag to speed things up, the other users experience a huge slowdown in websites performance. At some point the website becomes very slow and unresponsive when users + application makes the requests...
So now my question is following:
How can I limit the applications amount of webrequests it does at a specific moment?
For example if there are 10000 ports available through which app can make a web request. I wanna be able to tell to application to use lets say 10/15 threads at a time to make the request, and at the same time not to slow down the website to user so that there is no network throttling.
I read a few articles and some people were suggesting to use semaphore slim, but I have no idea how can I pair it up with my web request which looks like following:
private string MakeHTTPWebRequest(string requestXML)
{
var request = (HttpWebRequest)WebRequest.Create("https://api.ebay.com/ws/api.dll");
string GetItemTransactionXMLRequest = null;
byte[] bytes = null;
bytes = System.Text.Encoding.ASCII.GetBytes(requestXML);
ServicePointManager.DefaultConnectionLimit = 9999;
ServicePointManager.Expect100Continue = false;
request.Method = "POST";
request.ContentType = "application/xml";
request.Accept = "application/xml";
request.Proxy = null;
Stream requestStream = request.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Close();
using (var response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusCode == HttpStatusCode.OK)
{
Stream responseStream = response.GetResponseStream();
string responseStr = new StreamReader(responseStream).ReadToEnd();
responseStream.Flush();
responseStream.Close();
return responseStr;
}
return null;
}
}
This is how I do it currently:
Parallel.For(0,somelength,i=> List.Add(MakeHTTPWebRequest("Some xml request here")));
The method above gives me terrible network throttling. How can I do this in a manner where application would know if it's causing network throttling to reduce number of calls or await while user makes the request and then it continues the request?
At the same time this raises another question and issue, how can I set the timeout in this webrequest to unlimted xxx minutes so that app can wait till others are done with their requests so it can continue fetching the results from API...
Can someone help me out with this ?

You're setting some global variables every time you make an HTTP request. I'd recommend only setting them once.
I wanna be able to tell to application to use lets say 10/15 threads at a time to make the request
The fastest fix would be to just pass a ParallelOptions parameter to Parallel.For, setting MaxDegreeOfParallelism to 10/15.
I would also recommend considering making the code asynchronous, since this is an I/O-bound operation. That's where you would use SemaphoreSlim for throttling.
How can I do this in a manner where application would know if it's causing network throttling to reduce number of calls
That's a much harder problem. You'd have to measure your response times and feed them into a routine that establishes a "normal response time", and then starts throttling if the response times start getting too big. This is assuming that your app is throttled similarly to how a user would be.

WebRequest - GetRequestStream timing out with exception

I'm writing this short module where I have to modify an addressed resource with a PUT method. I'm using the WebRequest class to make this URI request and the GetRequestStream() to get the stream to write to.
However, it seems that after a couple of successful method calls (and using the PUT to modify resources) via this method below, my application hangs and then throws a WebException: The request timed out. error. Here's what the code looks like:
public void SendOffMessageToResource(int res_ID){
var httpWebRequest = WebRequest.Create ("http://192.168.x.x/api/sample_user/resources/1/state");
httpWebRequest.ContentType = "application/json";
httpWebRequest.Method = "PUT";
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream())) //here's where the VS seems to take a long long time to step over to the next line when the error happens.
{
string json = "{\"on\":false}";
streamWriter.Write(json);
streamWriter.Close();
}
}
I am already disposing the StreamWriter. I'm not even using the GetResponse() method, because all I need to do on this URI is actually modify the addressed resource with PUT method. I am not sure why it still throws an error and hangs the application. The search of previous threads only revealed that people should be using using statements to dispose resources, but I'm already doing that I think or perhaps I'm missing something? Do I always need to use GetResponse() to complete the request and dispose that always in addition to this?

Do I always need to use GetResponse() to complete the request and dispose that always in addition to this?
Yes, just that.

How do I resend a "failed" WebRequest?

I send POST and GET WebRequest that should support longer periods of internet being down. The idea is to queue the failed (timedout) webrequest and to try to resend them periodically until the internet is up again and all queued WebRequests are sent.
However, I seems that I cannot just reuse the old WebRequest. Do I need to set it up again?
IAsyncResult result = request.BeginGetResponse (o => {
callCallback (o);
}, state);
When request is just setup using:
var request = HttpWebRequest.Create (String.Format (#"{0}/{1}", baseServiceUrl, path));
request.Method = "GET";
request.ContentType = "application/xml; charset=UTF-8";
request.Headers.Add ("Authority", account.Session);
return request;
it works fine. But after a timeout (and request.Abort ();) and calling BeginGetResponse() on the same webrequest just freezes.

You cannot call BeginGetResponse() multiple times on the same HttpWebRequest. I'm not sure whether that's support on .NET, but it's not possible with Mono's implementation (and not something that'd be easy to change).
See also Can I reuse HttpWebRequest without disconnecting from the server? - the underlying network connection will be reused.

HTTP request not responding [duplicate]

This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Adjusting HttpWebRequest Connection Timeout in C#
In my code, I am calling a live chat api to get list of operators in the following format:
HttpWebRequest request
= WebRequest.Create("url-here") as HttpWebRequest;
request.Credentials
= new NetworkCredential(
username,
password
);
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
StreamReader reader = new StreamReader(response.GetResponseStream());
//dowhatever
}
Today the live chat api went down and because of the our site also went down. In this case the api request went on spinning and was in 'not responding' state.
How to fix this? I do not want continue waiting until the api responds. because it will cause my page also go on spinning. Is there way like timeout - or wait for 2 seconds then skip the live chat request part?
Thanks for the help!

You can use the WebRequest.Timeout property to set how long you'd like your code to wait before timing out:
var request = WebRequest.Create("url-here") as HttpWebRequest;
request.Credentials = new NetworkCredential(username, password);
request.Timeout = 30 * 1000; // wait for 30 seconds (30,000 milliseconds)

HttpWebRequest.GetRequestStream : What it does?

Code exemple:
HttpWebRequest request =
(HttpWebRequest)HttpWebRequest.Create("http://some.existing.url");
request.Method = "POST";
request.ContentType = "text/xml";
Byte[] documentBytes = GetDocumentBytes ();
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(documentBytes, 0, documentBytes.Length);
requestStream.Flush();
requestStream.Close();
}
When I do request.GetRequestStream (), there's nothing to send in the request. From the name of the method, and the intellisense it shows ("Get System.IO.Stream to use to write request data"), nothing indicates that this line of code will connect to the distant server.
But it seems it does...
Can anyone explain to me what HttpWebRequest.GetRequestStream () exactly does ?
Thanks for your enlightenments.

Getting the request stream does not trigger the post, but closing the stream does. Post data is sent to the server in the following way:
A connection is opened to the host
Send request and headers
Write Post data
Wait for a response.
The act of flushing and closing the stream is the final step, and once the input stream is closed (i.e. the client has sent what it needs to the server), then the server can return a response.

You use GetRequestStream() to synchronously obtain a reference to the upload stream. It is only after you have finished writing to the stream that the actual request is send.
However, I would suggest that you use the BeginGetRequestStream method instead of GetRequestStream. BeginGetRequestStream performs asynchronously and don't lock the current thread while the stream is being obtained. You pass a callback and a context to the BeginGetRequestStream. In the callback, you can call EndGetRequestStream() to finally grab a reference and repeat the writing steps listed above (for synchronous behavior). Example:
context.Request.BeginGetRequestStream(new AsyncCallback(Foo), context);
public void Foo(IAsyncResult asyncResult)
{
Context context = (Context)asyncResult.AsyncState;
try
{
HttpWebRequest request = context.Request;
using (var requestStream = request.EndGetRequestStream(asyncResult))
using (var writer = new StreamWriter(requestStream))
{
// write to the request stream
}
request.BeginGetResponse(new AsyncCallback(ProcessResponse), context);
}
Be very careful with BeginGetRequestStream. It never times out, thus you must add additional logic to your program to recover from situations where GetRequestStream will throw a timeout exception.
In general, threads are cheap. The async Begin/End methods of HttpWebRequest are only worth using if you will have 10,000 or more concurrent requests; because implementing timeouts is very tricky and error-prone. In general, using BeginGetRequestStream is premature optimization unless you need significant performance gains.