We Keep Coding

Avoid network throttling when making HTTP requests

I have a very specific problem when making HTTP webrequests. To be more specific, the application is making the requests 24/7 and updating my database table. Since other users are performing requests as well, I've came into a situation where when I make parallel web requests using parallel for loop with a combination of concurrent bag to speed things up, the other users experience a huge slowdown in websites performance. At some point the website becomes very slow and unresponsive when users + application makes the requests...
So now my question is following:
How can I limit the applications amount of webrequests it does at a specific moment?
For example if there are 10000 ports available through which app can make a web request. I wanna be able to tell to application to use lets say 10/15 threads at a time to make the request, and at the same time not to slow down the website to user so that there is no network throttling.
I read a few articles and some people were suggesting to use semaphore slim, but I have no idea how can I pair it up with my web request which looks like following:
private string MakeHTTPWebRequest(string requestXML)
{
var request = (HttpWebRequest)WebRequest.Create("https://api.ebay.com/ws/api.dll");
string GetItemTransactionXMLRequest = null;
byte[] bytes = null;
bytes = System.Text.Encoding.ASCII.GetBytes(requestXML);
ServicePointManager.DefaultConnectionLimit = 9999;
ServicePointManager.Expect100Continue = false;
request.Method = "POST";
request.ContentType = "application/xml";
request.Accept = "application/xml";
request.Proxy = null;
Stream requestStream = request.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Close();
using (var response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusCode == HttpStatusCode.OK)
{
Stream responseStream = response.GetResponseStream();
string responseStr = new StreamReader(responseStream).ReadToEnd();
responseStream.Flush();
responseStream.Close();
return responseStr;
}
return null;
}
}
This is how I do it currently:
Parallel.For(0,somelength,i=> List.Add(MakeHTTPWebRequest("Some xml request here")));
The method above gives me terrible network throttling. How can I do this in a manner where application would know if it's causing network throttling to reduce number of calls or await while user makes the request and then it continues the request?
At the same time this raises another question and issue, how can I set the timeout in this webrequest to unlimted xxx minutes so that app can wait till others are done with their requests so it can continue fetching the results from API...
Can someone help me out with this ?

You're setting some global variables every time you make an HTTP request. I'd recommend only setting them once.
I wanna be able to tell to application to use lets say 10/15 threads at a time to make the request
The fastest fix would be to just pass a ParallelOptions parameter to Parallel.For, setting MaxDegreeOfParallelism to 10/15.
I would also recommend considering making the code asynchronous, since this is an I/O-bound operation. That's where you would use SemaphoreSlim for throttling.
How can I do this in a manner where application would know if it's causing network throttling to reduce number of calls
That's a much harder problem. You'd have to measure your response times and feed them into a routine that establishes a "normal response time", and then starts throttling if the response times start getting too big. This is assuming that your app is throttled similarly to how a user would be.

xaml page doesn't wait for HttpWebResponse

In my Windows Phone 7 application I'm getting data from web service.
So on start of my application I check is my data is setted in Local Storage and if it's not (first run) I have made request by HttpWebRequest and wait for response.
SetupAppData(); //chack in local storage
if (CheckNetworkConnection(false))
if (appData == null)
GetAppData();
...
public void GetAppData()
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("some url");
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.BeginGetRequestStream(RequestData_Completed, request);
}
...
In RequestData_Completed I set request data and invoke request.BeginGetResponse(ResponseData_Completed, request);
Then when my response come back to the application I change it to my object and use in application.
But my problem is that I use this data in my xaml view before response come back. And I don't know how to make my view to wait for response.
I have tried to create some bool value isDataSetted, set it on true when my response come back and check its value in while loop but it block app and response never was handled.
Is there any solution for this issue?

Just have RequestData_Completed fire an event after the data is processed, and have your UI respond when the event occurs.

Cancel HTTPWebRequest when server goes down

I have created a Windows Service that calls a API (that returns JSON) with HTTPWebRequest.
The API doesn't return anything until it has something to "deliver". So I set the timeout quite high and lets the request wait until it receivs a response.
The problem is that when I test to turn off or disconnect the server running the API. The HTTPWebRequest doesn't stop the request. So I can't know if the API server has gone down.
The request code:
HttpWebRequest Request = (HttpWebRequest)HttpWebRequest.Create(Url);
Request.Method = "POST";
Request.ContentType = "application/x-www-form-urlencoded";
Request.ContentLength = PostData.Length;
Request.Timeout = Timeout;
Request.KeepAlive = true;
using (StreamWriter sw = new StreamWriter(Request.GetRequestStream()))
{
sw.Write(PostData);
}
using (HttpWebResponse Response = (HttpWebResponse)Request.GetResponse())
{
using (StreamReader sr = new StreamReader(Response.GetResponseStream()))
{
ResponseText = sr.ReadToEnd();
}
}
Is there anyway to "break" the request when the requested server goes down?
I have tried using the webbrowser to call the API server and after a while disconnect it and that return an error to the webpage.

you could use a background worker only cecking if the server is online. It has some disatvantages but may work fine.

It is always good to keep the requests asynchronous (See the BeginXXX methods in HttpWebRequest - http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx).
Using the asynchronous APIs ensures that you are not blocked until you get a response from the server.
In addition to using the asynchronous APIs, you can have a heart-beat requests (that could be just a HEAD HTTP request to a ping service on the server, which returns an empty body and HTTP 200 status), to keep track that the server is alive. If this request times out, then server is not alive - in which case, you can cancel / just 'forget' that the request has been made.

Weird Timeout on 4th HttpWebRequest in Multi-Threaded Application

I have a multi-threaded console application acting as a server. The server spawns a new thread every time a new client connects to the TcpListener:
//code copied from http://www.switchonthecode.com/tutorials/csharp-tutorial-simple- threaded-tcp-server
//blocks until a client has connected to the server
TcpClient client = tcpListener.AcceptTcpClient();
//create a thread to handle communication with connected client
Thread clientThread = new Thread(new ParameterizedThreadStart(HandleClientComm));
clientThread.Start(client);
The thread makes a number of HttpWebRequests using the following code:
public static HttpWebResponse HttpGet(string pRequestURI, string pArgs)
{
string requestURI = string.Format("{0}?{1}", pRequestURI, pArgs);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestURI);
request.Method = "GET";
request.ContentType = "application/x-www-form-urlencoded";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
return response;
}
The problem is that, I get a timeout on the FOURTH REQUEST. It is really weird and I cannot get the hang of it. The code worked just fine when it was in a single-threaded application. I am also making sure to close the response stream using:
response.Close();
The requestURI is correct, 'cause I tried copying and pasting it into my browser. Actually, it doesn't matter what the 4th request is (I've tried with different ones), I always get a timeout.
I guess it may be something related to thread-limits, but I really don't know how to solve it. Any suggestions would be greatly appreciated.
Thank you.

After a lot of blood, sweat and tears, I managed to solve it.
It was a stupid mistake really, turns out there was this one place where I was not closing the request.
For some (unknown) reason, this does not affect the requests when made from a Web Application, but, when issuing the requests from a Console Application, timeout problems are encountered.
Thank you #arx for your help - much appreciated.

request.ServicePoint.CloseConnectionGroup(request.ConnectionGroupName);
This line of code resolved my issue
Thanks

HttpWebRequest.GetRequestStream : What it does?

Code exemple:
HttpWebRequest request =
(HttpWebRequest)HttpWebRequest.Create("http://some.existing.url");
request.Method = "POST";
request.ContentType = "text/xml";
Byte[] documentBytes = GetDocumentBytes ();
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(documentBytes, 0, documentBytes.Length);
requestStream.Flush();
requestStream.Close();
}
When I do request.GetRequestStream (), there's nothing to send in the request. From the name of the method, and the intellisense it shows ("Get System.IO.Stream to use to write request data"), nothing indicates that this line of code will connect to the distant server.
But it seems it does...
Can anyone explain to me what HttpWebRequest.GetRequestStream () exactly does ?
Thanks for your enlightenments.

Getting the request stream does not trigger the post, but closing the stream does. Post data is sent to the server in the following way:
A connection is opened to the host
Send request and headers
Write Post data
Wait for a response.
The act of flushing and closing the stream is the final step, and once the input stream is closed (i.e. the client has sent what it needs to the server), then the server can return a response.

You use GetRequestStream() to synchronously obtain a reference to the upload stream. It is only after you have finished writing to the stream that the actual request is send.
However, I would suggest that you use the BeginGetRequestStream method instead of GetRequestStream. BeginGetRequestStream performs asynchronously and don't lock the current thread while the stream is being obtained. You pass a callback and a context to the BeginGetRequestStream. In the callback, you can call EndGetRequestStream() to finally grab a reference and repeat the writing steps listed above (for synchronous behavior). Example:
context.Request.BeginGetRequestStream(new AsyncCallback(Foo), context);
public void Foo(IAsyncResult asyncResult)
{
Context context = (Context)asyncResult.AsyncState;
try
{
HttpWebRequest request = context.Request;
using (var requestStream = request.EndGetRequestStream(asyncResult))
using (var writer = new StreamWriter(requestStream))
{
// write to the request stream
}
request.BeginGetResponse(new AsyncCallback(ProcessResponse), context);
}
Be very careful with BeginGetRequestStream. It never times out, thus you must add additional logic to your program to recover from situations where GetRequestStream will throw a timeout exception.
In general, threads are cheap. The async Begin/End methods of HttpWebRequest are only worth using if you will have 10,000 or more concurrent requests; because implementing timeouts is very tricky and error-prone. In general, using BeginGetRequestStream is premature optimization unless you need significant performance gains.