I want to send multiple WebRequest. I used a Parallel.For loop to do that but the loop runs once and the second time it gives error while getting response.
Error:
The operation has timed out
Code :
Parallel.For(0, 10, delegate(int i) {
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(
new Uri("http://www.mysite.com/service"));
string dataToSend = "Data";
byte[] buffer = System.Text.Encoding.GetEncoding(1252).
GetBytes(dataToSend);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = buffer.Length;
request.Host = "www.mysite.com";
Stream requestStream = request.GetRequestStream();
requestStream.Write(buffer, 0, buffer.Length);
requestStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
});
Most likely the problem is that you need to call response.Close() after you're done processing the response.
In addition to what Jim Mischel said about calling Close on the response, you also need to factor in that, by default, .NET limits an application to only two active HTTP connections per domain at once. To change this you can set System.Net.ServicePointManager.DefaultConnectionLimit programmatically or set the same thing via config using the <system.net><connectionManagement> configuration section.
Related
We use an HttpWebRequest to send query to a webservice with NetworkCredential like this :
Byte[] bytes = Encoding.UTF8.GetBytes("...");
HttpWebRequest request = (HttpWebRequest)WebRequest.CreateHttp(strURL);
request.Method = "POST";
request.ContentLength = bytes.Length;
request.ContentType = "application/xml";
request.Timeout = 10000;
request.KeepAlive = false;
NetworkCredential cred = new NetworkCredential
{
UserName = "...",
Password = "..."
};
request.Credentials = cred;
WebRequest.DefaultWebProxy.Credentials = CredentialCache.DefaultCredentials;
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(bytes, 0, bytes.Length);
}
HttpWebResponse resp = request.GetResponse() as HttpWebResponse;
To test it, we created a small console project with just this request and constant information (always the same).
After two successful requests, the third one systematically stops and ends in timeout.
But if we restart our application, we can immediately reissue two requests before the third one freezes again.
Do you have any idea of the cause? It looks like some information needs to be purged before we can continue.
-- EDIT --
I tried to start each request on a separate thread (Using Thread or Task) with the same result.
I change everything to use HttpClient and, now, everything works fine.
I am sending a large number of simultaneous requests to a particular web service with different data. To achieve this, I have created a number of threads(around 50 in number). Total number of requests per minute may increase up to 10000.
The application in the form of a windows service runs fine for a few minutes and then a operation time out error is encountered.
I have tried the usual suspects such as increasing DefaultConnectionLimit, closing the web response object. Since the requests do not take much time on server, I have also set the request Timeout and ReadWriteTimeout to 5 seconds.
Below is the code snippet which is called repeatedly by different threads.
// Below line is executed at the start of application
ServicePointManager.DefaultConnectionLimit = 15000;
// Below code is executed at repeatedly by different threads
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Host = hostName;
request.Proxy = null;
request.UserAgent = "Windows Service";
byte[] bytes = new byte[0];
if (body != null)
{
bytes = System.Text.Encoding.ASCII.GetBytes(body);
request.ContentType = "text/xml; encoding='utf-8'";
request.ContentLength = bytes.Length;
}
request.Method = "POST";
request.Timeout = 5000;
request.ReadWriteTimeout = 5000;
request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(Encoding.Default.GetBytes(username + ":" + password));
request.CookieContainer = this.cookieContainer;
if (body != null)
{
Stream requestStream = request.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Close();
}
HttpWebResponse httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
responseText = streamReader.ReadToEnd();
}
httpResponse.Close();
ServicePointManager.DefaultConnectionLimit limits the number of outgoing web requests to a given server. The default is generally 2 or 10.
If you are making 50 parallel calls to that web service, you should set ServicePointManager.DefaultConnectionLimit (at app startup) to a larger number (e.g. 40-50).
Additionally, you are not calling Close or Dispose on request. You should do this, or let using take care of it for you.
I’m working with a WPF .net 4.0 Application. I have a search bar. For each search token I need to do 8 http request to 8 separate URLs to get search results. I send 8 requests to server after 400 milliseconds once the user stops typing in search bar. Searching for 6 to 7 search-tokens results comes very nicely. But after that suddenly HttpWebRequest stops working silently. No exception was thrown, no response was received. I'm working with Windows 7, I disabled the firewall too. I don't know where the subsequent http requests are lost.
Can anyone show me lights to fix this issue?
Below is my code for HttpWebRequest call.
public static void SendReq(string url)
{
// Create a new HttpWebRequest object.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.ContentType = "application/x-www-form-urlencoded";
request.Proxy = new WebProxy("192.168.1.1", 8000);
// Set the Method property to 'POST' to post data to the URI.
request.Method = "POST";
// start the asynchronous operation
request.BeginGetRequestStream(new AsyncCallback(GetRequestStreamCallback), request);
}
private static void GetRequestStreamCallback(IAsyncResult asynchronousResult)
{
HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
// End the operation
Stream postStream = request.EndGetRequestStream(asynchronousResult);
string postData = this.PostData;
// Convert the string into a byte array.
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
// Write to the request stream.
postStream.Write(byteArray, 0, byteArray.Length);
postStream.Close();
// Start the asynchronous operation to get the response
request.BeginGetResponse(new AsyncCallback(GetResponseCallback), request);
}
private static void GetResponseCallback(IAsyncResult asynchronousResult)
{
HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
// End the operation
using(HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(asynchronousResult))
{
using(Stream streamResponse = response.GetResponseStream())
{
using(StreamReader streamRead = new StreamReader(streamResponse))
{
string responseString = streamRead.ReadToEnd();
Debug.WriteLine(responseString);
}
}
}
}
I think i am very late, but i still want to answer your question, may be it can be helpful to others. By default HTTP Request you make are HTTP 1.1 requests. And HTTP 1.1 Request by default has Keep-Alive connection. so when you make too many request to same server .net framework only make x no. of request.
you should close all your response by response.Close()
you can also specify how many simultaneous requests you can make.
ServicePointManager.DefaultConnectionLimit = 20;
Note that you have to set DefaultConnectionLimit before the first request you make. you can find more information
here on msdn.
All i can see is that in GetRequestStreamCallback you should replace
postStream.Write(byteArray, 0, postData.Length);
by
postStream.Write(byteArray, 0, byteArray.Length);
since these length aren't necessarily equal.
#Somnath
I am not sure if you found this answer, but if anyone else stumbles across this post with the same issue that Somnath and I were having.
We all try to do our due diligence in keeping memory clean and clear, but with streams we always will save unexplained issues if we make sure to flush the stream before we close it.
Replace This :
postStream.Write(byteArray, 0, byteArray.Length);
postStream.Close();
With This :
postStream.Write(byteArray, 0, byteArray.Length);
postStream.Flush();
postStream.Close();
I followed all suggestion provide by all of you but couldn't stop the silent failure of HTTP request. But I found a workaround. Even myself could not reach to a final conclusion till now.
But my workaround is working well as off now without any failure.
In SendReq(string url) function i have added the following lines of code
System.Net.ServicePointManager.DefaultConnectionLimit = 100; // Just selected a random number for testing greater than 2
System.Net.ServicePointManager.SetTcpKeepAlive(true, 30, 30); // 30 is based on my server i'm hitting
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
request.KeepAlive = true;
I'm calling an API hosted on Apache server to post data. I'm using HttpWebRequest to perform POST in C#.
API has both normal HTTP and secure layer (HTTPS) PORT on the server. When I call HTTP URL it works perfectly fine. However, when I call HTTPS it gives me time-out exception (at GetRequestStream() function). Any insights? I'm using VS 2010, .Net framework 3.5 and C#. Here is the code block:
string json_value = jsonSerializer.Serialize(data);
HttpWebRequest request = (HttpWebRequest)System.Net.WebRequest.Create("https://server-url-xxxx.com");
request.Method = "POST";
request.ProtocolVersion = System.Net.HttpVersion.Version10;
request.ContentType = "application/x-www-form-urlencoded";
byte[] buffer = Encoding.ASCII.GetBytes(json_value);
request.ContentLength = buffer.Length;
System.IO.Stream reqStream = request.GetRequestStream();
reqStream.Write(buffer, 0, buffer.Length);
reqStream.Close();
EDIT:
The console program suggested by Peter works fine. But when I add data (in JSON format) that needs to be posted to the API, it throws out operation timed out exception. Here is the code that I add to console based application and it throws error.
byte[] buffer = Encoding.ASCII.GetBytes(json_value);
request.ContentLength = buffer.Length;
I ran into the same issue. It seems like it is solved for me. I went through all my code making sure to invoke webResponse.Close() and/or responseStream.Close() for all my HttpWebResponse objects. The documentation indicates that you can close the stream or the HttpWebResponse object. Calling both is not harmful, so I did. Not closing the responses may cause the application to run out of connections for reuse, and this seems to affect the HttpWebRequest.GetRequestStream as far as I can observe in my code.
I don't know if this will help you with your specific problem but you should consider Disposing some of those objects when you are finished with them. I was doing something like this recently and wrapping stuff up in using statements seems to clean up a bunch of timeout exceptions for me.
using (var reqStream = request.GetRequestStream())
{
if (reqStream == null)
{
return;
}
//do whatever
}
also check these things
Is the server serving https in your local dev environment?
Have you set up your bindings *.443 (https) properly?
Do you need to set credentials on the request?
Is it your application pool account accessing the https resources or is it your account being passed through?
Have you thought about using WebClient instead?
using (WebClient client = new WebClient())
{
using (Stream stream = client.OpenRead("https://server-url-xxxx.com"))
using (StreamReader reader = new StreamReader(stream))
{
MessageBox.Show(reader.ReadToEnd());
}
}
EDIT:
make a request from console.
internal class Program
{
private static void Main(string[] args)
{
new Program().Run();
Console.ReadLine();
}
public void Run()
{
var request = (HttpWebRequest)System.Net.WebRequest.Create("https://server-url-xxxx.com");
request.Method = "POST";
request.ProtocolVersion = System.Net.HttpVersion.Version10;
request.ContentType = "application/x-www-form-urlencoded";
using (var reqStream = request.GetRequestStream())
{
using(var response = new StreamReader(reqStream )
{
Console.WriteLine(response.ReadToEnd());
}
}
}
}
Try this:
WebRequest req = WebRequest.Create("https://server-url-xxxx.com");
req.Method = "POST";
string json_value = jsonSerializer.Serialize(data); //Body data
ServicePointManager.Expect100Continue = false;
using (var streamWriter = new StreamWriter(req.GetRequestStream()))
{
streamWriter.Write(json_value);
streamWriter.Flush();
streamWriter.Close();
}
HttpWebResponse resp = req.GetResponse() as HttpWebResponse;
Stream GETResponseStream = resp.GetResponseStream();
StreamReader sr = new StreamReader(GETResponseStream);
var response = sr.ReadToEnd(); //Response
resp.Close(); //Close response
sr.Close(); //Close StreamReader
And review the URI:
Reserved characters. Send reserved characters by the URI can bring
problems ! * ' ( ) ; : # & = + $ , / ? # [ ]
URI Length: You should not exceed 2000 characters
I ran into this, too. I wanted to simulate hundreds of users with a Console app. When simulating only one user, everything was fine. But with more users came the Timeout exception all the time.
Timeout occurs because by default the ConnectionLimit=2 to a ServicePoint (aka website).
Very good article to read: https://venkateshnarayanan.wordpress.com/2013/04/17/httpwebrequest-reuse-of-tcp-connections/
What you can do is:
1) make more ConnectionGroups within a servicePoint, because ConnectionLimit is per ConnectionGroups.
2) or you just simply increase the connection limit.
See my solution:
private HttpWebRequest CreateHttpWebRequest<U>(string userSessionID, string method, string fullUrl, U uploadData)
{
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(fullUrl);
req.Method = method; // GET PUT POST DELETE
req.ConnectionGroupName = userSessionID; // We make separate connection-groups for each user session. Within a group connections can be reused.
req.ServicePoint.ConnectionLimit = 10; // The default value of 2 within a ConnectionGroup caused me always a "Timeout exception" because a user's 1-3 concurrent WebRequests within a second.
req.ServicePoint.MaxIdleTime = 5 * 1000; // (5 sec) default was 100000 (100 sec). Max idle time for a connection within a ConnectionGroup for reuse before closing
Log("Statistics: The sum of connections of all connectiongroups within the ServicePoint: " + req.ServicePoint.CurrentConnections; // just for statistics
if (uploadData != null)
{
req.ContentType = "application/json";
SerializeToJson(uploadData, req.GetRequestStream());
}
return req;
}
/// <summary>Serializes and writes obj to the requestStream and closes the stream. Uses JSON serialization from System.Runtime.Serialization.</summary>
public void SerializeToJson(object obj, Stream requestStream)
{
DataContractJsonSerializer json = new DataContractJsonSerializer(obj.GetType());
json.WriteObject(requestStream, obj);
requestStream.Close();
}
You may want to set timeout property, check it here http://www.codeproject.com/Tips/69637/Setting-timeout-property-for-System-Net-WebClient
I'm calling web service (my web service) like this:
var request = WebRequest.Create(Options.ServerUri + Options.AccountId + "/integration/trip") as HttpWebRequest;
request.Timeout = 20000; // 20 seconds should be plenty, no need for 100 seconds
request.ContentType = "application/json";
request.Headers.Add(HttpRequestHeader.Authorization, "Basic " + Convert.ToBase64String(Encoding.UTF8.GetBytes(Options.LoginName + ":" + Options.Password)));
request.Method = "POST";
var serializedData = (new JavaScriptSerializer()).Serialize(trip);
var bytes = Encoding.UTF8.GetBytes(serializedData);
request.ContentLength = bytes.Length;
var os = request.GetRequestStream();
os.Write(bytes, 0, bytes.Length);
os.Close();
request.GetResponse();
LoggingAndNotifications.LogAndNotify(string.Format("Success uploading trip: {0}", trip.TripId), false);
return true;
This code called repeatedly to post new objects. After about 3 calls I start getting timeouts on reguest.GetReponse()
There is no errors on server side, nothing in Event Log. It feels like "something" stops me from repeatedly hitting service. What should I look for? Is it possible with company firewall? Or something wrong with my code?
I think the issue is that you are not closing the response. Try editing your code as follows:
var response = request.GetResponse() as HttpWebResponse;
response.Close();
You should close the response as per the example in the doco.
WebRequest myRequest = WebRequest.Create("http://www.contoso.com");
// Return the response.
WebResponse myResponse = myRequest.GetResponse();
// Code to use the WebResponse goes here.
// Close the response to free resources.
myResponse.Close();
Hmm. The doco also says
Any public static (Shared in Visual Basic) members of this type are
thread safe. Any instance members are not guaranteed to be thread
safe.
You should probably ask for a lock of some kind.
Are you sure this is not caused by server side bugs?
It seems strange, as far as I known, the webrequest on .net4 is based on IOCP in lower layer, maybe you can try release web request/response resources after each loop.
Since the GetResponse() will return a stream, if you don't read from it, the real data may not transfer from server to client side. (I found this when I am trying to parse a response that I used peek(), and it always return an invalid value until the read() is called.)
So, try to read it or just close it.