I am trying to set a timeout for a special request which will take a long time to process. Because of this, I am trying to set the timeout, like this:
client.RequestFilter = r => {
r.Timeout = 1000000;
r.ReadWriteTimeout = 1000000;
}
However, these settings seem to have no effect; the request still times out in about 30 seconds. Is there some hack I can use to set the timeout properly ?
ETA: The response I'm receiving is a stream; I do it like this:
var stream = client.Send<Stream>(requestDto);
Is there a better way ?
ServiceStack's Service Clients is just a wrapper around HttpWebRequest so your code ends up setting the HttpWebRequest Timeout and ReadWriteTimeout properties directly.
The Request Filter gives you direct access to the HttpWebRequest instance used and setting the Timeout properties should work as expected. Other than that the only class that can modify behavior of .NET's HttpWebRequest is System.Net.ServicePointManager which lets you configure some properties like DefaultConnectionLimit and DnsRefreshTimeout, etc. But there's no additional Request Timeout properties.
The alternative solution you can try is to use ServiceStack's JsonHttpClient which as it's built on Microsoft's newer HttpClient library, you may have better luck with it. Although it's recommended to use the Async API's since the Sync API's are just blocking on the HttpClient's underlying Async API's.
For the API call itself, you should access the stream in a using block, e.g:
using (var stream = client.Send<Stream>(requestDto))
{
}
Related
Retrieving the resource at http://www.w3.org/TR/xmlschema11-1/XMLSchema.xsd takes around 10 seconds using the following mechanisms:
web browser
curl
Java URL.openConnection()
It's possible that the W3C site is applying some "throttling" - deliberately slowing the response to discourage bulk requests.
Trying to retrieve the same resource from a C# application on .NET, I get a timeout after about 60-70 seconds. I've tried a couple of different approaches, both with the same result:
System.Xml.XmlUrlResolver.GetEntity()
new WebClient().OpenRead(uri)
Anyone have any idea what's going on? Would another API, or some configuration options, solve the problem?
The problem is they are (probably) checking for a User-Agent string. If it's not present, they send you to purgatory. .NET's http clients do not set this by default.
So, give this a shot:
private static readonly HttpClient _client = new HttpClient();
public static async Task TestMe()
{
using (var req = new HttpRequestMessage(HttpMethod.Get,
"http://www.w3.org/TR/xmlschema11-1/XMLSchema.xsd"))
{
req.Headers.Add("user-agent",
"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X)");
using (var resp = await _client.SendAsync(req))
{
resp.EnsureSuccessStatusCode();
var data = await resp.Content.ReadAsStringAsync();
}
}
}
No idea why they do this; Maybe it's a bug in their back-end? (I sure wouldn't want to leave a socket open longer than it needs to be for no good reason). The request still takes 10-15 seconds, but it's better than the 120+ second timeout.
I have embedded a browser control into a .Net form and compiled it as a window's executable. The browser control is displaying our HTML5 image viewer. The application opens sockets so it can listen to "push" requests from various servers. This allows images to be pushed to individual user's desktops.
When an incoming image push request comes in, the application calls a REST service using RESTSharp to generate a token for the viewer to use to display the image.
As long as the requests are consistently arriving, everything works great. If there is a lull (10 minutes seems to be the time frame), then the RESTSharp request times out. It is almost as though the creation of a new instance of the RESTSharp artifacts are reusing the old ones in an attempted .Net optimization.
Here is the RESTSharp code I am using:
private async Task<string> postJsonDataToUrl(string lpPostData) {
IRestClient client = new RestClient(string.Format("{0}:{1}", MstrScsUrlBase, MintScsUrlPort));
IRestRequest request = new RestRequest(string.Format("{0}{1}{2}", MstrScsUrlContextRoot, MstrScsUrlPath, SCS_GENERATE_TOKEN_URL_PATH));
request.Timeout = 5000;
request.ReadWriteTimeout = 5000;
request.AddParameter("application/json", lpPostData, ParameterType.RequestBody);
IRestResponse response = await postResultAsync(client, request);
return response.Content;
} // postJsonDataToUrl
private static Task<IRestResponse> postResultAsync(IRestClient client, IRestRequest request) {
return client.ExecutePostTaskAsync(request);
} // PostResultAsync
This is the line where the time out occurs:
IRestResponse response = await postResultAsync(client, request);
I have tried rewriting this using .Net's HttpWebRequest and I get the same problem.
If I lengthen the RESTSharp timeouts, I am able to make calls to the server (using a different client) while the application is "timing out" so I know the server isn't the issue.
The initial version of the code did not have the await async call structure - that was added as an attempt to get more information on the problem.
I am not getting any errors other than the REST timeout.
I have had limited success with forcing a Garbage Collection with this call:
GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
Any thoughts?
It is possible you are hitting the connection limit for .Net apps, as in MS docs:
"By default, an application using the HttpWebRequest class uses a maximum of two persistent connections to a given server, but you can set the maximum number of connections on a per-application basis."
(https://learn.microsoft.com/en-us/dotnet/framework/network-programming/managing-connections).
Closing the connections should help, or you might be able to increase that limit, that is also in the doc
I ended up putting
GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
in a timer that fired every 2 minutes. This completely solved my issue.
This is very surprising to me since my HttpWebRequest code was wrapped in "using" statements so the resources should have been released properly. I can only conclude that .Net was optimizing the use of the class and was trying to reuse a stale class rather than allow me to create a new one from scratch.
A new way of doing things.
var body = #"{ ""key"": ""value"" }";
// HTTP package
var request = new RestRequest("https://localhost:5001/api/networkdevices", Method.Put);
request.AddHeader("Content-Type", "application/json");
request.AddHeader("Keep-Alive", "");// set "timeout=120" will work as well
request.Timeout = 120;
request.AddBody(body);
// HTTP call
var client = new RestClient();
RestResponse response = await client.ExecuteAsync(request);
Console.WriteLine(response.Content);
I need to change a HttpClient.Timeout property after it made a request(s). When I try, I get an exception:
This instance has already started one or more requests. Properties can only be modified before sending the first request.
Is there any way to avoid this?
There isn't much you can do to change this. This is just default behavior in the HttpClient implementation.
The Timeout property must be set before the GetRequestStream or GetResponse method is called.
From HttpClient.Timeout Remark Section
In order to change the timeout, it would be best to create a new instance of an HttpClient.
client = new HttpClient();
client.Timeout = 20; //set new timeout
Internally the Timeout property is used to set up a CancellationTokenSource which will abort the async operation when that timeout is reached. Since some overloads of the HttpClient methods accept CancellationTokens, we can create helper methods to have a custom timeouts for specific operations:
public async Task<string> GetStringAsync(string requestUri, TimeSpan timeout)
{
using (var cts = new CancellationTokenSource(timeout))
{
HttpResponseMessage response = await _httpClient.GetAsync(requestUri, cts.Token)
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
Lack of support for custom request-level timeouts has always been a shortcoming of HttpClient in my mind. If you don't mind a small library dependency, Flurl.Http [disclaimer: I'm the author] supports this directly:
"http://api.com/endpoint".WithTimeout(30).GetJsonAsync<T>();
This is a true request-level setting; all calls to the same host use a shared HttpClient instance under the hood, and concurrent calls with different timeouts will not conflict. There's a configurable global default (100 seconds initially, same as HttpClient).
i am after some advice/strategy on limiting http requests when consuming multiple web services. I feel i could do this if the requests were happening synchronously, but they are asynchronous and think i should try to perform the limit logic in a way that i wont block.
Due to the web app consuming multiple web services there will be different limits for different requests. I was thinking something like this, but aren't sure how to proceed in a no blocking manner:
request method:
public static Task<string> AsyncRequest(string url, enum webService)
{
using(LimitingClass limiter = new LimitingClass(webService))
{
//Perform async request
}
}
In the LimitingClass it will have logic like checking the last request for the given webservice, if it violates the limit then it will wait a certain amount of time. But in the mean time if another request comes in to a different webservice then i dont want that request to be blocked while the LimitingClass is waiting. Is there anything fundamentally wrong with the above approach? Should i open up a new thread with each LimitingClass instance?
Some pseudo code would be great if possible.
Many Thanks
UPDATE:
This is a simplified version of my current request method:
public static Task<string> MakeAsyncRequest(string url, string contentType)
{
HttpWebRequest request = //set up my request
Task<WebResponse> task = Task.Factory.FromAsync(
request.BeginGetResponse,
asyncResult => request.EndGetResponse(asyncResult),
(object)null);
return task.ContinueWith(t => ReadCallback(t.Result));
}
I just want to wrap this in a using to check the limits and which doesnt block other requests.
So how are you limiting an access to resource without blocking ?
I think you need to look at Semaphores - they allow to limits the number of threads that can access a resource or pool of resources concurrently.
When using the System.Net.WebClient.DownloadData() method I'm getting an unreasonably slow response time.
When fetching an url using the WebClient class in .NET it takes around 10 sec before I get a response, while the same page is fetched by my browser in under 1 sec.
And this is with data that's 0.5kB or smaller in size.
The request involves POST/GET parameters and a user agent header if perhaps that could cause problems.
I haven't (yet) tried if other ways to download data in .NET gives me the same problems, but I'm suspecting I might get similar results. (I've always had a feeling web requests in .NET are unusually slow...)
What could be the cause of this?
Edit:
I tried doing the exact thing using System.Net.HttpWebRequest instead, using the following method, and all requests finish in under 1 sec.
public static string DownloadText(string url)
var request = (HttpWebRequest)WebRequest.Create(url);
var response = (HttpWebResponse)request.GetResponse();
using (var reader = new StreamReader(response.GetResponseStream()))
{
return reader.ReadToEnd();
}
}
While this (old) method using System.Net.WebClient takes 15-30s for each request to finish:
public static string DownloadText(string url)
{
var client = new WebClient();
byte[] data = client.DownloadData(url);
return client.Encoding.GetString(data);
}
I had that problem with WebRequest. Try setting Proxy = null;
WebClient wc = new WebClient();
wc.Proxy = null;
By default WebClient, WebRequest try to determine what proxy to use from IE settings, sometimes it results in like 5 sec delay before the actual request is sent.
This applies to all classes that use WebRequest, including WCF services with HTTP binding.
In general you can use this static code at application startup:
WebRequest.DefaultWebProxy = null;
Download Wireshark here http://www.wireshark.org/
Capture the network packets and filter the "http" packets.
It should give you the answer right away.
There is nothing inherently slow about .NET web requests; that code should be fine. I regularly use WebClient and it works very quickly.
How big is the payload in each direction? Silly question maybe, but is it simply bandwidth limitations?
IMO the most likely thing is that your web-site has spun down, and when you hit the URL the web-site is slow to respond. This is then not the fault of the client. It is also possible that DNS is slow for some reason (in which case you could hard-code the IP into your "hosts" file), or that some proxy server in the middle is slow.
If the web-site isn't yours, it is also possible that they are detecting atypical usage and deliberately injecting a delay to annoy scrapers.
I would grab Fiddler (a free, simple web inspector) and look at the timings.
WebClient may be slow on some workstations when Automatic Proxy Settings in checked in the IE settings (Connections tab - LAN Settings).
Setting WebRequest.DefaultWebProxy = null; or client.Proxy = null didn't do anything for me, using Xamarin on iOS.
I did two things to fix this:
I wrote a downloadString function which does not use WebRequest and System.Net:
public static async Task<string> FnDownloadStringWithoutWebRequest(string url)
{
using (var client = new HttpClient())
{
//Define Headers
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var response = await client.GetAsync(url);
if (response.IsSuccessStatusCode)
{
string responseContent = await response.Content.ReadAsStringAsync();
//dynamic json = Newtonsoft.Json.JsonConvert.DeserializeObject(responseContent);
return responseContent;
}
Logger.DefaultLogger.LogError(LogLevel.NORMAL, "GoogleLoginManager.FnDownloadString", "error fetching string, code: " + response.StatusCode);
return "";
}
}
This is however still slow with Managed HttpClient.
So secondly, in Visual Studio Community for Mac, right click on your Project in the Solution -> Options -> set HttpClient implementation to NSUrlSession, instead of Managed.
Screenshot: Set HttpClient implementation to NSUrlSession instead of Managed
Managed is not fully integrated into iOS, doesn't support TLS 1.2, and thus does not support the ATS standards set as default in iOS9+, see here:
https://learn.microsoft.com/en-us/xamarin/ios/app-fundamentals/ats
With both these changes, string downloads are always very fast (<<1s).
Without both of these changes, on every second or third try, downloadString took over a minute.
Just FYI, there's one more thing you could try, though it shouldn't be necessary anymore:
//var authgoogle = new OAuth2Authenticator(...);
//authgoogle.Completed...
if (authgoogle.IsUsingNativeUI)
{
// Step 2.1 Creating Login UI
// In order to access SFSafariViewController API the cast is neccessary
SafariServices.SFSafariViewController c = null;
c = (SafariServices.SFSafariViewController)ui_object;
PresentViewController(c, true, null);
}
else
{
PresentViewController(ui_object, true, null);
}
Though in my experience, you probably don't need the SafariController.
Another alternative (also free) to Wireshark is Microsoft Network Monitor.
What browser are you using to test?
Try using the default IE install. System.Net.WebClient uses the local IE settings, proxy etc. Maybe that has been mangled?
Another cause for extremely slow WebClient downloads is the destination media to which you are downloading. If it is a slow device like a USB key, this can massively impact download speed. To my HDD I could download at 6MB/s, to my USB key, only 700kb/s, even though I can copy files to this USB at 5MB/s from another drive. wget shows the same behavior. This is also reported here:
https://superuser.com/questions/413750/why-is-downloading-over-usb-so-slow
So if this is your scenario, an alternative solution is to download to HDD first and then copy files to the slow medium after download completes.