would like to change my methods from HttpWebRequest.BeginGetRequestStream and HttpWebRequest.BeginGetResponse to methods which use async and Tasks. I need to use these methods because I work on Windows Phone and other methods like GetRequestStream which is not async are not available on this platform.
This is my method DownloadString:
private static Action<string> completed;
public static void DownloadString(Action<string> c)
{
completed = c;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri("linkhere", UriKind.Absolute));
request.Method = "POST";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5Accept: */*";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.BeginGetRequestStream(RequestCallback, request);
}
private static void RequestCallback(IAsyncResult ar)
{
HttpWebRequest request = (HttpWebRequest)ar.AsyncState;
Stream postStream = request.EndGetRequestStream(ar);
//some post data
request.BeginGetResponse(ResponseCallback, request);
}
private static void ResponseCallback(IAsyncResult ar)
{
HttpWebRequest webRequest = (HttpWebRequest)ar.AsyncState;
HttpWebResponse response = (HttpWebResponse)webRequest.EndGetResponse(ar);
Stream streamResponse = response.GetResponseStream();
StreamReader streamReaders = new StreamReader(streamResponse);
string content = streamReaders.ReadToEnd();
streamResponse.Close();
streamReaders.Close();
response.Close();
completed.Invoke(content);
}
For the next step I used the following link:
Converting Callback Functions to Task Pattern
Now I want to have a wrapper method like this:
public async static Task<string> DownloadWithoutAction()
{
TaskCompletionSource<string> tcs = new TaskCompletionSource<string>();
var task = tcs.Task;
Action<string> callback = tcs.SetResult;
await Task.Factory.StartNew(() =>
{
DownloadString(callback);
}, TaskCreationOptions.AttachedToParent);
return await Task.FromResult(task.Result);
}
The Problem:
This does not work because the callbacks are async and so the thread stops before the callbacks are fired. This is only my guess. How can I work around?
Thanks in advance
First, I strongly recommend that you upgrade to HttpClient.
But if you want to do it the harder way, then you would do something like this:
public static Task<string> DownloadStringAsync()
{
TaskCompletionSource<string> tcs = new TaskCompletionSource<string>();
DownloadString(result => tcs.SetResult(result));
return tcs.Task;
}
You could use FromAsync method from TaskFactory:
var request = HttpWebRequest.CreateHttp("http://url");
var stream = await Task.Factory.FromAsync<Stream>(request.BeginGetRequestStream, request.EndGetRequestStream, TaskCreationOptions.None);
var response = (HttpWebResponse)(await Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, TaskCreationOptions.None));
The method using Tasks could be:
private async Task<string> DownloadString()
{
var request = (HttpWebRequest)WebRequest.Create(new Uri("linkhere", UriKind.Absolute));
request.Method = "POST";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5Accept: */*";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
using (var response = (HttpWebResponse)(await Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, TaskCreationOptions.None)))
using (var streamResponse = new StreamReader(response.GetResponseStream()))
{
return await streamResponse.ReadToEndAsync();
}
}
Related
I'm getting the file size of remote urls and I just noticed the difference between HttpClient and httpWebRequest.
I compared and I noticed that httpclient is taking too much data.
this is a big issue for me because, in the Philippines we are only have limited data
Could you please tell me what's wrong with my httpclient class? I can't figure out what is causing the high data usage
HttpClient
HttpClientHandler handler = new HttpClientHandler()
{
Proxy = null,
UseProxy = false,
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
var client = new HttpClient(handler);
client.DefaultRequestHeaders.Add("Method", "GET");
client.DefaultRequestHeaders.Add("Referer", uriPath.AbsoluteUri);
client.DefaultRequestHeaders.Add("Origin", "https://www.samplesite.com");
client.DefaultRequestHeaders.ConnectionClose = true;
client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36");
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("deflate"));
using (HttpResponseMessage response = await client.GetAsync(uriPath, HttpCompletionOption.ResponseHeadersRead, token).ConfigureAwait(false))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
var resultTask = response.Content.ReadAsStringAsync();
var timeoutTask = Task.Delay(3000);
var completed = await Task.WhenAny(resultTask, timeoutTask).ConfigureAwait(false);
if (completed == timeoutTask)
return null;
return await resultTask;
}
HttpWebRequest
var webRequest = (HttpWebRequest)WebRequest.Create(uriPath);
webRequest.Method = "HEAD";
webRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36";
using (var webResponse = await webRequest.GetResponseAsync())
{
return await Task.Run(() => webResponse.Headers.Get("Content-Length"), token);
}
You are using different HTTP methods GET in case of HttpClient & HEAD in case of WebRequest. To get file size you will enough HEAD method in both cases
The HTTP GET method requests a representation of the specified resource. Requests using GET should only retrieve data.
The HTTP HEAD method requests the headers that are returned if the specified resource would be requested with an HTTP GET method. Such a request can be done before deciding to download a large resource to save bandwidth, for example.
You need to change this code line
client.DefaultRequestHeaders.Add("Method", "GET");
it MUST BE
client.DefaultRequestHeaders.Add("Method", "HEAD");
A response to a HEAD method does not have a body in contradistinction to GET
UPD: use SendAsync method (not GetAsync)
HttpClientHandler handler = new HttpClientHandler();
using var client = new HttpClient(handler);
string requestUri = "enter request uri";
HttpRequestMessage message = new HttpRequestMessage(HttpMethod.Head, requestUri);
using var response = await client.SendAsync(message);
Finally it's solved Big thanks to #Dmitry
Here's the HttpClient updated code
public static async Task<string> GetTotalBytes(Uri uriPath)
{
HttpClientHandler handler = new HttpClientHandler();
handler.Proxy = null;
using (var client = new HttpClient(handler))
{
HttpRequestMessage message = new HttpRequestMessage(HttpMethod.Head, uriPath);
using (var response = await client.SendAsync(message))
{
response.EnsureSuccessStatusCode();
var lenght = response.Content.Headers.ContentLength;
return lenght.ToString();
}
}
}
RESULT (HttpClient):
Total URL: 355
Filsize: 1.14 GB
Elapsed Time: 0h 0m 2s 51.3ms
and here's HttpWebRequest (Maybe someone will need)
var webRequest = (HttpWebRequest)WebRequest.Create(uriPath);
webRequest.Method = "HEAD";
webRequest.Proxy = null;
using (var webResponse = await webRequest.GetResponseAsync())
{
return await Task.Run(() => webResponse.Headers.Get("Content-Length"), token);
}
I've a problem with a C# HTTP GET request.
I send two requests to server:
The first request returns the value of JSESSIONID and it's work:
private string GetSessionId(){
{
string id = null;
try
{
var webRequest = WebRequest.Create("http://www.xxxxxxxxxxxxxxx.it/home") as HttpWebRequest;
var response = "";
webRequest.Date = DateTime.UtcNow;
webRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
webRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
webRequest.Headers.Add("Accept-Language", "it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7,de-DE;q=0.6,de;q=0.5");
webRequest.Connection = "keep-alive";
webRequest.Headers.Add("Cache-Control", "max-age=0");
webRequest.Host = "www.xxxxxxxxxxxxxxxx.it";
webRequest.Headers.Add("Upgrade-Insecure-Requests", "1");
webRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36";
if (webRequest != null)
{
webRequest.Method = "GET";
using (WebResponse resp = webRequest.GetResponse())
{
using (Stream s = resp.GetResponseStream())
{
using (StreamReader sr = new StreamReader(s))
{
id = resp.Headers["Set-Cookie"];
response = sr.ReadToEnd();
Console.WriteLine("Response1> " + resp.Headers);
}
}
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
return id;
}
The second request should return the server response to my search but it return an error "302 Moved Temporarily" or "Attention the session has expired".
private void RequestFromServer(string url)
{
string session = GetSessionId().Split(';')[0];
Console.WriteLine("Session> {0}",session);
try
{
HttpWebRequest webRequest = WebRequest.Create(url) as HttpWebRequest;
var response = "";
if (webRequest != null)
{
Uri target = new Uri(url);
webRequest.Method = "GET";
webRequest.Timeout = 20000;
webRequest.Date = DateTime.UtcNow;
webRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
webRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
webRequest.Headers.Add("Accept-Language", "it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7,de-DE;q=0.6,de;q=0.5");
webRequest.Headers.Add("Cache-Control", "max-age=0");
webRequest.Connection = "keep-alive";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.Headers.Add("Cookie", session);
webRequest.Host = "www.xxxxxxxxxxxxxx.it";
webRequest.Headers.Add("Origin", "http://www.xxxxxxxxxxxx.it");
webRequest.Referer = "http://www.xxxxxxxxxxxxxxxx.it/ricerca/atto/contratti/originario?reset=true&normativi=false";
webRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36";
webRequest.Headers.Add("Upgrade-Insecure-Requests", "1");
var res = webRequest.GetResponse();
using (Stream s = res.GetResponseStream())
{
using (StreamReader sr = new StreamReader(s))
{
response = sr.ReadToEnd();
Console.Write(res.Headers);
Console.WriteLine(response);
}
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
Can someone help me with this problem?
What you probably need here is a CookieContainer. Instead of
webRequest.Headers.Add("Cookie", session);
you should do something like
webRequest.CookieContainer = new CookieContainer();
Cookie cookie = new new Cookie("JSESSIONID", yourSessionIdHere);
// adjust domain accordingly
cookie.Domain = "www.xxxxxxxxxxxxxxx.it";
httpRequest.CookieContainer.Add(cookie);
Code is untested.
The MSDN site also states:
For security reasons, cookies are disabled by default. If you want to use cookies, use the CookieContainer property to enable cookies.
That means in order to use cookies you have to set the CookieContainer explicitly.
I've created HttpClient that I'm using for sending requests:
public static void Initialize()
{
handler = new HttpClientHandler() { UseCookies = false, AllowAutoRedirect = true };
http = new HttpClient(handler) { BaseAddress = new Uri("http://csgolounge.com/mytrades") };
http.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36");
}
After that I'm creating instance of custom class that stores the cookies string for an account (something like id=xxxxxxxx; tkz=xxxxxxxxxx; token=xxxxxxxxxxx.
That's how I'm sending a post request:
public async Task Bump()
{
//if (Bumpable)
//{
var req = new HttpRequestMessage(HttpMethod.Post, "http://www.csgolounge.com/ajax/bumpTrade.php");
req.Headers.Add("Cookie", cookieString);
req.Headers.Add("X-Requested-With", "XMLHttpRequest");
req.Headers.Add("Referer", "http://csgolounge.com/mytrades"); //Not really sure if this does anything but I've run out of smart ideas long time ago
/*Dictionary<string, string> postData = new Dictionary<string, string>()
{
{"trade", offer_id}
};
var encoded = new FormUrlEncodedContent(postData);
*/
req.Content = new StringContent("&trade="+Offer_id, Encoding.UTF8, "application/x-www-form-urlencoded"); //Desperation.. decided to change the encoded dictionary to StringContent
var res = await SteamAccount.http.SendAsync(req);
var html = await res.Content.ReadAsStringAsync();
//}
}
I don't get what's wrong with this code. It seems correct to me.
Also, when I set AllowAutoRedirect = false it returns 301: Moved Permanently error, while normally it returns 200 with no HTML no matter what I pass as content.
What am I doing wrong?
Edit: Here's the JavaScript function I'm basing my request on:
function bumpTrade(trade) {
$.ajax({
type: "POST",
url: "ajax/bumpTrade.php",
data: "trade=" + trade
});
}
I've worked with more complex AJAX before, but this just doesn't seem to work no matter what I do.
Edit: I've lost my patience and switched to HttpWebRequest instead.
Now the method looks like this:
public async Task BumpLegacy()
{
while (true)
{
try
{
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("http://csgolounge.com/ajax/bumpTrade.php");
var cc = new CookieContainer();
MatchCollection mc = Regex.Matches(Account.CookieString, #"\s?([^=]+)=([^;]+);");
foreach (Match m in mc)
cc.Add(new Cookie(m.Groups[1].Value, m.Groups[2].Value, "/", "csgolounge.com"));
httpWebRequest.CookieContainer = cc;
byte[] bytes = Encoding.ASCII.GetBytes("trade=" + Offer_id);
httpWebRequest.Referer = "http://csgolounge.com/mytrades";
httpWebRequest.Headers.Add("X-Requested-With", "XMLHttpRequest");
httpWebRequest.Method = "POST";
httpWebRequest.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36";
httpWebRequest.ContentLength = (long)bytes.Length;
var g = await httpWebRequest.GetRequestStreamAsync();
await g.WriteAsync(bytes, 0, bytes.Count());
g.Close();
var res = await httpWebRequest.GetResponseAsync();
res.Close();
break;
}
catch
{
}
}
}
Maybe I'm just dumb but for me it doesn't seem all that different. Are there some key differences that can be the cause?
Here is code from one of my working systems that submits a POST request through an HTTPClient.
[Route("resource")]
public async Task<dynamic> CreateResource([FromBody]Resource resource)
{
if (resource == null) return BadRequest();
dynamic response = null;
resource.Topic = GetDataFromSomewhereElse();
var message = new PostMessage(resource).BodyContent;
dynamic postRequest = new
{
Message = message
};
var post = JsonConvert.SerializeObject(postRequest);
HttpContent content = new StringContent(post, Encoding.UTF8, "application/json");
using (var client = new HttpClient())
{
client.Timeout = TimeSpan.FromMinutes(1);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
try
{
client.BaseAddress = #"http://localhost:51145/";
HttpResponseMessage postResponse = await client.PostAsync("Resource", content); //"Resource" is a route exposed on the remote host
string json = await postResponse.Content.ReadAsStringAsync();
if (postResponse.StatusCode == HttpStatusCode.BadRequest) return BadRequest();
if (postResponse.StatusCode == HttpStatusCode.InternalServerError) return InternalServerError();
if (postResponse.StatusCode == HttpStatusCode.NotFound) return NotFound();
return json;
}
catch(Exception ex)
{
return InternalServerError(ex);
}
}
}
[Edit]
The "PostMessage" was modified to remove domain-specific details. Here is how BodyContent is defined inside the real "PostMessage" from my solution, to provide you enough context to understand what that "message" actually is and how it works into the sample.
public string BodyContent
{
get
{
string content = "";
Type type = this.GetType();
Assembly assembly = Assembly.GetExecutingAssembly();
string resource = String.Format("{0}.{1}", type.Namespace, this.EmbeddedResourceName);
Stream stream = assembly.GetManifestResourceStream(resource);
StreamReader reader = new StreamReader(stream);
content = reader.ReadToEnd();
return content;
}
}
...and here is PostRequest (again, with domain-specific details trimmed)
public class PostRequest
{
public string Message { get;set; }
}
I've used the following code to perform asynchronous HTTP request in C#.
private static Task GetUrl(string url)
{
var request = (HttpWebRequest)WebRequest.Create(url);
request.UserAgent =
"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36";
request.Accept = "text/html";
return Task
.Factory
.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, url)
.ContinueWith(t =>
{
if (t.IsCompleted)
{
using (var stream = t.Result.GetResponseStream())
{
using (var reader = new StreamReader(stream))
{
//Console.WriteLine("-- Successfully downloaded {0} --", t.AsyncState);
//Console.WriteLine(reader.ReadToEnd());
}
}
}
else if (t.IsFaulted)
{
Console.WriteLine("There was an error downloading {0} - {1}", t.AsyncState, t.Exception);
}
});
}
However I'm not sure how I should modify the above code to support HTTP post as well. Any help is appreciated!
In particular I'd like to know how I should add BeginGetRequestStream and EndGetRequestStream into the current function...
Use the Method property of the HttpWebRequest:
request.Method = "POST";
In HttpClientHandler have propertie AllowAutoRedirect, but on build app for WindowsPhone throw exception:
HttpClientHandler.AllowAutoRedirect is not supported on this platform. Please check HttpClientHandler.SupportsRedirectConfiguration before using HttpClientHandler.AllowAutoRedirect.
I really want to prevent autoredirect.
I tried to use HttpWebRequest:
var client = (HttpWebRequest) WebRequest.Create(connectionUrl);
client.Headers["AllowAutoRedirect"] = "false";
client.Method = "GET";
client.Headers["UserAgent"] = #"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31";
client.ContentType = "application/json";
client.Headers["ContentLength"] = client.ToString().Length.ToString();
client.BeginGetResponse(Callback, client);
private void Callback(IAsyncResult ar)
{
var requestState =(HttpWebRequest) ar.AsyncState;
using (var postStream = requestState.EndGetRequestStream(ar))
{}
}
this code throw exception on EndGetRequestStream: "Value does not fall within the expected range"
I look forward to your help.
I think you are receiving the ArgumentException: Value does not fall within the expected range because you are initiating a BeginGetResponse() on the client but then doing a EndGetRequestStream() in your callback where instead you should call EndGetResponse(). Setting AllowAutoRedirect works fine you just need to fix your code. Try this:
var client = (HttpWebRequest)WebRequest.Create(connectionUrl);
client.AllowAutoRedirect = false;
client.Method = "GET";
client.BeginGetResponse(Callback, client);
private void Callback(IAsyncResult ar) {
var state = (HttpWebRequest)ar.AsyncState;
using (var response = state.EndGetResponse(ar)) {
var streamResponse = response.GetResponseStream();
var streamRead = new StreamReader(streamResponse);
var responseString = streamRead.ReadToEnd();
}
}
Why not use HttpClient with and HttpClientHandler??
HttpClientHandler clientHandler = new HttpClientHandler();
clientHandler.AllowAutoRedirect = false;
// Create an HttpClient using the HttpClientHandler
HttpClient client = new HttpClient(clientHandler);
More info: http://blogs.msdn.com/b/henrikn/archive/2012/08/07/httpclient-httpclienthandler-and-httpwebrequesthandler.aspx