HttpClient high data usage when getting remote file size - c#

I'm getting the file size of remote urls and I just noticed the difference between HttpClient and httpWebRequest.
I compared and I noticed that httpclient is taking too much data.
this is a big issue for me because, in the Philippines we are only have limited data
Could you please tell me what's wrong with my httpclient class? I can't figure out what is causing the high data usage
HttpClient
HttpClientHandler handler = new HttpClientHandler()
{
Proxy = null,
UseProxy = false,
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
var client = new HttpClient(handler);
client.DefaultRequestHeaders.Add("Method", "GET");
client.DefaultRequestHeaders.Add("Referer", uriPath.AbsoluteUri);
client.DefaultRequestHeaders.Add("Origin", "https://www.samplesite.com");
client.DefaultRequestHeaders.ConnectionClose = true;
client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36");
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("deflate"));
using (HttpResponseMessage response = await client.GetAsync(uriPath, HttpCompletionOption.ResponseHeadersRead, token).ConfigureAwait(false))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
var resultTask = response.Content.ReadAsStringAsync();
var timeoutTask = Task.Delay(3000);
var completed = await Task.WhenAny(resultTask, timeoutTask).ConfigureAwait(false);
if (completed == timeoutTask)
return null;
return await resultTask;
}
HttpWebRequest
var webRequest = (HttpWebRequest)WebRequest.Create(uriPath);
webRequest.Method = "HEAD";
webRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36";
using (var webResponse = await webRequest.GetResponseAsync())
{
return await Task.Run(() => webResponse.Headers.Get("Content-Length"), token);
}

You are using different HTTP methods GET in case of HttpClient & HEAD in case of WebRequest. To get file size you will enough HEAD method in both cases
The HTTP GET method requests a representation of the specified resource. Requests using GET should only retrieve data.
The HTTP HEAD method requests the headers that are returned if the specified resource would be requested with an HTTP GET method. Such a request can be done before deciding to download a large resource to save bandwidth, for example.
You need to change this code line
client.DefaultRequestHeaders.Add("Method", "GET");
it MUST BE
client.DefaultRequestHeaders.Add("Method", "HEAD");
A response to a HEAD method does not have a body in contradistinction to GET
UPD: use SendAsync method (not GetAsync)
HttpClientHandler handler = new HttpClientHandler();
using var client = new HttpClient(handler);
string requestUri = "enter request uri";
HttpRequestMessage message = new HttpRequestMessage(HttpMethod.Head, requestUri);
using var response = await client.SendAsync(message);

Finally it's solved Big thanks to #Dmitry
Here's the HttpClient updated code
public static async Task<string> GetTotalBytes(Uri uriPath)
{
HttpClientHandler handler = new HttpClientHandler();
handler.Proxy = null;
using (var client = new HttpClient(handler))
{
HttpRequestMessage message = new HttpRequestMessage(HttpMethod.Head, uriPath);
using (var response = await client.SendAsync(message))
{
response.EnsureSuccessStatusCode();
var lenght = response.Content.Headers.ContentLength;
return lenght.ToString();
}
}
}
RESULT (HttpClient):
Total URL: 355
Filsize: 1.14 GB
Elapsed Time: 0h 0m 2s 51.3ms
and here's HttpWebRequest (Maybe someone will need)
var webRequest = (HttpWebRequest)WebRequest.Create(uriPath);
webRequest.Method = "HEAD";
webRequest.Proxy = null;
using (var webResponse = await webRequest.GetResponseAsync())
{
return await Task.Run(() => webResponse.Headers.Get("Content-Length"), token);
}

Related

I try API request post with body type raw to get a token but need understand what is wrong

My method to get tokenKey is :
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(url);
httpRequest.Method = "POST";
httpRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 OPR/52.0.2871.99";
string tokenResponse = null;
HttpClient client = new HttpClient();
HttpResponseMessage response = null;
try
{
client.DefaultRequestHeaders.Add("x-api-key", "key");
if (method.Equals("POST"))
{
httpRequest.Accept = "application/json";
httpRequest.ContentType = "application/json";
var data = #"{""username"":#"""+ login + #""",""password"" :#"""+ password + #"""}";
using (var streamWriter = new StreamWriter(httpRequest.GetRequestStream()))
{
streamWriter.Write(data);
}
var httpResponse = (HttpWebResponse)httpRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
tokenResponse = streamReader.ReadToEnd();
}
}
}
catch (Exception ex)
{
throw new Exception(ex.Message);
}
return tokenResponse;
after var httpResponse = (HttpWebResponse)httpRequest.GetResponse();
this error msg: The underlying connection was closed: Unexpected error on a send.
x-api-key is okay but i don't put the original as the username and password
i do it easy with postman
postman request done
Can anyone help me understand where I'm going wrong?
Could this be a case of mismatch TLS versions? See this answer here:
C# HttpWebRequest The underlying connection was closed: An unexpected error occurred on a send
Also it looks like your HttpWebRequest won't have the correct headers, because you are setting them on the HttpClient object instead. The HttpClient also doesn't seem to be being used for this call.
See documentation, how to use HttpWebRequest and how to write body properly to request. I think the problem is with a request stream.

API error 401 Unauthorized laravel C Sharp

I am developing an application using C # that calls a RESTApi de laravel. Through Postman it works correctly, but instead of C# no, it returns error 401.
If I remove the following if in the Laravel controller it works fine:
if ($ request-> isJson ()) {
The header Content-Type is set to application/json
Laravel Code
function getResult(Request $request, $id)
{
if ($request->isJson()) {
// Eloquent
$times = Result::selectRaw('THE SELECT')
->where('ID', $Id)
->get();
$result = [];
foreach($times as $key => $time)
{
...........
}
sort($result);
return response()->json(['results'=>$result], 200);
}
return response()->json(['error' => 'Unauthorized'], 401, []);
}
C#
public static async Task<dynamic> GETTimes(int eventID, int stageID)
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(baseUrl);
client.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "application/json");
client.DefaultRequestHeaders.TryAddWithoutValidation("cache-control", "no-cache");
client.DefaultRequestHeaders.Add("User-Agent", #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36");
// Add the Authorization header with the AccessToken.
//client.DefaultRequestHeaders.Add("Authorization", "Bearer " + accessToken);
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);
// create the URL string.
string url = string.Format("api/v1/events/{0}/results/{1}", eventID, stageID);
// make the request
HttpResponseMessage response = await client.GetAsync(url);
// parse the response and return the data.
string jsonString = await response.Content.ReadAsStringAsync();
object responseData = JsonConvert.DeserializeObject(jsonString);
return (dynamic)responseData;
}
}

httpclient throw exception on redirect

I am trying to download a webpage with HttpClient, that's my code:
private async Task<string> _doRequest(string url)
{
string result = string.Empty;
var client = HttpClient;
using(var request = new HttpRequestMessage()
{
RequestUri = new Uri(url),
Method = HttpMethod.Get
}){
using (HttpResponseMessage response = client.SendAsync(request).Result)
if (response.Headers.Location == null)
{
using (HttpContent content = response.Content)
{
result = await content.ReadAsStringAsync();
}
}
else
{
result = await _doRequest(response.Headers.Location.ToString());
}
};
return result;
}
HttpClient is a static variable initialized as follow:
var handler = new HttpClientHandler();
handler.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
handler.AllowAutoRedirect = false;
HttpClient = new HttpClient(handler);
HttpClient.DefaultRequestHeaders.Add("User-Agent", #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36");
When I try to execute the code with url = "https://www.gls-italy.com/?option=com_gls&view=track_e_trace&mode=search&numero_spedizione=TE170187747&tipo_codice=nazionale"
I get the following:
Which is what lead me to try with curl:
And here I am lost. To me it looks like a valid 302 with location, but for some reasons HttpClient believe otherwise and just throw an exception.
To be clear initially I was relying on AllowAutoRedirect default value and thrust HttpClient to do the redirect, it didn't work, I was getting this same exception which lead me to try to manage it myself. But with no success.
Anyone knows what's happening? How to make it work?
Thanks in advance.
Objects at Class level:
HttpClientHandler Http_Handler = new HttpClientHandler();
HttpClient Http_Client = new HttpClient();
CookieContainer HttpClCookieJar = new CookieContainer();
HttpClient setup stub:
private void HttpClient_Setup()
{
Http_Handler.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
Http_Handler.AllowAutoRedirect = false;
Http_Handler.CookieContainer = HttpClCookieJar;
Http_Handler.UseCookies = true;
Http_Handler.UseDefaultCredentials = true;
Http_Client.Timeout = new TimeSpan(30000);
Http_Client = new HttpClient(Http_Handler);
Http_Client.DefaultRequestHeaders.Add("User-Agent", #"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0");
Http_Client.DefaultRequestHeaders.Add("Accept-Language", "it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3");
Http_Client.DefaultRequestHeaders.Add("Accept", "*/*");
Http_Client.DefaultRequestHeaders.Add("Cache-Control", "no-cache");
}
Async HttpClient request:
public async Task<string> HttpClient_Request(string RequestURL)
{
string _responseHtml = string.Empty;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 |
SecurityProtocolType.Tls11 |
SecurityProtocolType.Tls12;
try
{
using (HttpRequestMessage _requestMsg = new HttpRequestMessage())
{
_requestMsg.Method = HttpMethod.Get;
_requestMsg.RequestUri = new Uri(RequestURL);
using (HttpResponseMessage _response = await Http_Client.SendAsync(_requestMsg))
{
using (HttpContent _content = _response.Content)
{
_responseHtml = await _content.ReadAsStringAsync();
};
};
};
}
catch (HttpRequestException eW)
{
Console.WriteLine("Message: {} Source: {1}", eW.Message, eW.Source);
}
catch (Exception eX)
{
Console.WriteLine("Message: {} Source: {1}", eX.Message, eX.Source);
}
return _responseHtml;
}

HttpClient is ignoring the `www.` part of the Request URI

I'm trying to send a GET request with HttpClient:
public static HttpClient Client { get
{
var handler = new HttpClientHandler() { AutomaticDecompression = System.Net.DecompressionMethods.Deflate | System.Net.DecompressionMethods.GZip };
var http = new HttpClient(handler) { BaseAddress = new Uri("https://www.example.com/") };
http.DefaultRequestHeaders.Accept.Clear();
http.DefaultRequestHeaders.Accept.TryParseAdd("*/*");
http.DefaultRequestHeaders.AcceptLanguage.TryParseAdd("en-US;q=0.6,en;q=0.4");
http.DefaultRequestHeaders.AcceptEncoding.TryParseAdd("gzip, deflate, br");
http.DefaultRequestHeaders.Host = "example.com";
http.DefaultRequestHeaders.Add("X-Requested-With", "XMLHttpRequest");
http.DefaultRequestHeaders.Add("Connection", "keep-alive");
http.DefaultRequestHeaders.Add("Keep-Alive", "600");
http.DefaultRequestHeaders.UserAgent.TryParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
ServicePointManager
.ServerCertificateValidationCallback +=
(sender, cert, chain, sslPolicyErrors) => true;
return http;
}
...
using (var client = HttpHelper.Client)
using (var res = await client.GetAsync("api/stuff/filter?order=" + order + "&page=" + page))
if (res.IsSuccessStatusCode)
{
var json = await res.Content.ReadAsStringAsync();
var response = JsonConvert.DeserializeObject<AnimeResponse>(json);
return response;
}
Unfortunately this sends a request without the www. part and results in 301 status code as well as many, many failed redirects to the same address (without the www. part).
How can I fix this?
Edit: When I re-send the request with Fiddler it also returns 301, but the n it redirects correctly to wwww.. The HttpClient doesn't, and just redirects to the same www-less URI.

The same code works using HttpWebRequest but not using HttpRequestMessage

I've created HttpClient that I'm using for sending requests:
public static void Initialize()
{
handler = new HttpClientHandler() { UseCookies = false, AllowAutoRedirect = true };
http = new HttpClient(handler) { BaseAddress = new Uri("http://csgolounge.com/mytrades") };
http.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36");
}
After that I'm creating instance of custom class that stores the cookies string for an account (something like id=xxxxxxxx; tkz=xxxxxxxxxx; token=xxxxxxxxxxx.
That's how I'm sending a post request:
public async Task Bump()
{
//if (Bumpable)
//{
var req = new HttpRequestMessage(HttpMethod.Post, "http://www.csgolounge.com/ajax/bumpTrade.php");
req.Headers.Add("Cookie", cookieString);
req.Headers.Add("X-Requested-With", "XMLHttpRequest");
req.Headers.Add("Referer", "http://csgolounge.com/mytrades"); //Not really sure if this does anything but I've run out of smart ideas long time ago
/*Dictionary<string, string> postData = new Dictionary<string, string>()
{
{"trade", offer_id}
};
var encoded = new FormUrlEncodedContent(postData);
*/
req.Content = new StringContent("&trade="+Offer_id, Encoding.UTF8, "application/x-www-form-urlencoded"); //Desperation.. decided to change the encoded dictionary to StringContent
var res = await SteamAccount.http.SendAsync(req);
var html = await res.Content.ReadAsStringAsync();
//}
}
I don't get what's wrong with this code. It seems correct to me.
Also, when I set AllowAutoRedirect = false it returns 301: Moved Permanently error, while normally it returns 200 with no HTML no matter what I pass as content.
What am I doing wrong?
Edit: Here's the JavaScript function I'm basing my request on:
function bumpTrade(trade) {
$.ajax({
type: "POST",
url: "ajax/bumpTrade.php",
data: "trade=" + trade
});
}
I've worked with more complex AJAX before, but this just doesn't seem to work no matter what I do.
Edit: I've lost my patience and switched to HttpWebRequest instead.
Now the method looks like this:
public async Task BumpLegacy()
{
while (true)
{
try
{
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("http://csgolounge.com/ajax/bumpTrade.php");
var cc = new CookieContainer();
MatchCollection mc = Regex.Matches(Account.CookieString, #"\s?([^=]+)=([^;]+);");
foreach (Match m in mc)
cc.Add(new Cookie(m.Groups[1].Value, m.Groups[2].Value, "/", "csgolounge.com"));
httpWebRequest.CookieContainer = cc;
byte[] bytes = Encoding.ASCII.GetBytes("trade=" + Offer_id);
httpWebRequest.Referer = "http://csgolounge.com/mytrades";
httpWebRequest.Headers.Add("X-Requested-With", "XMLHttpRequest");
httpWebRequest.Method = "POST";
httpWebRequest.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36";
httpWebRequest.ContentLength = (long)bytes.Length;
var g = await httpWebRequest.GetRequestStreamAsync();
await g.WriteAsync(bytes, 0, bytes.Count());
g.Close();
var res = await httpWebRequest.GetResponseAsync();
res.Close();
break;
}
catch
{
}
}
}
Maybe I'm just dumb but for me it doesn't seem all that different. Are there some key differences that can be the cause?
Here is code from one of my working systems that submits a POST request through an HTTPClient.
[Route("resource")]
public async Task<dynamic> CreateResource([FromBody]Resource resource)
{
if (resource == null) return BadRequest();
dynamic response = null;
resource.Topic = GetDataFromSomewhereElse();
var message = new PostMessage(resource).BodyContent;
dynamic postRequest = new
{
Message = message
};
var post = JsonConvert.SerializeObject(postRequest);
HttpContent content = new StringContent(post, Encoding.UTF8, "application/json");
using (var client = new HttpClient())
{
client.Timeout = TimeSpan.FromMinutes(1);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
try
{
client.BaseAddress = #"http://localhost:51145/";
HttpResponseMessage postResponse = await client.PostAsync("Resource", content); //"Resource" is a route exposed on the remote host
string json = await postResponse.Content.ReadAsStringAsync();
if (postResponse.StatusCode == HttpStatusCode.BadRequest) return BadRequest();
if (postResponse.StatusCode == HttpStatusCode.InternalServerError) return InternalServerError();
if (postResponse.StatusCode == HttpStatusCode.NotFound) return NotFound();
return json;
}
catch(Exception ex)
{
return InternalServerError(ex);
}
}
}
[Edit]
The "PostMessage" was modified to remove domain-specific details. Here is how BodyContent is defined inside the real "PostMessage" from my solution, to provide you enough context to understand what that "message" actually is and how it works into the sample.
public string BodyContent
{
get
{
string content = "";
Type type = this.GetType();
Assembly assembly = Assembly.GetExecutingAssembly();
string resource = String.Format("{0}.{1}", type.Namespace, this.EmbeddedResourceName);
Stream stream = assembly.GetManifestResourceStream(resource);
StreamReader reader = new StreamReader(stream);
content = reader.ReadToEnd();
return content;
}
}
...and here is PostRequest (again, with domain-specific details trimmed)
public class PostRequest
{
public string Message { get;set; }
}

Categories