We Keep Coding

HttpClient high data usage when getting remote file size

I'm getting the file size of remote urls and I just noticed the difference between HttpClient and httpWebRequest.
I compared and I noticed that httpclient is taking too much data.
this is a big issue for me because, in the Philippines we are only have limited data
Could you please tell me what's wrong with my httpclient class? I can't figure out what is causing the high data usage
HttpClient
HttpClientHandler handler = new HttpClientHandler()
{
Proxy = null,
UseProxy = false,
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
var client = new HttpClient(handler);
client.DefaultRequestHeaders.Add("Method", "GET");
client.DefaultRequestHeaders.Add("Referer", uriPath.AbsoluteUri);
client.DefaultRequestHeaders.Add("Origin", "https://www.samplesite.com");
client.DefaultRequestHeaders.ConnectionClose = true;
client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36");
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("deflate"));
using (HttpResponseMessage response = await client.GetAsync(uriPath, HttpCompletionOption.ResponseHeadersRead, token).ConfigureAwait(false))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
var resultTask = response.Content.ReadAsStringAsync();
var timeoutTask = Task.Delay(3000);
var completed = await Task.WhenAny(resultTask, timeoutTask).ConfigureAwait(false);
if (completed == timeoutTask)
return null;
return await resultTask;
}
HttpWebRequest
var webRequest = (HttpWebRequest)WebRequest.Create(uriPath);
webRequest.Method = "HEAD";
webRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36";
using (var webResponse = await webRequest.GetResponseAsync())
{
return await Task.Run(() => webResponse.Headers.Get("Content-Length"), token);
}

You are using different HTTP methods GET in case of HttpClient & HEAD in case of WebRequest. To get file size you will enough HEAD method in both cases
The HTTP GET method requests a representation of the specified resource. Requests using GET should only retrieve data.
The HTTP HEAD method requests the headers that are returned if the specified resource would be requested with an HTTP GET method. Such a request can be done before deciding to download a large resource to save bandwidth, for example.
You need to change this code line
client.DefaultRequestHeaders.Add("Method", "GET");
it MUST BE
client.DefaultRequestHeaders.Add("Method", "HEAD");
A response to a HEAD method does not have a body in contradistinction to GET
UPD: use SendAsync method (not GetAsync)
HttpClientHandler handler = new HttpClientHandler();
using var client = new HttpClient(handler);
string requestUri = "enter request uri";
HttpRequestMessage message = new HttpRequestMessage(HttpMethod.Head, requestUri);
using var response = await client.SendAsync(message);

Finally it's solved Big thanks to #Dmitry
Here's the HttpClient updated code
public static async Task<string> GetTotalBytes(Uri uriPath)
{
HttpClientHandler handler = new HttpClientHandler();
handler.Proxy = null;
using (var client = new HttpClient(handler))
{
HttpRequestMessage message = new HttpRequestMessage(HttpMethod.Head, uriPath);
using (var response = await client.SendAsync(message))
{
response.EnsureSuccessStatusCode();
var lenght = response.Content.Headers.ContentLength;
return lenght.ToString();
}
}
}
RESULT (HttpClient):
Total URL: 355
Filsize: 1.14 GB
Elapsed Time: 0h 0m 2s 51.3ms
and here's HttpWebRequest (Maybe someone will need)
var webRequest = (HttpWebRequest)WebRequest.Create(uriPath);
webRequest.Method = "HEAD";
webRequest.Proxy = null;
using (var webResponse = await webRequest.GetResponseAsync())
{
return await Task.Run(() => webResponse.Headers.Get("Content-Length"), token);
}

System.Net.Http.HttpRequestException with UnauthorizedException

I am trying to convert my Python code into C# code but I got some exceptions during the translation.
What I want is to fetch json/string data from the url. I am new to C# network programming. I tried several ways, read there documents, google their usages, but I still couldn't find out the correct way, as I keep getting the exceptions in the title.
This is my Python code that works:
url = 'https://c.y.qq.com/soso/fcgi-bin/client_search_cp'
params = {
'ct': 24,
'qqmusic_ver': 1298,
'new_json': 1,
'remoteplace':'sizer.yqq.lyric_next',
'searchid': 63514736641951294,
'aggr': 1,
'cr': 1,
'catZhida': 1,
'lossless': 0,
'sem': 1,
't': 7,
'p': 1,
'n': 1,
'w': keyword,
'g_tk': 1714057807,
'loginUin': 0,
'hostUin': 0,
'format': 'json',
'inCharset': 'utf8',
'outCharset': 'utf-8',
'notice': 0,
'platform': 'yqq.json',
'needNewCode': 0
}
headers = {
'content-type': 'application/json',
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:22.0) Gecko/20100101 Firefox/22.0',
'referer':'https://y.qq.com/portal/search.html'
}
result = requests.get(url, params = params, headers = headers)
This is the C# code that I have tried:
public static async Task<string> SearchLyrics(string keyword)
{
keyword = Uri.EscapeUriString(keyword);
// method 1
string uri = $"https://c.y.qq.com/soso/fcgi-bin/client_search_cp?ct=24&qqmusic_ver=1298&new_json=1&remoteplace=sizer.yqq.lyric_next&searchid=63514736641951294&aggr=1&cr=1&catZhida=1&lossless=0&sem=1&t=7&p=1&n=1&w={keyword}&g_tk=1714057807&loginUin=0&hostUin=0&format=json&inCharset=utf8&outCharset=utf-8&notice=0&platform=yqq.json&needNewCode=0";
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(uri);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
using (var request = new HttpRequestMessage(HttpMethod.Get, uri))
{
request.Headers.Add("User-Agent", "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:22.0) Gecko/20100101 Firefox/22.0");
request.Headers.Add("referer", "https://y.qq.com/portal/search.html");
var response = await client.SendAsync(request);
//response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
return await response.Content.ReadAsStringAsync();
}
}
// method 2
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Add("ct", "24");
client.DefaultRequestHeaders.Add("qqmusic_ver", "1298");
client.DefaultRequestHeaders.Add("new_json", "1");
client.DefaultRequestHeaders.Add("remoteplace", "sizer.yqq.lyric_next");
client.DefaultRequestHeaders.Add("searchid", "63514736641951294");
client.DefaultRequestHeaders.Add("aggr", "1");
client.DefaultRequestHeaders.Add("catZhida", "1");
client.DefaultRequestHeaders.Add("lossless", "0");
client.DefaultRequestHeaders.Add("t", "7");
client.DefaultRequestHeaders.Add("p", "1");
client.DefaultRequestHeaders.Add("n", "1");
client.DefaultRequestHeaders.Add("w", keyword);
client.DefaultRequestHeaders.Add("g_tk", "1714057807");
client.DefaultRequestHeaders.Add("loginUin", "0");
client.DefaultRequestHeaders.Add("hostUin", "0");
client.DefaultRequestHeaders.Add("format", "json");
client.DefaultRequestHeaders.Add("inCharset", "utf8");
client.DefaultRequestHeaders.Add("outCharset", "utf-8");
client.DefaultRequestHeaders.Add("notice", "0");
client.DefaultRequestHeaders.Add("platform", "yqq.json");
client.DefaultRequestHeaders.Add("needNewCode", "0");
using (var request = new HttpRequestMessage(HttpMethod.Get, "https://c.y.qq.com/soso/fcgi-bin/client_search_cp"))
{
request.Headers.Add("User-Agent", "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:22.0) Gecko/20100101 Firefox/22.0");
request.Headers.Add("referer", "https://y.qq.com/portal/search.html");
var response = await client.SendAsync(request);
//response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
return await response.Content.ReadAsStringAsync();
}
}
}

The problem is that I didn't enable network access ability for my uwp app. Adding it in the Package.appxmanifest would solve this issue.
My code is right but I have made some improvements according to the Microsoft Document:
public static async Task<string> SearchLyrics(string keyword)
{
string uri = $"https://c.y.qq.com/soso/fcgi-bin/client_search_cp?ct=24&qqmusic_ver=1298&new_json=1&remoteplace=sizer.yqq.lyric_next&searchid=63514736641951294&aggr=1&cr=1&catZhida=1&lossless=0&sem=1&t=7&p=1&n=1&w={Uri.EscapeUriString(keyword)}&g_tk=1714057807&loginUin=0&hostUin=0&format=json&inCharset=utf8&outCharset=utf-8&notice=0&platform=yqq.json&needNewCode=0";
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.UserAgent.TryParseAdd("Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:22.0) Gecko/20100101 Firefox/22.0");
var response = await client.GetAsync(uri);
response.EnsureSuccessStatusCode();
var content = await response.Content.ReadAsStringAsync();
Windows.Data.Json.JsonObject json = Windows.Data.Json.JsonObject.Parse(content);
return json.GetNamedObject("data").GetNamedObject("lyric").GetNamedArray("list").GetObjectAt(0).GetNamedString("content");
}
}

API error 401 Unauthorized laravel C Sharp

I am developing an application using C # that calls a RESTApi de laravel. Through Postman it works correctly, but instead of C# no, it returns error 401.
If I remove the following if in the Laravel controller it works fine:
if ($ request-> isJson ()) {
The header Content-Type is set to application/json
Laravel Code
function getResult(Request $request, $id)
{
if ($request->isJson()) {
// Eloquent
$times = Result::selectRaw('THE SELECT')
->where('ID', $Id)
->get();
$result = [];
foreach($times as $key => $time)
{
...........
}
sort($result);
return response()->json(['results'=>$result], 200);
}
return response()->json(['error' => 'Unauthorized'], 401, []);
}
C#
public static async Task<dynamic> GETTimes(int eventID, int stageID)
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(baseUrl);
client.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "application/json");
client.DefaultRequestHeaders.TryAddWithoutValidation("cache-control", "no-cache");
client.DefaultRequestHeaders.Add("User-Agent", #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36");
// Add the Authorization header with the AccessToken.
//client.DefaultRequestHeaders.Add("Authorization", "Bearer " + accessToken);
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);
// create the URL string.
string url = string.Format("api/v1/events/{0}/results/{1}", eventID, stageID);
// make the request
HttpResponseMessage response = await client.GetAsync(url);
// parse the response and return the data.
string jsonString = await response.Content.ReadAsStringAsync();
object responseData = JsonConvert.DeserializeObject(jsonString);
return (dynamic)responseData;
}
}

C# HttpClient Authentication

I cant force to work HttpClient Post request...
or rather, I can't understand what I'm doing wrong..
My code below and this's sketch
#region Method 2
HttpClient h = new HttpClient();
var values = new Dictionary<string, string>
{
//{ "csrf_token" , "aacc347b22df2a7b3a20c9674ba59cf4279c8d16:1552967983844" },
//{ "track_id", "501c86f19b745703e9ca54991e90dc3a87" },
//{ "password", "Oguhokuu13" }
};
CookieContainer container = new CookieContainer();
h.DefaultRequestHeaders.Add("Accept", "application/json, text/javascript, */*; q=0.01");
h.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, br");
h.DefaultRequestHeaders.Add("Accept-Language", "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7");
h.DefaultRequestHeaders.Add("Connection", "keep-alive");
h.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/x-www-form-urlencoded")); // Content-Type
h.DefaultRequestHeaders.AcceptCharset.Add(new StringWithQualityHeaderValue("UTF-8"));
h.DefaultRequestHeaders.Add("Origin", "https://passport.yandex.ru");
h.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36");
// h.DefaultRequestHeaders.Add("Referer", "https://passport.yandex.ru/auth/welcome?retpath=https%3A%2F%2Ftaxi-cabinet.mobile.yandex.ru%2Fnews%2F");
h.DefaultRequestHeaders.Add("Host", "passport.yandex.ru");
h.DefaultRequestHeaders.Add("X-Requested-With", "XMLHttpRequest");
var content = new FormUrlEncodedContent(values);
var response = h.PostAsync("https://passport.yandex.ru/auth/add", content);
response.Wait();
var responseString = response.Result.Content.ReadAsStringAsync().Result;
textBox1.Text = responseString.ToString();
#endregion
Then I get response:
\u001f?\b\0\0\0\0\0\0\u0003sN???/Q\b?\u000f\u000eQ?O,-??OLI?\u0002\0d(,h\u0016\0\0\0"
Idk what I'm doing wrong

Garbage text in GetResponseStream()

I keep getting garbage text like this
�\b\0\0\0\0\0\0�\a`I�%&/m�{J�J��t�\b�`$ؐ#������iG#
in my html variable at the end of this function. As you can see, by the commented out code, I have tried both methods of inserting Cookies.
I am very new to HTTPwebRequest / Response methods like this. But from everything I can find on the web I am setting up my method correctly. I would love some help if possible.
Also, when using Fiddler to decode my requests, not all of my cookies are getting sent. I am only sending 1 utma, 1 utmb, 1 utmc, and 1 utmz when my code runs. However, when I log into the site normally, i receive 2 utma, 1 utmb, 2 utmc, and 2 utmz.
I feel that this is the source of my connecting issues but I am not sure.
static void FillCookieJar()
{
Console.WriteLine("Filling cookie jar...\r\n");
try
{
string parameters = "SUPER LONG POST DATA found from TEXTVIEW in Fidler";
Uri target = new Uri("https://foo.bar.com/UserSignIn");
byte[] buffer = Encoding.ASCII.GetBytes(parameters);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(target);
//Cookie chipOne = new Cookie("__utma", "XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XX") { Domain = target.Host };
//Cookie chipTwo = new Cookie("__utma", "XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.X") { Domain = target.Host };
//Cookie chipThree = new Cookie("__utmb", "XXXXXXXXX.X.XX.XXXXXXXXX") { Domain = target.Host };
//Cookie chipFour = new Cookie("__utmc", "XXXXXXXXX") { Domain = target.Host };
//Cookie chipFive = new Cookie("__utmc", "XXXXXXXXX") { Domain = target.Host };
//Cookie chipSix = new Cookie("__utmz", "XXXXXXXXX.XXXXXXXXX.X.X.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)") { Domain = target.Host };
//Cookie chipSeven = new Cookie("__utmz", "XXXXXXXXX.XXXXXXXXX.XX.X.utmcsr=titlesource.com|utmccn=(referral)|utmcmd=referral|utmcct=/") { Domain = target.Host };
//cookieJar.Add(chipOne);
//cookieJar.Add(chipTwo);
//cookieJar.Add(chipThree);
//cookieJar.Add(chipFour);
//cookieJar.Add(chipFive);
//cookieJar.Add(chipSix);
//cookieJar.Add(chipSeven);
request.Headers.Add("__utma", "XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XX");
request.Headers.Add("__utma", "XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.XXXXXXXXX.X");
request.Headers.Add("__utmb", "XXXXXXXXX.X.XX.XXXXXXXXX");
request.Headers.Add("__utmc", "XXXXXXXXX");
request.Headers.Add("__utmc", "XXXXXXXXX");
request.Headers.Add("__utmz", "XXXXXXXXX.XXXXXXXXX.X.X.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)");
request.Headers.Add("__utmz", "XXXXXXXXX.XXXXXXXXX.XX.X.utmcsr=titlesource.com|utmccn=(referral)|utmcmd=referral|utmcct=/");
request.CookieContainer = cookieJar;
request.Method = WebRequestMethods.Http.Post;
request.KeepAlive = true;
request.Accept = "*/*";
request.Headers.Add("Cache-Control", "no-cache");
request.Headers.Add("Accept-Encoding: gzip,deflate,sdch");
request.Headers.Add("Accept-Language: en-US,en;q=0.8");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36";
request.Headers.Add("X-Requested-With: XMLHttpRequest");
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.Referer = "https://foo.bar.com/UserSignIn";
request.Headers.Add("Origin", "https://foo.bar.com");
request.Headers.Add("X-MicrosoftAjax", "Delta=true");
request.ContentLength = buffer.Length;
Stream PostData = request.GetRequestStream();
PostData.Write(buffer, 0, buffer.Length);
PostData.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream Answer = response.GetResponseStream();
StreamReader _Answer = new StreamReader(Answer);
String html = _Answer.ReadToEnd();
response.Close();
}
catch (Exception ex)
{
Console.WriteLine("Error FillingCookieJar");
}
}

You need to decompress the gzipped stream to get back the plain text, which is likely in UTF-8 rather than ASCII (look for a charset attribute on the response's Content-Type header).
You can use the AutomaticDecompression property to automatically decompress the content.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(target);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
The url you see is UrlEncoded. You can use HttpUtility.UrlDecode to get the URL to look like https://foo.bar.com/Vendor.
string decoded = HttpUtility.UrlDecode(html);