I've tried everything I could find on these pages, Webclient, HttpClient, HttpWebRequest etc. I've looked at fiddler and copied all the cookies, added all the same headers and user agents etc. These all work for almost every other page I try, but for some reason Toys'R'Us just times out. Everything was working fine yesterday, it just seems to have stopped working today.
First time I've posted here but could somebody just try any method for downloading the html for a random example of:
http://www.toysrus.co.uk/toys/transformers-robots-in-disguise-strongarm-one-step-changers/0148891
No matter what I've tried it just seems to hang and timeout. The really annoying thing is that it loads fine in chrome, internet explorer, fiddler... I just can't seem to get it to download in c# regardless of downloadstring, HttpWebRequest etc, it just seems to hang and timeout.
To be clear, I had this working yesterday and has been for months, something must have changed on their servers but I can't figure out what.
If somebody could try it and test if it's just my setup that's failing I'd greatly appreciate it.
very odd this, finally got it working by adding these specific headers:
Accept: text/html, application/xhtml+xml, /
Accept-Language: en-GB
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Accept-Encoding: gzip, deflate
on a webclient downloadstring. I'm sure I tried this yesterday... but it's now working today. They must have changed something back their end I guess. Anyway... can carry on now. Thanks again Gianlucca.
Related
I have an older C# MVC 4.7 web app that has a specific $.post call. Running locally from Visual Studio 2019 in any browser, I have no problem at all. The call goes through and posts the partial page to the div as expected. Running from our test URL, it gives me an error in Edge and Chrome, but not in Firefox. It returns a 411 error, which I know means it's a content length issue. The problem is that others accessing the site do not run across that issue in any browser. Since it works on others' machines and on one particular browser on my local machine, I suspect it's a security setting or something along those lines for that particular site. I've cleared out the settings, reset to factory default, removed extensions, assured that's up-to-date and tested it with and without antivirus software interaction.
This fails:
$.post('/Controller/Method', function (data) {
$('#container').html(data);
});
But this works:
$.post("/Controller/OtherMethod", { paramOne: varOne, paramTwo: varTwo }, function (data) {
$("#container").html(data);
});
Both functions work in Firefox when running from the test URL, only second one works in Edge/Chrome from test URL.
Any ideas on what I might need to check?
Here's the header from the failing call in Dev Tools:
General
Request URL: https://[url]/Controller/Method?param=123
Request Method: POST
Status Code: 411
Remote Address: [remote_ip]
Referrer Policy: strict-origin-when-cross-origin
Response Headers
content-type: text/html; charset=us-ascii
date: Thu, 19 Aug 2021 17:08:18 GMT
server: Microsoft-HTTPAPI/2.0
Request Headers
:authority: [url]
:method: POST
:path: Controller/Method?param=123
:scheme: https
accept: /
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: no-cache
content-length: 0
cookie: [cookie info]
origin: https://[url]
pragma: no-cache
referer: https://[url]/Controller/Method?param=123
sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google
Chrome";v="92"
sec-ch-ua-mobile: ?0
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159
Safari/537.36
x-requested-with: XMLHttpRequest
After experiencing the same problem in both Chrome and Edge across two different websites, I think I have the answer:
The problem appears to be BitDefender antivirus.
If I disable BitDefender protection, the post requests succeed.
EDIT:
Bizarrely, if I then re-enable BitDefender protection, the issue doesn't come back.
After stumbling across the solution, I found a few references to people having the same problem: https://community.bitdefender.com/en/discussion/88573/status-411-length-required-during-browsing https://support.mozilla.org/en-US/questions/1344632
I'm trying to create an app using C# to retrieve my data cap infos from my ISP site. The page is this one but I suspect it can't be accessed from outside their network so if anyone needs more information just ask.
The page loads through AJAX the remaining traffic quota and displays it in the page after it's loaded. Right now I already have a working app using HtmlAgilityPack but it pretty hideous considering I'm using a WebBrowser control to load the page in background, wait for five seconds, parse the page's HTML with the library and see if it finds the necessary HTML string; if not the timer resets and repeats itself until the javascript has done its thing and loaded the data cap infos.
I want to somehow replicate what the webpage does and call the server and ask directly for the infos without creating a web browser instance in background and waiting for the infos to load.
Is it possible?
URL http://internet.tre.it/calls/checkMSISDN.aspx?g=2518607185932962118&h=UItDOr88/CtwONsfqfLgblVuTAysHYKc3kh6mLgiX0He49TU0I9lc56O8mWVhxzd3yFUDFF08P/Ng/5cg2nLtefFfjUIBq/QNQalmmSnKkQ=&mc=22299&acid=0&_=1541582209456
Headers
Host: internet.tre.it
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept: application/json, text/javascript, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://internet.tre.it/
X-Requested-With: XMLHttpRequest
DNT: 1
Connection: keep-alive
Cookie: cookiesAccepted=1; _iub_cs-160673=%7B%22id%22%3A160673%2C%22consent%22%3Atrue%2C%22timestamp%22%3A%222018-04-16T15%3A42%3A10.978Z%22%2C%22version%22%3A%220.13.22%22%7D; ASP.NET_SessionId=n2wz2brfaepfj2klo0nqfwaw; pageVisit=c73074b54dbe40d49a715aeb9a0f4ea8; 148162__148162_d32646f682e342dba303540b0f10dac1=1
Response
Album of the JSON response. (I blacked out those two lines because they were respectively my own name and my phone number)
The response being a json string, I recommend the following :
Write code to download the json string from the URL. See for instance
https://stackoverflow.com/a/11891101/4180382
Copy the whole json string from your F12 response tab
In Visual Studio create a new class file
Click Edit > Paste special > Paste Json as classes.
In your code you will need the name of the first class that you pasted. It is the parent class. I would say it is 'Banners', but verify.
var obj = JsonConvert.DeserializeObject < Banners>(downloadedJson);
Now you can loop through the Menu array to extract all of the info you need.
And you are done! If all the info is in the json, no need to use HtmlAgilityPack. Let me know how you fare.
I have implemented Unity.mvc3 into my project but now my Knockout AJAX methods are starting a Longpolling process and I do not know why, can anyone help please?
This is the JSON response (I have not got a clue why I am getting
this)- {"C":"d-13044D90-B,0|E,2|F,0","M":[]}
A%2F%2Flocalhost%3A1764%2Findex.html&browserName=Chrome&tid=5&_=1403698144789
Request Method:GET Status Code:200 OK Request Headersview source
Accept:text/plain, /; q=0.01 Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8 Cache-Control:no-cache
Connection:keep-alive Content-Type:application/x-www-form-urlencoded;
charset=UTF-8
Cookie:__RequestVerificationToken=j6VFekQ7Po1EfD9wSUK4e4A_ts1SVuGIbRwDG727whnb8l--9P5v5maO-FhCOjFLitRIegjYixEX9698kZR_JWHvo7lUmFYfOwVNjwQ7Hhg1
Host:localhost:7356
Pragma:no-cache
User-Agent:Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/35.0.1916.153 Safari/537.36 Query String
Parametersview sourceview URL encoded transport:longPolling
connectionToken:AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAANVlw/ETQ70WfNHf1KMvu6gAAAAACAAAAAAAQZgAAAAEAACAAAACnUs1Jjk7unYzODmvpJNj7Nbvay/Dx4kSOH+V/CtVDTwAAAAAOgAAAAAIAACAAAAAm7NaW0uCoayuqpNT8z8+uLy/Uio+Nbh8g+VDE0X1/8TAAAAAKXR/1gbPBSIz2WagA2zJI6Te45f63pWeiYkXBGYlYeO+WbWbkTycWNrGmRqaUY8JAAAAAsCIr6JEw/gAUfIClKEjm3cBXG3+I33yhob1f3jMrvmLQTeDC7hphp1SAz+utVN28VstEmeExHyyuycRP/upWIw==
messageId:d-13044D90-B,0|E,2|F,0
browserName:Chrome tid:5
_:1403698144789 Response Headersview source Access-Control-Allow-Credentials:true
Content-Type:application/json; charset=UTF-8 Date:Wed, 25 Jun 2014
12:09:10 GMT Server:Microsoft-HTTPAPI/2.0 Transfer-Encoding:chunked
X-Content-Type-Options:nosniff
And this is the answer, add key="vs:EnableBrowserLink" value="false" to the web config between the appSettings tags the automatic polling has stopped, I found out this happens by default in VS2012 and now no more auto polling :-) - . I also cannot believe no one posted anything for this but anyways this worked a treat.
I am trying to download a file from a server using System.Web.
It actually works, but some links give me trouble. The links look like this:
http://cdn.somesite.com/r1KH3Z%2FaMY6kLQ9Y4nVxYtlfrcewvKO9HLTCUBjU8IBAYnA3vzE1LGrkqMrR9Nh3jTMVFZzC7mxMBeNK5uY3nx5K0MjUaegM3crVpFNGk6a6TW6NJ3hnlvFuaugE65SQ4yM5754BM%2BLagqYvwvLAhG3DKU9SGUI54UAq3dwMDU%2BMl9lUO18hJF3OtzKiQfrC/the_file.ext
The code looks basically like this:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(link);
WebResponse response = request.getResponse();
getResponse() always throws an exception (Error 400 Bad Request).
However, I know the link works because I can download the file with Firefox without problems.
I also tried to decode the link with Uri.UnescapeDataString(link), but that link wont even work in Firefox.
Other links work perfectly fine this way.. just these won't work.
Edit:
Okay, i found something out using wireshark:
If i open the link using Firefox, this is sent:
&ME3#"dM*PNyAo PA:]GET /r1KH3Z%2FaMY6kLQ9Y4nVxYp5DyNc49t5kJBybvjbcsJJZ0IUJBtBWCgri3zfTERQught6S8ws1a%2BCo0RS5w3KTmbL7i5yytRpn2QELEPUXZTGYWbAg5eyGO2yIIbmGOcFP41WdrFRFcfk4hAIyZ7rs4QgbudzcrJivrAaOTYkEnozqmdoSCCY8yb1i22YtEAV/epd_outpost_12adb.flv HTTP/1.1
Host: cdn.somesite.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20100101 Firefox/12.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Connection: keep-alive
I think only the first line is the problem, because WebRequest.Create(link) decodes the url:
&MEz.#!dM/nP9#~P>.GET /r1KH3Z/aMY6kLQ9Y4nVxYp5DyNc49t5kJBybvjbcsJJZ0IUJBtBWCgri3zfTERQught6S8ws1a%2BCo0RS5w3KTmbL7i5yytRpn2QELEPUXZTGYWbAg5eyGO2yIIbmGOcFP41WdrFRFcfk4hAIyZ7rs6Mmh1EsQQ4vJVYUwtbLBDNx9AwCHlWDfzfSWIHzaaIo/epd_outpost_12adb.flv HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20100101 Firefox/12.0
Host: cdn.somesite.com
( %2F is replaced with / )
Another edit:
I found out that the Uri class decodes the url automatically:
Uri uri = new Uri(link); //link is not decoded
Debug.WriteLine(uri.ToString()); //link is decoded here.
How can I prevent this?
Thanks in advance for your help.
By default, the Uri class will not allow an escaped / character (%2f) in a URI (even though this appears to be legal in my reading of RFC 3986).
Uri uri = new Uri("http://example.com/embed%2fded");
Console.WriteLine(uri.AbsoluteUri); // prints: http://example.com/embed/ded
(Note: don't use Uri.ToString to print URIs.)
According to the bug report for this issue on Microsoft Connect, this behaviour is by design, but you can work around it by adding the following to your app.config or web.config file:
<uri>
<schemeSettings>
<add name="http" genericUriParserOptions="DontUnescapePathDotsAndSlashes" />
</schemeSettings>
</uri>
(Since WebRequest.Create(string) just delegates to WebRequest.Create(Uri), you would need to use this workaround no matter which method you call.)
This has now changed in .NET 4.5. By default you can now use escaped slashes. I posted more info on this (including screenshots) in the comments here: GETting a URL with an url-encoded slash
In this answer to a Meta question, Jeff states that he has fixed the problem where visits created by the "Top Sites" feature in Safari were counted for the Fanatic and Enthusiast badges.
My question is: how can this be done? How I programatically tell a "top sites" visit and a regular visit apart?
I'm looking for a general solution, although I would like to implement code that differentiates between those visit types in C# for an ASP.NET MVC site.
NOTE: I'm not quite sure how to tag this question. If you have any ideas, please retag!
I have just looked at the headers that safari issues on the request and you can see an x-purpose: preview value. Check for this header to avoid false positives on visits:
GET http://www.yahoo.com/ HTTP/1.1
Host: www.yahoo.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0 Safari/533.16
X-Purpose: preview
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-US
Accept-Encoding: gzip, deflate
Connection: keep-alive
Connection: keep-alive
Hope this helps!
Just off the top of my head here, couldn't it be that this "Fanatic-badgecounter" only counts when you've requested at least more than one different page in one day?