I use this to navigate to a website
doc = web.Load("http://google.com/search?btnI=1&q=[my keyword]") //I'm Feeling Lucky
Then I need the url of navigated website... How can I get it?
You could use HtmlWeb.ResponseUri property which gets the URI of the Internet resource that actually responded to the request.
An example - googling for "cookies":
var web = new HtmlWeb();
var doc = web.Load("http://google.com/search?btnI=1&q=cookies");
var responseUrl = web.ResponseUri;
gets the http://en.wikipedia.org/wiki/HTTP_cookie.
You can get the url of current browser by :
string url = HttpContext.Current.Request.Url.AbsoluteUri;
Looks like Sam1 might have given you the right answer (I have no real experience with the HTML Agility Pack) for a one or two instance endeavor.
That being said, if you intend to make a lot of calls to Google using keywords so that you can retrieve the top result (i.e. the "I'm Feeling Lucky" result), then I would highly suggest you use Google's Custom Search API (https://developers.google.com/custom-search/v1/overview).
It would use MUCH less bandwidth if you are pulling JSON results using this API.
Usage of the API only allows for 100 free queries per day. This might fall within your application requirements, but it also might not. If you have the means, I would suggest supporting Google by paying if you intend to make thousands of queries.
There are two things to note here. First - using "http://google.com" in the above URL without the "www" forces a 301 redirect to "http://www.google.com" so you should include the www to keep things simple.
The second is that opening the URL (with the www) performs a 302 redirect. The destination is inside the response headers. So if you can catch that 302 response, you can get the URL Google will send you to, before it sends you there.
Here are the response and request headers for the first request, in which Google performs a 301 redirect to the www domain.
Response Headers
Cache-Control public, max-age=2592000
Content-Length 244
Content-Type text/html; charset=UTF-8
Date Mon, 18 Feb 2013 14:14:40 GMT
Expires Wed, 20 Mar 2013 14:14:40 GMT
Location http://www.google.com/search?btnI=1&q=html5
Server gws
X-Frame-Options SAMEORIGIN
X-XSS-Protection 1; mode=block
Request Headers
Accept text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.5
Connection keep-alive
Cookie PREF=ID=5d01155d00a8d706:U=49fab5927df1f8ad:FF=0:TM=1359732743:LM=1360874099:S=byw-1-fgfbcRWdPN; NID=67=NpFNjRkjTFtyrcYPE-pQeJiMFEgWMWdyVMVpbYATZySlsw63Hz4FCw2Tcr4tynhAhyq1vnuPqmdFBOC65Nd-048ZxrgP_HVtKbVCe7psi-G2aMvsOUbiBl1xYks2xK2K
DNT 1
Host google.com
User-Agent Mozilla/5.0 (Windows NT 6.2; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0
And the response/request headers for the 302 that takes me to the destination page. You can see the destination URL is returned. I've bolded it in the copy.
Response Headers
Cache-Control private
Content-Length 231
Content-Type text/html; charset=UTF-8
Date Mon, 18 Feb 2013 14:14:41 GMT
Location http://en.wikipedia.org/wiki/HTML5
Server gws
X-Frame-Options SAMEORIGIN
X-XSS-Protection 1; mode=block
Request Headers
Accept text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.5
Connection keep-alive
Cookie PREF=ID=5d01155d00a8d706:U=49fab5927df1f8ad:FF=0:TM=1359732743:LM=1360874099:S=byw-1-fgfbcRWdPN; NID=67=NpFNjRkjTFtyrcYPE-pQeJiMFEgWMWdyVMVpbYATZySlsw63Hz4FCw2Tcr4tynhAhyq1vnuPqmdFBOC65Nd-048ZxrgP_HVtKbVCe7psi-G2aMvsOUbiBl1xYks2xK2K
DNT 1
Host www.google.com
User-Agent Mozilla/5.0 (Windows NT 6.2; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0
Related
I have an older C# MVC 4.7 web app that has a specific $.post call. Running locally from Visual Studio 2019 in any browser, I have no problem at all. The call goes through and posts the partial page to the div as expected. Running from our test URL, it gives me an error in Edge and Chrome, but not in Firefox. It returns a 411 error, which I know means it's a content length issue. The problem is that others accessing the site do not run across that issue in any browser. Since it works on others' machines and on one particular browser on my local machine, I suspect it's a security setting or something along those lines for that particular site. I've cleared out the settings, reset to factory default, removed extensions, assured that's up-to-date and tested it with and without antivirus software interaction.
This fails:
$.post('/Controller/Method', function (data) {
$('#container').html(data);
});
But this works:
$.post("/Controller/OtherMethod", { paramOne: varOne, paramTwo: varTwo }, function (data) {
$("#container").html(data);
});
Both functions work in Firefox when running from the test URL, only second one works in Edge/Chrome from test URL.
Any ideas on what I might need to check?
Here's the header from the failing call in Dev Tools:
General
Request URL: https://[url]/Controller/Method?param=123
Request Method: POST
Status Code: 411
Remote Address: [remote_ip]
Referrer Policy: strict-origin-when-cross-origin
Response Headers
content-type: text/html; charset=us-ascii
date: Thu, 19 Aug 2021 17:08:18 GMT
server: Microsoft-HTTPAPI/2.0
Request Headers
:authority: [url]
:method: POST
:path: Controller/Method?param=123
:scheme: https
accept: /
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: no-cache
content-length: 0
cookie: [cookie info]
origin: https://[url]
pragma: no-cache
referer: https://[url]/Controller/Method?param=123
sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google
Chrome";v="92"
sec-ch-ua-mobile: ?0
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159
Safari/537.36
x-requested-with: XMLHttpRequest
After experiencing the same problem in both Chrome and Edge across two different websites, I think I have the answer:
The problem appears to be BitDefender antivirus.
If I disable BitDefender protection, the post requests succeed.
EDIT:
Bizarrely, if I then re-enable BitDefender protection, the issue doesn't come back.
After stumbling across the solution, I found a few references to people having the same problem: https://community.bitdefender.com/en/discussion/88573/status-411-length-required-during-browsing https://support.mozilla.org/en-US/questions/1344632
I have a C# .NET application that is writing a zipped file back to the client for download. However, the browser does not receive the file or it rejects the file. The browser does not show any notifications. I have tried it both on Firefox and Chrome.
I have captured the request and response from the client and server using Fiddler:
Request:
POST http://localhost:62526/Reports/_Report_RewardLetters HTTP/1.1
Host: localhost:62526
Connection: keep-alive
Content-Length: 228
Accept: */*
Origin: http://localhost:62526
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
DNT: 1
Referer: http://localhost:62526/Reports
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Response:
HTTP/1.1 200 OK
Cache-Control: private
Transfer-Encoding: chunked
Content-Type: application/octet-stream
Server: Microsoft-IIS/10.0
content-dispostion: filename=Letter.zip
X-AspNet-Version: 4.0.30319
X-SourceFiles: =?UTF-8?B?QzpcVXNlcnNcc2FpbmlfaFxTb3VyY2VcUmVwb3NcUmVzZWFyY2hPZmZpY2VEYXNoYm9hcmRcUmVzZWFyY2hPZmZpY2VEYXNoYm9hcmRcUmVwb3J0c1xfUmVwb3J0X1Jld2FyZExldHRlcnM=?=
X-Powered-By: ASP.NET
Date: Thu, 11 Feb 2016 23:13:22 GMT
....Truncated the file contents.....
My code:
Response.Clear();
Response.ClearHeaders();
Response.ClearContent();
Response.AddHeader("content-dispostion", "filename=MyFile.zip");
Response.ContentType = "application/octet-stream";
Response.Flush();
Response.WriteFile(myfile);
Response.Flush();
Response.End();
I have tried numerous combinations of Response.Flush(), Response.Clear(), HttpContext.ApplicationInstance.CompleteRequest(), Response.BinaryWrite(), Response.TransmitFile(), etc. but none seem to work. Additionally, I have in my code the necessary checks to determine the existence of the file.
From the fiddler captures, I think there is something wrong in the encoding or the server response of the file being sent to the client whereby the browser is rejecting the file without any notification.
Thanks for your help!
Just a thought: do you need the Response.Flush statements? Setting your headers, writing your file and then calling Response.End should be enough.
Also, set the ContentType to "application/zip, application/octet-stream"
I had made a mistake that was causing all the fuss. The view was using an Ajax form to make the request instead of a normal Html form.
I fixed the problem by changing my controller and view according to the information present here:
http://geekswithblogs.net/rgupta/archive/2014/06/23/downloading-file-using-ajax-and-jquery-after-submitting-form-data.aspx
Download Excel file via AJAX MVC
I have code that works, but I'm having a difficult time making the connection as to why it works.
I have code in an AngularJS factory function that makes this call:
$http.get('http://webServerName/PrestoWebApi/api/apps/')
And this is the Web API controller (C#):
[EnableCors(origins: "http://webServerName", headers: "*", methods: "*")]
public class AppsController : ApiController
The source of the call would be a user's computer, for example, a laptop with the name JoesLaptop. And that laptop could run anywhere. (Currently, this is all running inside one LAN, but the user could be anywhere.)
So why does specifying the web server name within the EnableCors attribute work? Isn't the request coming from the browser on Joe's laptop and not from the web server itself?
Edit
If I remove the EnableCors attribute, I get this error in the F12 tools in the browser:
XMLHttpRequest cannot load http://webServerName/PrestoWebApi/api/apps/. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://webServerName' is therefore not allowed access.
Edit 2
Request:
GET http://fs-6103.fs.local/PrestoWebApi/api/apps/ HTTP/1.1
Host: fs-6103.fs.local
Connection: keep-alive
Accept: application/json, text/plain, */*
Origin: http://fs-6103
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36
Referer: http://fs-6103/PrestoWebApi/app/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Response:
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/7.5
Access-Control-Allow-Origin: http://fs-6103
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Fri, 31 Oct 2014 18:30:05 GMT
Content-Length: 2931171
If webServerName is rendering an HTML page that is starting an asynchronous request to webServerName, then CORS doesn't apply and your server will serve that resource anyway.
I'm pretty sure that must be your case.
UPDATE
Based on the latest edits of the question and the comments that the OP has made bellow this answer, this is what must be happening.
The HTTP server that it's serving both the main HTML page and the API resource is the same, therefore there shouldn't be any need to EnableCORS. However, according to the headers of the Request the page is being served from http://fs-6103 and the $http.get is made to http://fs-6103.fs.local. That should explain everything.
UPDATE 2
Ok, I'm willing to bet that this is what's happening here:
The main page is being served by http://fs-6103
The $http.get is made towards: http://fs-6103.fs.local/
So far I'm not speculating, this is what the request is saying
The OP must have [EnableCors(origins: "http://fs-6103", headers: "*", methods: "*")] set into the API controller.
When this is disabled the OP is getting the error: No 'Access-Control-Allow-Origin' header is present on the requested resource, as it should be expected. And when the OP enables it everything works as expected.
Browsers/clients handle the security, and generally restrict things to single origin, meaning they only accept stuff from the server they made the request to. Enabling cors in the header (ACAO) or wherever lets that server tell the browser, "hey those other Cross origin resources are with me." The browser will generally go along with that.
Using a TCP Client for http post. I get a different result than excepted.
No HTTP 200 OK...
Here is my Request:
GET / HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Accept-Language: de-DE
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0)
UA-CPU: AMD64
Accept-Encoding: gzip, deflate
Host:www.mywebsite.com
Connection: Keep-Alive
Cache-Control: max-age=0
Here is my Response:
HTTP/1.1 302 Moved Temporarily
Server: Apache-Coyote/1.1
Location: http://somewhere.com (url changed)
Content-Length: 0
Date: Fri, 03 Aug 2012 10:38:14 GMT
Its tell me to redirect to "Location".
The next request should be a GET to http://somewhere.com with a refer to www.mywebsite.com ?
A other scenario is a https link. Its often a total different IP than the HostEntry returns. How to get the IP ? Is there an easy way in c# ?
Yes a 302 means that the url is not at the location you want (the host may have moved it, or prefers you use www instead of the apex). So just use a HTTP GET for the new URL.
In terms of the IP Address, I'm afraid its not so simple, you can have multiple IP Addresses for a hostname
string stackoverflow= "stackoverflow.com"
IPAddress[] addresslist = Dns.GetHostAddresses(stackoverflow);
foreach (IPAddress theaddress in addresslist)
{
Console.WriteLine(theaddress.ToString());
}
There are lots of situations where more than one IP is used (mainly load balancing situations).
This is going to sound quite bizarre. I have one ASP .NET MVC 2 application. Works great. Routing is not very complicated. I am running this on windows 2003 IIS 6, so I have to use the {controller}.aspx routing configuration.
Anyhow, I've set the same MVC 2 application up twice under different virtual directories on IIS. This is literally a copy/paste of the existing app files to the new directory. The virtual directory / app name is of course different for each.
Routing works perfect in the original application. In the second, I can't seem to get the login page to post to the login post action, let alone make it beyond that.
The most complicated url in my application is something akin to:
http://server:90/appName/controller.aspx/AnAction?StartPeriod=2008
My global.asax.cs looks like:
routes.IgnoreRoute("{resource}.html/{*pathInfo}");
routes.IgnoreRoute("{resource}.axd/{*pathInfo}");
routes.MapRoute(
"Default",
"{controller}.aspx/{action}/{id}",
new { controller = "Account", action = "LogOn", id = UrlParameter.Optional }
);
routes.MapRoute(
"Root",
"",
new { controller = "Account", action = "LogOn", id = "" }
);
The application only has 2 controllers: Account & Mpa. Again, the exact same compiled code exists in both places on the same IIS server under different app names/virtual directories. Can anyone think of a reason I would get different behavior in the copied one (doesn't work/can't find views/throws 404's for just about everything beyond displaying the LogOn page of Account controller)?
EDIT 1
Another thing worth mentioning is that I'm using ValidateAntiForgeryToken attributes on all post methods in the Mpa.aspx controller. This is the controller that should be redirected to after the post to Account.aspx (which doesn't seem to really happen). I thought the issue was that the salt value used for the ValidateAntiForgeryToken was the same for both applications since I copied & pasted. I've since changed the value for the original application, but I'm still getting the same results.
FYI: Here are the details returned from firebug.
**GET Account.aspx**
http://server:90/MpaDemo/Account.aspx 200 OK 2.9 KB 15ms
ParamsHeadersPostPutResponseCacheHTML
Response Headersview source
Date Thu, 26 Aug 2010 20:42:57 GMT
Server Microsoft-IIS/6.0
X-Powered-By ASP.NET
X-AspNet-Version 2.0.50727
X-AspNetMvc-Version 2.0
Set-Cookie ASP.NET_SessionId=z1q43ezg2hvtx255i5y5df21; path=/; HttpOnly
Cache-Control private
Content-Type text/html; charset=utf-8
Content-Length 2921
Request Headersview source
Host server:90
User-Agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 (.NET CLR 3.5.30729)
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 115
Connection keep-alive
Authorization NTLM TlRMTVNTUAADAAAAGAAYAHoAAAAYABgAkgAAAAoACgBIAAAADgAOAFIAAAAaABoAYAAAAAAAAACqAAAABYKIogUBKAoAAAAPdABtAGsAbgBiAGIAMAAwADUAagB4AGIATQBTAEoAQQBLAE0ATwBMADIANAAxADAAOADIJ4z3L+WAUAAAAAAAAAAAAAAAAAAAAABhigoN+1bIPZirxXzNQHWNIu/rx4Senq8=
**Account.aspx POST**
Date Thu, 26 Aug 2010 20:44:26 GMT
Server Microsoft-IIS/6.0
X-Powered-By ASP.NET
X-AspNet-Version 2.0.50727
X-AspNetMvc-Version 2.0
Location /MpaDemo/Mpa.aspx/CustomErrorView?aspxerrorpath=/MpaDemo/Account.aspx
Cache-Control private
Content-Type text/html; charset=utf-8
Content-Length 200
Request Headersview source
Host SERVER:90
User-Agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 (.NET CLR 3.5.30729)
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 115
Connection keep-alive
Referer http://SERVER:90/MpaDemo/Account.aspx
Cookie ASP.NET_SessionId=z1q43ezg2hvtx255i5y5df21
**GET CustomErrorView**
Response Headersview source
Date Thu, 26 Aug 2010 20:44:26 GMT
Server Microsoft-IIS/6.0
X-Powered-By ASP.NET
X-AspNet-Version 2.0.50727
Cache-Control private
Content-Type text/html; charset=utf-8
Content-Length 1534
Request Headersview source
Host server:90
User-Agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 (.NET CLR 3.5.30729)
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 115
Connection keep-alive
Referer http://server:90/MpaDemo/Account.aspx
Cookie ASP.NET_SessionId=z1q43ezg2hvtx255i5y5df21
Maybe switch from IIS 6 to IIS Developer Express which should run on 2003 and support all the new MVC features.
Quote from ScottGu's site:
•It works on Windows XP and higher operating systems – giving you a full IIS 7.x developer feature-set on all OS platforms
http://weblogs.asp.net/scottgu/archive/2010/06/28/introducing-iis-express.aspx
http://blogs.iis.net/vaidyg/archive/2010/07/06/introducing-iis-developer-express.aspx
Download
Okay, this is annoying. I forgot to check permissions on the folders I copied. Apparently, attempting to write to the log & not being able to threw an error, which my try/catch block caught, but then in the catch, I try logging the error, which threw another one, which my general catch all page should have caught, but didn't because the web.config had it under another controller.
sigh
Long story short, I removed the controller requirement for the shared error page like so:
Was: defaultRedirect="~/Mpa/CustomErrorView" Is now: defaultRedirect="~/CustomErrorView"
The view for the error page is in the Shared directory of the Views folder.
Lastly, I added the correct permissions to my logging folder.
Now it all works.
Wow. This took me 6 hours to figure out. I feel really, really lame.