I am trying to log into a website to send SMS via a windows phone 7 app. I have 2 providers working but when I try Vodafone I am running into an error.
From what I gather it seems that the response does not contain cookies, or they are not being read. The request logs in ok and the response I get back is the correct page but it contains no cookies.
The Url:
RequestUrl = String.Format("https://www.vodafone.ie/myv/services/login/Login.shtml?username={0}&password={1}", userSettings.Username, userSettings.Password),
The Request:
Request = (HttpWebRequest)WebRequest.Create((requestCollection.CurrentRequest().RequestUrl));
if (Request.CookieContainer == null)
{
Request.CookieContainer = cookieJar.CookieContainer;
Request.AllowAutoRedirect = true;
Request.AllowReadStreamBuffering = true;
Request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6";
}
Where the code errors as the response cookies could not be evaluated:
public void AddCookiesToContainer(HttpWebResponse response)
{
CookieCollection.Add(response.Cookies);
CookieContainer.Add(response.ResponseUri, CookieCollection);
}
And below is the debugger showing no cookies :(
Which line of the code has the error?
Have you verified that the service does return cookies? (i.e. If you make the same request from a PC)
Edit:
The remote host is returning cookies in it's redirection to the index page but in that page there are no cookies in the response. This would explain why there are no cookies in the collection when you try and use it.
Verify this behaviour against a PC client, inspect the body of the response from index.jsp ans this may contain information to help debug and check the documentation on how the process is supposed to work.
Related
I am trying to download file from these links by using C# WebClient, but I am getting 403 error.
https://www.digikey.com/product-search/download.csv?FV=ffe00035&quantity=0&ColumnSort=0&page=5&pageSize=500
https://www.digikey.com/product-search/download.csv?FV=ffe00035&quantity=0&ColumnSort=0&page=4&pageSize=500
I tried to use different user agents, accept encoding etc.
I replaced and tried https to http from url, but no success.
When I paste these urls in Chrome or FireFox or IE, I am able to download file, sometimes it give 403 error, then I replace https to http from url, it downloads. But no success in webclient
Tried Fiddler to inspect, no success
Can someone try in your system, solve this problem.
Here is my code:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
WebClient client= new WebClient();
Uri request_url = new Uri("https://www.digikey.com/product-search/download.csv?FV=ffe00035&quantity=0&ColumnSort=0&page=5&pageSize=500);
//tried http also http://www.digikey.com/product-search/download.csv?FV=ffe00035&quantity=0&ColumnSort=0&page=5&pageSize=500
client.Headers.Add("user-agent", " Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0");
client.DownloadFile(request_url, #"E:\123.csv");
I know there are many threads related to this topic, I tried all of them, no success, please don't mark duplicate. Try in your system, this <10 lines of code.
Note: the same code is working for other websites, only for this website it is giving error.
As I mentioned in my comment the issue here is that the server is expecting a cookie (specifically 'i10c.bdddb') to be present and is giving a 403 error when it's not. However, the cookie is sent with the 403 response. So you can make an initial junk request that will fail but give you the cookie. After this you can then proceed as normal.
Through some trial and error I was able to get the CSV using the code below:
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
CookieContainer cookieContainer = new CookieContainer();
Uri baseUri = new Uri("https://www.digikey.com");
using (HttpClientHandler handler = new HttpClientHandler() { CookieContainer = cookieContainer })
using (HttpClient client = new HttpClient(handler) { BaseAddress = baseUri})
{
//The User-Agent is required (what values work would need to be tested)
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0");
//Make our initial junk request that will fail but get the cookie
HttpResponseMessage getCookiesResponse = await client.GetAsync("/product-search/download.csv");
//Check if we actually got cookies
if (cookieContainer.GetCookies(baseUri).Count > 0)
{
//Try getting the data
HttpResponseMessage dataResponse = await client.GetAsync("product-search/download.csv?FV=ffe00035&quantity=0&ColumnSort=0&page=4&pageSize=500");
if(dataResponse.StatusCode == HttpStatusCode.OK)
{
Console.Write(await dataResponse.Content.ReadAsStringAsync());
}
}
else
{
throw new Exception("Failed to get cookies!");
}
}
Notes
Even with the right cookie if you don't send a User-Agent header the server will return a 403. I'm not sure what the server expects in terms of a user agent, I just copied the value my browser sends.
In the check to see if cookies have been set it would be a good idea to verify you actually have the 'i10c.bdddb' cookie instead of just checking if there are any cookies.
This is just a quick bit of sample code so it's not the cleanest. You may want to look into FormUrlEncodedContent to send the page number and other parameters.
I tested with your URL and I was able to reproduce your error. Any requests that I try with the querystring parameter quantity=0 seems to fail with a HTTP Error 403.
I would suggest requesting a quantity greater than zero.
A HTTP 403 status code mean forbidden, so there is a problem with your credentials. It doesn't seem to be like you're sending any. If you add them into your header this should work fine like this:
client.Headers.Add("Authorization", "token");
or sending them like this:
client.UseDefaultCredentials = true;
client.Credentials = new NetworkCredential("username", "password");
Most likely the links are working through web browsers is because you have already authenticated and the browser is sending the credentials/token.
I have this issue with Digi-key too.
The solution for me is to turn off my VPN service.
I'm writing an interface to scrape info from a service. The link is behind a login, so I keep a copy of the cookies and then attempt to loop through the pages to get stats for our users.
The urls to hit are of the format: https://domain.com/groups/members/1234
for the first page, and each subsequent page appends ?page=X
string vUrl = "https://domain.com/groups/members/1234";
if (pageNumber > 1) vUrl += "?page=" + (pageNumber).ToString();
HttpWebRequest groupsRequest = (HttpWebRequest)WebRequest.Create(vUrl);
groupsRequest.CookieContainer = new CookieContainer();
groupsRequest.CookieContainer.Add(cookies); //recover cookies First request
groupsRequest.Method = WebRequestMethods.Http.Get;
groupsRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36";
groupsRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
groupsRequest.UseDefaultCredentials = true;
groupsRequest.AutomaticDecompression = DecompressionMethods.GZip;
groupsRequest.Headers.Add("Accept-Language", "en-US,en;q=0.8");
groupsRequest.Headers.Add("Cache-Control", "max-age=0");
HttpWebResponse getResponse = (HttpWebResponse)groupsRequest.GetResponse();
This works fine for the first page and I get the data back that I need, but with each subsequent pass, the queryString is ignored. Debugging at the last line shows that RequestUri.Query for the request is correct, but the response RequestUri.Query is blank. So it has the effect of always returning page 1 data.
I've tried to mimic the request headers that I see via Inspect in Chrome, but I'm stuck. Help?
when you put that url that is failing into a browser does it work? Because it is a GET, the browser should make the same request and tell you if it is working. If it does not work in the browser, then perhaps you are missing something aside from the query string?
If it does work, then maybe use fiddler and find out exactly what headers, cookies, and query string values are being sent to make 100% sure that you are sending the correct request. It could be that the query string is not enough information to get the data that you need from the request that you are sending.
If you still can't get it then fiddler the request when you send it through the browser and then use this fiddler extension to turn the request into code and see whats up.
I'm trying to check the redirect location of a url but am always getting the wrong results. For example, for the url http://www.yellowpages.com.eg/Mjg3NF9VUkxfMTEwX2h0dHA6Ly93d3cubG90dXMtYWlyLmNvbV8=/Lotus-Air/profile.html, it redirects to http://www.lotus-air.com with a type of redirect 302 Found (you can test it on the this service http://www.internetofficer.com/seo-tool/redirect-check/), however am getting "http://mobile.yellowpages.com.eg/" as the webResp.GetResponseHeader("Location") . My Code is as follows:
string url = #"http://www.yellowpages.com.eg/Mjg3NF9VUkxfMTEwX2h0dHA6Ly93d3cubG90dXMtYWlyLmNvbV8=/Lotus-Air/profile.html";
HttpWebRequest webReq = WebRequest.Create(url) as HttpWebRequest;
webReq.Method = "HEAD";
webReq.AllowAutoRedirect = false;
HttpWebResponse webResp = webReq.GetResponse() as HttpWebResponse;
txtOutput.Text += webResp.StatusCode.ToString() + "\r\n" ;
txtOutput.Text += webResp.GetResponseHeader("Location") + "\r\n";
txtOutput.Text += webResp.ResponseUri.ToString();
webResp.Close();
Thanks.
Yehia
They are probably sending different redirects based on the user agent, so you get one result in a browser and another in your code.
You could use a HTTP debugging proxy to get an understanding of the headers moving back and forth and enables to you to change your user-agent to help test Ben's theory (I +1'd that).
A good one is Fiddler - Web Debugging Proxy free and easy to use/
The screenshot below shows me changing the useragent to an old IEMobile one "Mozilla/4.0 (compatible; MSIE 6.0; Windows CE; IEMobile 6.12; en-US; KIN.Two 1.0)", which redirects me to mobile.yellowpages.com.eg
n.b. changing to an ipad useragent takes you to iphone.yellowpages.com.eg
As Ben pointed out, it redirects based on user agent. Just add some user agent (this one is for chrome):
webReq.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13";
For me it redirects to http://www.lotus-air.com.
When i trying to load html from server by https, it returning an error code 500: but when i open same link in browser it works fine: is there any way to do this? I'm using Webclient and also sending a useragent information to the server:
HttpWebRequest req1 = (HttpWebRequest)WebRequest.Create("mobile.unibet.com/";);
req1.UserAgent = #"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5";
var response1 = req1.GetResponse();
var responsestream1 = response1.GetResponseStream();
David is correct, this generally happens when the server is expecting some headers that is not passed through, in your case Accept
this code works now
string requestUrl = "https://mobile.unibet.com/unibet_index.t";
var request = (HttpWebRequest)WebRequest.Create(requestUrl);
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.UserAgent = "//Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
using (var response = request.GetResponse() as HttpWebResponse)
{
using (var sr = new StreamReader(response.GetResponseStream()))
{
var responsestring = sr.ReadToEnd();
if (!string.IsNullOrEmpty(responsestring))
{
Console.WriteLine(responsestring);
}
}
}
This should probably be a comment but there's not enough room in the comment for all the questions... I don't think the question has enough information to answer with any level of confidence.
A 500 error means a problem at the server. The short answer is that the browser is sending some content that the WebClient is not.
The WebClient may not be sending headers that are expected by the server. Does the server require authentication? Is this a page on a company that you've contracted with that perhaps provided you with credentials or an API key that was Do you need to add HTTP Authorization?
If this is something you're doing with a company that you've got a partnership with, you should be able to ask them to help trace why you're getting a 500 error. Otherwise, you may need to provide us with a code sample and more details so we can offer more suggestions.
I need to access to service from windows-client? that can be called by ajax - GET request. and returns XML
if i using HttpWebRequest request = HttpWebRequest.Create...
for ex url: http://site.com/UtilBillAjaxServlet?event=GET_PAMENT_CENT_DUE&SERVICEPROIDER=providername&SERVICETYPE=BROADBAND&CONSUMERNUMBER=195100601
And it return's 0-length response (in browser it retun correct response)
i think problem is - server detects that query as non-xhttp query (is there any difference?)
Thank you.
You should use fiddler or any other sniffer for tracing that.
But for doing what you want just use the following:
http://support.microsoft.com/default.aspx/kb/307023
It's possible that the service only responds to requests coming from a browser; I'd find that a little strange, but not unheard of.
However, if that is the case you can emulate a browser request:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(yourUri);
// Pretend to be IE6!
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; windows NT 5.1)";
request.Method = "GET";
request.AllowAutoRedirect = true;
request.KeepAlive = true;