Im trying to download the source code for this webpage for a school project using c#.
this is the page im trying to get:
http://www.epicurious.com/tools/fooddictionary/entry?id=1650
I have tried code such as
HttpWebRequest request = (HttpWebRequest)
WebRequest.Create("http://www.epicurious.com/tools/fooddictionary/entry?id=1650");
and by using
WebClient client = new WebClient();
string value = client.DownloadString("http://www.epicurious.com/tools/fooddictionary/entry?id=1650");
and neither methods are getting me the page source. Any help would be appreciated.
using HttpWebRequest.Create
try
{
WebRequest req = HttpWebRequest.Create("http://www.epicurious.com/tools/fooddictionary/entry?id=1650");
req.Method = "GET";
string source;
using (StreamReader reader = new StreamReader(req.GetResponse().GetResponseStream()))
{
source = reader.ReadToEnd();
}
}
catch (Exception ex)
{
//Log the exception
MessageBox.Show(ex.ToString());
}
Using DownloadString
WebClient client = new WebClient();
string reply = client.DownloadString("http://www.epicurious.com/tools/fooddictionary/entry?id=1650");
Above methods are working fine for me. Check the exception if any..
How about this...
string sourceCode = "";
Uri site = new Uri("http://www.epicurious.com/tools/fooddictionary/entry?id=1650");
WebRequest request = WebRequest.Create(site);
using(StreamReader reader = new StreamReader(request.GetResponse().GetResponseStream(), Encoding.ASCII)){
sourceCode = reader.ReadToEnd();
}
The "using" statement will close your streams for you. NOTE: Calling close on a stream will also call class on any streams it is using, this is why you only need the single using statement
Related
I'm quite new to c# and I'm having an issue when trying to call an API to post some CSV data using the following code:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://myurl");
request.Method = "POST";
request.Headers.Add("X-Api-Key:" + apiKey);
request.Headers.Add("X-Api-Secret:" + apiSecret);
request.ContentType = "text/csv";
using (StreamWriter swCSVData = new StreamWriter(request.GetRequestStream()))
{
StreamReader csvReader = new StreamReader(File.OpenRead("C:\\Test\\conf-ato-sprmbrinfo-batch-001.csv"));
while (!csvReader.EndOfStream)
{
swCSVData.WriteLine(csvReader.ReadLine());
}
csvReader.Close();
}
HttpWebResponse response = null;
try
{
Console.WriteLine("Attempting response....");
response = (HttpWebResponse)request.GetResponse();
Console.WriteLine("Got response....");
using (Stream responseStream = response.GetResponseStream())
{
if (responseStream != null)
{
using (StreamReader reader = new StreamReader(responseStream))
{
strResponseValue = reader.ReadToEnd();
}
}
}
Console.WriteLine(strResponseValue);
File.WriteAllText("C:\\Test\\apioutput.txt",strResponseValue);
}
catch ( Exception ex)
{
strResponseValue = "{\"errorMessages\":[\"" + ex.Message.ToString() + "\"],\"errors\":{}}";
}
If I run this code using the post test at ptsv2.com, then I can see that the api call looks like I would expect and I get a response. However, when I change my code to point to my required URL it seems to just hang on the following line:
response = (HttpWebResponse)request.GetResponse();
Does anyone have any idea as to why this could be happening?
Also happy for any recommendations to make my code better.
Thanks in advance.
It turns out this was a security issue once I could see the exception being returned. I had to add the following line of code for my request:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
Thanks to the people who took the time to comment on my original question.
So I'm following the WingTip Toy tutorial, and I know its sort of old but it was error free until I got to the point where I needed to checkout with PayPal using the sandbox developing tool
This is the code where the error occurs
//Retrieve the Response returned from the NVP API call to PayPal.
HttpWebResponse objResponse = (HttpWebResponse)objRequest.GetResponse();
string result;
using (StreamReader sr = new StreamReader(objResponse.GetResponseStream()))
{
result = sr.ReadToEnd();
}
return result;
And this is the error im getting when i run it
[ProtocolViolationException: You must write ContentLength bytes to the request stream before calling [Begin]GetResponse.]
Please note I'm a beginner
Edit: The full code is here
public string HttpCall(string NvpRequest)
{
string url = pEndPointURL;
string strPost = NvpRequest + "&" + buildCredentialsNVPString();
strPost = strPost + "&BUTTONSOURCE=" + HttpUtility.UrlEncode(BNCode);
HttpWebRequest objRequest = (HttpWebRequest)WebRequest.Create(url);
objRequest.Timeout = Timeout;
objRequest.Method = "POST";
objRequest.ContentLength = strPost.Length;
try
{
using (StreamWriter myWriter = new StreamWriter(objRequest.GetRequestStream()))
{
myWriter.Write(strPost);
}
}
catch (Exception)
{
// No logging for this tutorial.
}
//Retrieve the Response returned from the NVP API call to PayPal.
HttpWebResponse objResponse = (HttpWebResponse)objRequest.GetResponse();
string result;
using (StreamReader sr = new StreamReader(objResponse.GetResponseStream()))
{
result = sr.ReadToEnd();
}
return result;
}
objRequest.ContentLength = strPost.Length;
What are you trying to do here? The framework sets the content length automatically. The content length is in bytes but you have given the number of characters.
That's why the error complains: You have written a different number of characters than you said you would write.
Delete that line.
Your code would become a lot simpler if you used HttpClient. Should be about 5 lines.
i'm having a problem with this simple code which sends a request using provided url and reads html from responce. Looks like it's something with encoding of cyrillic symbols after ?q=, but i can't see why (url was actually obtained from browser address bar, not generated or anything else).
url =
"http://www.avito.ru/nizhniy_novgorod/kvartiry/sdam/na_dlitelnyy_srok?q=%D0%93%D0%B5%D0%BD%D0%B5%D1%80%D0%B0%D0%BB%D0%B0+%D0%98%D0%B2%D0%BB%D0%B8%D0%B5%D0%B2%D0%B0+10%D0%BA1";
string html = "";
try
{
Uri uri = new Uri(url);
WebRequest request = WebRequest.Create(uri);
request.Timeout = 100000;
using (WebResponse responce = request.GetResponse())
{
Stream stream = responce.GetResponseStream();
StreamReader reader = new StreamReader(stream);
html = reader.ReadToEnd();
}
}
catch (Exception e)
{
System.Console.WriteLine(e.Message);
}
Error occures in GetResponce() method. The message is:
The request was aborted: The connection was closed unexpectedly.
You should cast your request and response to HttpWebRequest and HttpWebResponse, respectively.
var url = "http://www.avito.ru/nizhniy_novgorod/kvartiry/sdam/na_dlitelnyy_srok?q=%D0%93%D0%B5%D0%BD%D0%B5%D1%80%D0%B0%D0%BB%D0%B0+%D0%98%D0%B2%D0%BB%D0%B8%D0%B5%D0%B2%D0%B0+10%D0%BA1";
string html = "";
try
{
Uri uri = new Uri(uri);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Timeout = 100000;
using (HttpWebResponse responce = (HttpWebResponse)request.GetResponse())
{
Stream stream = responce.GetResponseStream();
StreamReader reader = new StreamReader(stream);
html = reader.ReadToEnd();
Console.WriteLine(html);
}
}
catch (Exception e)
{
System.Console.WriteLine(e.Message);
}
Also, It seems that the url http://www.avito.ru/nizhniy_novgorod/kvartiry/sdam/na_dlitelnyy_srok?q=%D0%93%D0%B5%D0%BD%D0%B5%D1%80%D0%B0%D0%BB%D0%B0+%D0%98%D0%B2%D0%BB%D0%B8%D0%B5%D0%B2%D0%B0+10%D0%BA1 is invalid.
Using Fiddler, the url returns a 404 error.
When I run this code to call the Google API, all I get is a 'Bad Request' error, but I don't know where I'm going wrong.
The code is being returned no problem from the authorization page on Google, it's when the code gets to the part below that it fails. Please could someone tell me where I'm going wrong here?
I'm aware that there are libraries for this, but I'm trying to understand how to do this the RESTful way as a learning exercise.
Thanks
var code = Request.QueryString["code"];
var accessToken = string.Empty;
var req0 = WebRequest.Create("https://accounts.google.com/o/oauth2/token");
req0.Method = "POST";
string postData = string.Format("code={0}&client_id={1}&client_secret={2}&redirect_uri= {3}&grant_type=authorization_code",
code, //the code i got back
"xxxx.apps.googleusercontent.com",
"xxx",
Url.Encode("http://localhost/home/callback")
); //my return URI
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
req0.ContentType = "application/x-www-form-urlencoded";
req0.ContentLength = byteArray.Length;
using (Stream dataStream = req0.GetRequestStream())
{
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();
}
try
{
using (WebResponse response = req0.GetResponse())
{
using (var dataStream = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(dataStream))
{
string responseFromServer = reader.ReadToEnd();
var ser = new JavaScriptSerializer();
accessToken = ser.DeserializeObject(responseFromServer).ToString();
}
}
}
}
catch (WebException wex)
{
Debug.WriteLine(wex.ToString());
}
catch (Exception ex)
{
Debug.WriteLine(ex.ToString());
}
The Google API is very finicky when it comes to what you have entered into the API Console for your client id return URI, and what you enter in your code to call it.
I was missing the trailing forward slash. That was all. Lesson learnt...
Thanks to Jon and Sandeep for help point me in the right direction by comparing what was in the network traffic, and what I should have been sending.
I try download file from server with FileWebRequest. But I get error:
Method on download is here:
public string HttpFileGetReq(Uri uri, int reqTimeout, Encoding encoding)
{
try
{
string stringResponse;
var req = (FileWebRequest)WebRequest.Create(uri);
req.Timeout = reqTimeout;
req.Method = WebRequestMethods.File.DownloadFile;
var res = (FileWebResponse)req.GetResponse();
//using (var receiveStream = res.GetResponseStream())
//using (var readStream = new StreamReader(receiveStream,encoding))
//{
// stringResponse = readStream.ReadToEnd();
//}
return stringResponse="0K";
}
catch (WebException webException)
{
throw webException;
}
}
Usage is here:
public dynamic LoadRoomMsg(IAccount account, string roomId)
{
try
{
string uri = string.Format("http://www-pokec.azet.sk/_s/chat/nacitajPrispevky.php?{0}&lok={1}&lastMsg=0&pub=0&prv=0&r=1295633087203&changeroom=1" , account.SessionId, roomId);
var htmlStringResult = HttpFileGetReq(new Uri(uri), ReqTimeout, EncodingType);
//var htmlStringResult = _httpReq.HttpGetReq(new Uri(string.Format("{0}{1}?{2}&lok=", PokecUrl.RoomMsg,account.SessionId,roomId)),
// ReqTimeout, account.Cookies, EncodingType);
if (!string.IsNullOrEmpty(htmlStringResult))
{
return true;
}
return false;
}
catch (Exception exception)
{
throw exception;
}
}
URL on file is here.
I would like read this file to string variable, that’s all. If anyone have some time and can help me I would be very glad to him.
Your URL (http://...) will produce a HttpWebRequest. You can check with the debugger.
Form MSDN:
The FileWebRequest class implements
the WebRequest abstract base class for
Uniform Resource Identifiers (URIs)
that use the file:// scheme to request
local files.
Note the file:// and local files in there.
Tip: Just use the WebClient class.
Rather than implement your own web streams allow the .NET framework to do it all for you with WebClient, for example:
string uri = string.Format(
"http://www-pokec.azet.sk/_s/chat/nacitajPrispevky.php?{0}&lok={1}&lastMsg=0&pub=0&prv=0&r=1295633087203&changeroom=1",
account.SessionId,
roomId);
System.Net.WebClient wc = new System.Net.WebClient();
string webData = wc.DownloadString(uri);
...parse the webdata response here...
Looking at the response from the URL you posted:
{"reason":0}
parsing that should be a simple task with a little string manipulation.
Change FileWebRequest and FileWebResponse to HttpWebRequest and HttpWebResponse.
It doesn't matter that what you're downloading may be a file; as far as the .NET Framework is concerned, you're just retrieving a page from a website.
FileWebRequest is for file:// protocols. Since you're using an http:// url, you want to use HttpWebRequest.
public string HttpFileGetReq(Uri uri, int reqTimeout, Encoding encoding)
{
string stringResponse;
var req = (HttpWebRequest)WebRequest.Create(uri);
req.Timeout = reqTimeout;
var res = (HttpWebResponse)req.GetResponse();
using (var receiveStream = res.GetResponseStream())
{
using (var readStream = new StreamReader(receiveStream,encoding))
{
return readStream.ReadToEnd();
}
}
}