Im in a middle of teaching myself to code so do pardon the ignorance.
So my question is, what do I need to read/learn in order to be able to output the HTML of a particular website (e.g google.com) to console?
Thanks.
I would suggest you start here: http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest%28v=VS.90%29.aspx
Essentially, you create the HttpWebRequest and then call the GetResponse() method. You can then read the response stream and output it to your console.
Use HttpWebRequest to create a request and output the response to the console.
using System;
using System.IO;
using System.Net;
using System.Text;
namespace Examples.System.Net
{
public class WebRequestGetExample
{
public static void Main ()
{
// Create a request for the URL.
WebRequest request = WebRequest.Create ("http://www.contoso.com/default.html");
// If required by the server, set the credentials.
request.Credentials = CredentialCache.DefaultCredentials;
// Get the response.
HttpWebResponse response = (HttpWebResponse)request.GetResponse ();
// Display the status.
Console.WriteLine (response.StatusDescription);
// Get the stream containing content returned by the server.
Stream dataStream = response.GetResponseStream ();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader (dataStream);
// Read the content.
string responseFromServer = reader.ReadToEnd ();
// Display the content.
Console.WriteLine (responseFromServer);
// Cleanup the streams and the response.
reader.Close ();
dataStream.Close ();
response.Close ();
}
}
}
Most browsers allow you to right-click and select "View Source", that's the easiest way to see the HTML.
Have a look at the WebClient class, particularly the example at the bottom of the MSDN page.
This will do the trick:
WebClient client = new WebClient();
Stream data = client.OpenRead("www.google.com");
StreamReader reader = new StreamReader(data);
string str = reader.ReadLine();
Console.WriteLine(str);
Related
I want to get a link from a TextBox and download a file from link.
But before downloading file, I want to know the size of the file in advance and create an empty file with that size. but I can't.
and another question, I want to show percentage of download progress. How can I know data is downloaded and I should update the percentage?
WebRequest request = WebRequest.Create(URL);
WebResponse response = request.GetResponse();
totalSize = request.ContentLength;//always is -1
using (FileStream f = new FileStream(savePath, FileMode.Create))
{
f.SetLength(totalSize);
}
System.IO.StreamReader reader = new
System.IO.StreamReader(response.GetResponseStream());
WebClient client = new WebClient();
client.DownloadFile (URL, savePath);
The best way would be to use the WebClient with its DownloadFile Function, which has an async callback for events like Completed or ProgressChanged.
Getting the size of the file in advance would be a step harder though.
Im trying to create a web service which gets to a URL e.g. www.domain.co.uk/prices.csv and then reads the csv file. Is this possible and how? Ideally without downloading the csv file?
You could use:
public string GetCSV(string url)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream());
string results = sr.ReadToEnd();
sr.Close();
return results;
}
And then to split it:
public static void SplitCSV()
{
List<string> splitted = new List<string>();
string fileList = getCSV("http://www.google.com");
string[] tempStr;
tempStr = fileList.Split(',');
foreach (string item in tempStr)
{
if (!string.IsNullOrWhiteSpace(item))
{
splitted.Add(item);
}
}
}
Though there are plenty of CSV parsers out there and i would advise against rolling your own. FileHelpers is a good one.
// Download the file to a specified path. Using the WebClient class we can download
// files directly from a provided url, like in this case.
System.Net.WebClient client = new WebClient();
client.DownloadFile(url, csvPath);
Where the url is your site with the csv file and the csvPath is where you want the actual file to go.
In your Web Service you could use the WebClient class to download the file, something like this ( I have not put any exception handling, not any using or Close/Dispose calls, just wanted to give the idea you can use and refine/improve... )
using System.Net;
WebClient webClient = new WebClient();
webClient.DownloadFile("http://www.domain.co.uk/prices.csv");
then you can do anything you like with it once the file content is available in the execution flow of your service.
if you have to return it to the client as return value of the web service call you can either return a DataSet or any other data structure you prefer.
Sebastien Lorion's CSV Reader has a constructor that takes a Stream.
If you decided to use this, your example would become:
void GetCSVFromRemoteUrl(string url)
{
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
using (CsvReader csvReader = new CsvReader(response.GetResponseStream(), true))
{
int fieldCount = csvReader.FieldCount;
string[] headers = csvReader.GetFieldHeaders();
while (csvReader.ReadNextRecord())
{
//Do work with CSV file data here
}
}
}
The ever popular FileHelpers also allows you to read directly from a stream.
The documentation for WebRequest has an example that uses streams. Using a stream allows you to parse the document without storing it all in memory
I am writing an app in c#,
Is there a way to download a HTML page by giving my program its URL only.
Foe example my program will get the URL www.google.com and download the HTML page?
Use WebClient.DownloadString().
Use the WebClient class.
This is extracted from a sample on the msdn doc page:
using System;
using System.Net;
using System.IO;
public static string Download (string uri)
{
WebClient client = new WebClient ();
Stream data = client.OpenRead (uri);
StreamReader reader = new StreamReader (data);
string s = reader.ReadToEnd ();
data.Close ();
reader.Close ();
return s;
}
the page at https://qrng.physik.hu-berlin.de/ provides a high bit rate quantum number generator web service and I'm trying to access that service.
However I could not manage to do so. This is my current code:
using System;
using System.Collections.Generic;
using System.Linq;
using S=System.Text;
using System.Security.Cryptography;
using System.IO;
namespace CS_Console_App
{
class Program
{
static void Main()
{
System.Net.ServicePointManager.Expect100Continue = false;
var username = "testuser";
var password = "testpass";
System.Diagnostics.Debug.WriteLine(Post("https://qrng.physik.hu-berlin.de/", "username="+username+"&password="+password));
Get("http://qrng.physik.hu-berlin.de/download/sampledata-1MB.bin");
}
public static void Get(string url)
{
var my_request = System.Net.WebRequest.Create(url);
my_request.Credentials = System.Net.CredentialCache.DefaultCredentials;
var my_response = my_request.GetResponse();
var my_response_stream = my_response.GetResponseStream();
var stream_reader = new System.IO.StreamReader(my_response_stream);
var content = stream_reader.ReadToEnd();
System.Diagnostics.Debug.WriteLine(content);
stream_reader.Close();
my_response_stream.Close();
}
public static string Post(string url, string data)
{
string vystup = null;
try
{
//Our postvars
byte[] buffer = System.Text.Encoding.ASCII.GetBytes(data);
//Initialisation, we use localhost, change if appliable
System.Net.HttpWebRequest WebReq = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(url);
//Our method is post, otherwise the buffer (postvars) would be useless
WebReq.Method = "POST";
//We use form contentType, for the postvars.
WebReq.ContentType = "application/x-www-form-urlencoded";
//The length of the buffer (postvars) is used as contentlength.
WebReq.ContentLength = buffer.Length;
//We open a stream for writing the postvars
Stream PostData = WebReq.GetRequestStream();
//Now we write, and afterwards, we close. Closing is always important!
PostData.Write(buffer, 0, buffer.Length);
PostData.Close();
//Get the response handle, we have no true response yet!
System.Net.HttpWebResponse WebResp = (System.Net.HttpWebResponse)WebReq.GetResponse();
//Let's show some information about the response
Console.WriteLine(WebResp.StatusCode);
Console.WriteLine(WebResp.Server);
//Now, we read the response (the string), and output it.
Stream Answer = WebResp.GetResponseStream();
StreamReader _Answer = new StreamReader(Answer);
vystup = _Answer.ReadToEnd();
//Congratulations, you just requested your first POST page, you
//can now start logging into most login forms, with your application
//Or other examples.
}
catch (Exception ex)
{
throw ex;
}
return vystup.Trim() + "\n";
}
}
}
I'm having 403 forbidden error when I try to do a get request on http://qrng.physik.hu-berlin.de/download/sampledata-1MB.bin.
After debugging abit, I've realised that even though I've supplied a valid username and password, the response html that was sent after my POST request indicate that I was actually not logon to the system after my POST request.
Does anyone know why is this the case, and how may I work around it to call the service?
Bump. can anyone get this to work or is the site just a scam?
The site is surely not a scam. I developed the generator and I put my scientific reputation in it. The problem is that you are trying to use the service in a way that was not intended. The sample files were really only meant to be downloaded manually for basic test purposes. Automated access to fetch data into an application was meant to be implemented through the DLLs we provide.
On the other hand, I do not know of any explicit intent to prevent your implementation to work. I suppose if a web browser can log in and fetch data, some program should be able to do the same. Maybe only the login request is just a little more complicated. No idea. The server software was developed by someone else and I cannot bother him with this right now.
Mick
Actually, the generator can now also be purchased. See here:
http://www.picoquant.com/products/pqrng150/pqrng150.htm
Have you tried to change this
my_request.Credentials = System.Net.CredentialCache.DefaultCredentials
to
my_request.Credentials = new NetworkCredential(UserName,Password);
as described on MSDN page?
In my application I use the WebClient class to download files from a Webserver by simply calling the DownloadFile method. Now I need to check whether a certain file exists prior to downloading it (or in case I just want to make sure that it exists). I've got two questions with that:
What is the best way to check whether a file exists on a server without transfering to much data across the wire? (It's quite a huge number of files I need to check)
Is there a way to get the size of a given remote file without downloading it?
Thanks in advance!
WebClient is fairly limited; if you switch to using WebRequest, then you gain the ability to send an HTTP HEAD request. When you issue the request, you should either get an error (if the file is missing), or a WebResponse with a valid ContentLength property.
Edit: Example code:
WebRequest request = WebRequest.Create(new Uri("http://www.example.com/"));
request.Method = "HEAD";
using(WebResponse response = request.GetResponse()) {
Console.WriteLine("{0} {1}", response.ContentLength, response.ContentType);
}
When you request file using the WebClient Class, the 404 Error (File Not Found) will lead to an exception. Best way is to handle that exception and use a flag which can be set to see if the file exists or not.
The example code goes as follows:
System.Net.HttpWebRequest request = null;
System.Net.HttpWebResponse response = null;
request = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create("www.example.com/somepath");
request.Timeout = 30000;
try
{
response = (System.Net.HttpWebResponse)request.GetResponse();
flag = 1;
}
catch
{
flag = -1;
}
if (flag==1)
{
Console.WriteLine("File Found!!!");
}
else
{
Console.WriteLine("File Not Found!!!");
}
You can put your code in respective if blocks.
Hope it helps!
What is the best way to check whether a file exists on a server
without transfering to much data across the wire?
You can test with WebClient.OpenRead to open the file stream without reading all the file bytes:
using (var client = new WebClient())
{
Stream stream = client.OpenRead(url);
// ^ throws System.Net.WebException: 'Could not find file...' if file is not present
stream.Close();
}
This will indicate if the file exists at the remote location or not.
To fully read the file stream, you would do:
using (var client = new WebClient())
{
Stream stream = client.OpenRead(url);
StreamReader sr = new StreamReader(stream);
Console.WriteLine(sr.ReadToEnd());
stream.Close();
}
In case anyone stuck with ssl certificate issue
ServicePointManager.ServerCertificateValidationCallback = new RemoteCertificateValidationCallback
(
delegate { return true; }
);
WebRequest request = WebRequest.Create(new Uri("http://.com/flower.zip"));
request.Method = "HEAD";
using (WebResponse response = request.GetResponse())
{
Console.WriteLine("{0} {1}", response.ContentLength, response.ContentType);
}