We Keep Coding

C# downloading a csv file from https site

I am trying to download a file into my app. From what I can see if I put this link into a browser I get a good file downloaded so I know the file is available for me to get. I have other files to download from different sites and they all work well but this one will not download for me in a usable manner. I guess I do not understand something, do you know the key fact I am missing?
After many attempts I have now coded the following
private string url =
"https://coronavirus.data.gov.uk/api/v1/data?filters=areaType=nation&structure={\"Name\":\"areaName\",\"date\":\"date\",\"FirstDose\":\"cumPeopleVaccinatedFirstDoseByPublishDate\",\"SecondDose\":\"cumPeopleVaccinatedSecondDoseByPublishDate\"}&format=csv";
private void btn_wc1_Click(object sender, EventArgs e)
{
WebClient wc = new WebClient();
wc.Encoding = Encoding.UTF8;
wc.DownloadFile(url, "wc1_uk_data.csv");
}
private void btn_wc2_Click(object sender, EventArgs e)
{
using (var webClient = new WebClient())
{
webClient.Encoding = Encoding.UTF8;
string s = webClient.DownloadString(url);
File.WriteAllText(#"wc2_uk_data.csv", s);
}
}
private async void btn_https_Click(object sender, EventArgs e)
{
HttpClient _client = new HttpClient();
byte[] buffer = null;
try
{
HttpResponseMessage task = await _client.GetAsync(url);
Stream task2 = await task.Content.ReadAsStreamAsync();
using (MemoryStream ms = new MemoryStream())
{
await task2.CopyToAsync(ms);
buffer = ms.ToArray();
}
File.WriteAllBytes("https_uk_data.csv", buffer);
}
catch
{
}
}
private void btn_wc3_Click(object sender, EventArgs e)
{
using (var webClient = new WebClient())
{
webClient.Encoding = Encoding.UTF8;
byte[] myDataBuffer = webClient.DownloadData(url);
MemoryStream ms = new MemoryStream(myDataBuffer);
FileStream f = new FileStream(Path.GetFullPath(Path.Combine(Application.StartupPath, "wc3_uk_data.csv")),
FileMode.OpenOrCreate);
ms.WriteTo(f);
f.Close();
ms.Close();
}
}
Using the following UI
All the different functions above will download a file but none of the downloaded files is usable. It seems like it is not the file but maybe something to do with information regarding the file. As my app does not know what to do with this I never reply with what ever the other end wants. I guess if I replied the next set of data I got would be the file.
If I put the URL into a browser then I get a file that is good. This link is good at the moment.
https://coronavirus.data.gov.uk/api/v1/data?filters=areaType=nation&structure={"Name":"areaName","date":"date","FirstDose":"cumPeopleVaccinatedFirstDoseByPublishDate","SecondDose":"cumPeopleVaccinatedSecondDoseByPublishDate"}&format=csv
Anyone got any idea on what I need to do in my app to get the file like the browser does?

You need to set the WebClient.Encoding before calling DownloadString
string url = "https://coronavirus.data.gov.uk/api/v1/data?filters=areaType=nation&structure={\"Name\":\"areaName\",\"date\":\"date\",\"FirstDose\":\"newPeopleReceivingFirstDose\",\"SecondDose\":\"newPeopleReceivingSecondDose\"}&format=csv";
using (var webClient = new WebClient())
{
webClient.Encoding = Encoding.UTF8;
string s = webClient.DownloadString(url);
}
Here is a related question:
WebClient.DownloadString results in mangled characters due to encoding issues, but the browser is OK

XML - help with RSS UTF-8 support

I used this solution to read and parse a RSS feed from an ASP.NET website. This worked perfectly. However, when trying it on another site, an error occurs because "System does not support 'utf8' encoding." Below I have included an extract of my code.
private void Form1_Load(object sender, EventArgs e)
{
lblFeed.Text = ProcessRSS("http://buypoe.com/external.php?type=RSS2", "ScottGq");
}
public static string ProcessRSS(string rssURL, string feed)
{
WebRequest request = WebRequest.Create(rssURL);
WebResponse response = request.GetResponse();
StringBuilder sb = new StringBuilder("");
Stream rssStream = response.GetResponseStream();
XmlDocument rssDoc = new XmlDocument();
rssDoc.Load(rssStream);
XmlNodeList rssItems = rssDoc.SelectNodes("rss/channel/item");
string title = "";
string link = "";
...
The error occurs at "rssDoc.Load(rssStream);". Any help in encoding the xml correctly would be appreciated.

use the following code for encoding
System.IO.StreamReader stream = new System.IO.StreamReader
(response.GetResponseStream(), System.Text.Encoding.GetEncoding("utf-8"));

Uploading a text box into a text file on an ftp C#

I am trying to stream a multiline textbox into a text file on an ftp server. Can someone tell me where I may be going wrong?
private void btnSave_Click(object sender, EventArgs e)
{
UriBuilder b = new UriBuilder();
b.Host = "ftp.myserver.com";
b.UserName = "user";
b.Password = "pass";
b.Port = 21;
b.Path = "/myserver.com/directories/" + selected + ".txt";
b.Scheme = Uri.UriSchemeFtp;
Uri g = b.Uri;
System.Net.FtpWebRequest c = (System.Net.FtpWebRequest)System.Net.FtpWebRequest.Create(g);
c.Method = System.Net.WebRequestMethods.Ftp.DownloadFile;
System.Net.FtpWebResponse d = (System.Net.FtpWebResponse)c.GetResponse();
System.IO.Stream h = d.GetResponseStream;
System.IO.StreamWriter SW = new System.IO.StreamWriter(h);
String[] contents = textBox1.Lines.ToArray();
for (int i = 0; i < contents.Length; i++)
{
SW.WriteLine(contents[i]);
}
h.Close();
SW.Close();
d.Close();
}
The error I am getting is this line:
System.IO.StreamWriter SW = new System.IO.StreamWriter(h);
Stream was not writable.
Any ideas?

The response stream from an FTP site is data from the site to you. You'd need the request stream... but then you wouldn't want a method of DownloadFile - you're not downloading, you're uploading, so you want the UploadFile method.
Additionally:
You're not closing anything if exceptions are thrown: use using blocks for this.
It's a bad idea to do network access like this on the UI thread; the UI thread will block (so the whole UI will hang) while the FTP request is happening. Use a background thread instead.

To upload a file you need to use the FtpWebRequest class.
Quote:
When using an FtpWebRequest object to
upload a file to a server, you must
write the file content to the request
stream obtained by calling the
GetRequestStream method or its
asynchronous counterparts, the
BeginGetRequestStream and
EndGetRequestStream methods. You must
write to the stream and close the
stream before sending the request.
For an example of uploading a file (which you can change to writing stream content as in your example) see here.

Taken from MSDN and slightly modified:
public static bool UploadFileOnServer(string fileName, Uri serverUri)
{
// The URI described by serverUri should use the ftp:// scheme.
// It contains the name of the file on the server.
// Example: ftp://contoso.com/someFile.txt.
// The fileName parameter identifies the file
// to be uploaded to the server.
if (serverUri.Scheme != Uri.UriSchemeFtp)
{
return false;
}
// Get the object used to communicate with the server.
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(serverUri);
request.Method = WebRequestMethods.Ftp.UploadFile;
StreamReader sourceStream = new StreamReader(fileName);
byte [] fileContents = Encoding.UTF8.GetBytes(sourceStream.ReadToEnd());
sourceStream.Close();
request.ContentLength = fileContents.Length;
// This example assumes the FTP site uses anonymous logon.
request.Credentials = new NetworkCredential ("anonymous","janeDoe#contoso.com");
Stream requestStream = request.GetRequestStream();
requestStream.Write(fileContents, 0, fileContents.Length);
requestStream.Close();
FtpWebResponse response = (FtpWebResponse) request.GetResponse();
Console.WriteLine("Upload status: {0}",response.StatusDescription);
response.Close();
return true;
}

How to submit http form using C#

I have a simple html file such as
<form action="http://www.someurl.com/page.php" method="POST">
<input type="text" name="test"><br/>
<input type="submit" name="submit">
</form>
Edit: I may not have been clear enough with the question
I want to write C# code which submits this form in the exact same manner that would occur had I pasted the above html into a file, opened it with IE and submitted it with the browser.

Here is a sample script that I recently used in a Gateway POST transaction that receives a GET response. Are you using this in a custom C# form? Whatever your purpose, just replace the String fields (username, password, etc.) with the parameters from your form.
private String readHtmlPage(string url)
{
//setup some variables
String username = "demo";
String password = "password";
String firstname = "John";
String lastname = "Smith";
//setup some variables end
String result = "";
String strPost = "username="+username+"&password="+password+"&firstname="+firstname+"&lastname="+lastname;
StreamWriter myWriter = null;
HttpWebRequest objRequest = (HttpWebRequest)WebRequest.Create(url);
objRequest.Method = "POST";
objRequest.ContentLength = strPost.Length;
objRequest.ContentType = "application/x-www-form-urlencoded";
try
{
myWriter = new StreamWriter(objRequest.GetRequestStream());
myWriter.Write(strPost);
}
catch (Exception e)
{
return e.Message;
}
finally {
myWriter.Close();
}
HttpWebResponse objResponse = (HttpWebResponse)objRequest.GetResponse();
using (StreamReader sr =
new StreamReader(objResponse.GetResponseStream()) )
{
result = sr.ReadToEnd();
// Close and clean up the StreamReader
sr.Close();
}
return result;
}

Your HTML file is not going to interact with C# directly, but you can write some C# to behave as if it were the HTML file.
For example: there is a class called System.Net.WebClient with simple methods:
using System.Net;
using System.Collections.Specialized;
...
using(WebClient client = new WebClient()) {
NameValueCollection vals = new NameValueCollection();
vals.Add("test", "test string");
client.UploadValues("http://www.someurl.com/page.php", vals);
}
For more documentation and features, refer to the MSDN page.

You can use the HttpWebRequest class to do so.
Example here:
using System;
using System.Net;
using System.Text;
using System.IO;
public class Test
{
// Specify the URL to receive the request.
public static void Main (string[] args)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create (args[0]);
// Set some reasonable limits on resources used by this request
request.MaximumAutomaticRedirections = 4;
request.MaximumResponseHeadersLength = 4;
// Set credentials to use for this request.
request.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse response = (HttpWebResponse)request.GetResponse ();
Console.WriteLine ("Content length is {0}", response.ContentLength);
Console.WriteLine ("Content type is {0}", response.ContentType);
// Get the stream associated with the response.
Stream receiveStream = response.GetResponseStream ();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader (receiveStream, Encoding.UTF8);
Console.WriteLine ("Response stream received.");
Console.WriteLine (readStream.ReadToEnd ());
response.Close ();
readStream.Close ();
}
}
/*
The output from this example will vary depending on the value passed into Main
but will be similar to the following:
Content length is 1542
Content type is text/html; charset=utf-8
Response stream received.
<html>
...
</html>
*/

Response.Write("<script> try {this.submit();} catch(e){} </script>");

I needed to have a button handler that created a form post to another application within the client's browser. I landed on this question but didn't see an answer that suited my scenario. This is what I came up with:
protected void Button1_Click(object sender, EventArgs e)
{
var formPostText = #"<html><body><div>
<form method=""POST"" action=""OtherLogin.aspx"" name=""frm2Post"">
<input type=""hidden"" name=""field1"" value=""" + TextBox1.Text + #""" />
<input type=""hidden"" name=""field2"" value=""" + TextBox2.Text + #""" />
</form></div><script type=""text/javascript"">document.frm2Post.submit();</script></body></html>
";
Response.Write(formPostText);
}

I had a similar issue in MVC (which lead me to this problem).
I am receiving a FORM as a string response from a WebClient.UploadValues() request, which I then have to submit - so I can't use a second WebClient or HttpWebRequest. This request returned the string.
using (WebClient client = new WebClient())
{
byte[] response = client.UploadValues(urlToCall, "POST", new NameValueCollection()
{
{ "test", "value123" }
});
result = System.Text.Encoding.UTF8.GetString(response);
}
My solution, which could be used to solve the OP, is to append a Javascript auto submit to the end of the code, and then using #Html.Raw() to render it on a Razor page.
result += "<script>self.document.forms[0].submit()</script>";
someModel.rawHTML = result;
return View(someModel);
Razor Code:
#model SomeModel
#{
Layout = null;
}
#Html.Raw(#Model.rawHTML)
I hope this can help anyone who finds themselves in the same situation.

How to retrieve a webpage with C#?

How to retrieve a webpage and diplay the html to the console with C# ?

Use the System.Net.WebClient class.
System.Console.WriteLine(new System.Net.WebClient().DownloadString(url));

I have knocked up an example:
WebRequest r = WebRequest.Create("http://www.msn.com");
WebResponse resp = r.GetResponse();
using (StreamReader sr = new StreamReader(resp.GetResponseStream()))
{
Console.WriteLine(sr.ReadToEnd());
}
Console.ReadKey();
Here is another option, using the WebClient this time and do it asynchronously:
static void Main(string[] args)
{
System.Net.WebClient c = new WebClient();
c.DownloadDataCompleted +=
new DownloadDataCompletedEventHandler(c_DownloadDataCompleted);
c.DownloadDataAsync(new Uri("http://www.msn.com"));
Console.ReadKey();
}
static void c_DownloadDataCompleted(object sender,
DownloadDataCompletedEventArgs e)
{
Console.WriteLine(Encoding.ASCII.GetString(e.Result));
}
The second option is handy as it will not block the UI Thread, giving a better experience.

// Save HTML code to a local file.
WebClient client = new WebClient ();
client.DownloadFile("http://yoursite.com/page.html", #"C:\htmlFile.html");
// Without saving it.
string htmlCode = client.DownloadString("http://yoursite.com/page.html");