C# WebClient unable to download csv file from link - c#

Here is my code tries to download the csv file from mentioned url but getting and error.
string remoteUri = "https://www.nseindia.com/api/corporates-corporateActions?index=equities&from_date=30-07-2020&to_date=06-08-2020&csv=true";
string fileName = #"C:\test.csv";
WebClient myWebClient = new WebClient();
myWebClient.DownloadFile(remoteUri,fileName);
Getting an error Fatal Error: Execution time limit was exceeded.
But hitting the above url in browser downloads the csv file

This is because the HTTP server is expecting the following headers in your request:
Accept-Language: fr,fr-FR;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
You can try them in another REST client. I've been able to reproduce your problem with ARC.

There are some issues in that URL from where you are trying to get CSV file.I checked Your code with other CSV URL Its working fine. Bellow is the test CSV URL.
string remoteUri = "http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv";
string fileName = #"Y:\Users Data\Sagar\Test\test.csv";
WebClient myWebClient = new WebClient();
myWebClient.Headers.Add("contentType", "text/csv");
await myWebClient.DownloadFileTaskAsync(remoteUri, fileName);

Related

404 error when trying to upload crash to hockeyapp

I'm trying to upload crash manually to HockeyApp using public API. When calling the api link using Postman and uploading crash.log file it works fine but when I try to do the same from C# code I get 404 error.
Here is my code:
string log = ""; //log content
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("*/*"));
var content = new MultipartFormDataContent();
var stringContent = new StringContent(log);
stringContent.Headers.ContentType = System.Net.Http.Headers.MediaTypeHeaderValue.Parse("text/plain");
content.Add(stringContent, "log", "crash.log");
var response = await this.client.PostAsync("https://rink.hockeyapp.net/api/2/apps/[APP_ID]/crashes/upload", content);
}
I was using WireShark to analyse the request that Postman is sending and tried to make mine look exactly the same. The only difference I see is that request from C# code has filename* field in Content-Disposition for the attachment while the one from Postman doesn't:
Content-Disposition: form-data; name="log"; filename="crash.log"; filename*=utf-8''%22crash.log%22
It might be worth mentioning that the code is written in portable library in Xamarin project.
Following #Lukas Spieß sugestion I asked the question on HockeyApp support. Apparently they don't handle quotes in the boundary header. The one thing I missed comparing Postman request and mine.
Here is the solution:
var contentTypeString = content.Headers.ContentType.ToString().Replace("\"", "");
content.Headers.Remove("Content-Type");
content.Headers.TryAddWithoutValidation("Content-Type", contentTypeString);

Uncompressing xml feed

My application downloads a zipped xml file from the web and tries to create XML reader:
var fullReportUrl = "http://..."; // valid url here
//client below is an instance of HttpClient
var fullReportResponse = client.GetAsync(fullReportUrl).Result;
var zippedXmlStream = fullReportResponse.Content.ReadAsStreamAsync().Result;
XmlReader xmlReader = null;
using(var gZipStream = new GZipStream(zippedXmlStream, CompressionMode.Decompress))
{
try
{
xmlReader = XmlReader.Create(gZipStream, settings);
}
catch (Exception xmlEx)
{
}
}
When I try to create XML reader I get an error:
"The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.
When I use the URL in the browser I succesfully download a zip file with a well formatted XML in it. My OS is able to unzip it without any issues. I examined the first two characters of the downloaded file and they appear to be 'PK' which is consistent with a ZIP format.
I might be missing a step in stream transformations. What am I doing wrong?
You don't need to use GzipStream for decompressing any http response with HttpClient. You can use HttpClientHandler AutomaticDecompression to make HttpClient decompress the request automatically for you.
HttpClientHandler handler = new HttpClientHandler()
{
// both gzip and deflate
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
using (var client = new HttpClient(handler))
{
var fullReportResponse = client.GetAsync(fullReportUrl).Result;
}
Edit 1:
Web Servers won't gzip output all the requests. First they check accept-encoding header, if the header is set and it is something like Accept-Encoding: deflate, gzip;q=1.0, *;q=0.5 the web server understands the client could support gzip or deflate so Web Server might ( depends on the app logic or server configuration ) compress the output into gzip or deflate. In your scenario I don't think you have set accept-encoding header so the web response will return uncompressed. Although I recommend you to try the code above.
Read more about accept-encoding on MDN

Where does strange MIME types come from?

I have a web service for uploading files written in C#.
Front-end application is written in Javascript / HTML5 (using https://github.com/blueimp/jQuery-File-Upload)
Recently, I was reviewing server logs and found some strange MIME types for PDF files that where sent by client browser, for example:
application/unknown
application/force-download
application/force-download/n
application/force-download\n
[application/pdf]
Some of them are causing .NET framework throwing exception:
MultipartMemoryStreamProvider streamProvider = new MultipartMemoryStreamProvider();
await Request.Content.ReadAsMultipartAsync(streamProvider);
"Message Error parsing MIME multipart body part header byte 156 of data segment System.Byte[]."
I don't have a clue what to do with that.
Try checking the Content-Type in the request.
Content-Disposition:
form-data;
name="imagefile";
filename="C:\Users\Pictures\sid.png"
Content-Type:
(notice the blank Content-Type, it should be Content-Type: image/png )
// within WebAPI you can use code below to log the request body
string requestBody = await Request.Content.ReadAsStringAsync();

Can't Programatically Upload csv File with WebClient .Net 4.0

I am attempting to upload a csv file but I can't seem to get it to work programatically. If I use Postman in Chrome to send the file it works and here is what it sends (Fiddler output):
------WebKitFormBoundary2YsMyLR3QAPruTy4
Content-Disposition: form-data; name="Content-Type"; filename="613022.csv"
Content-Type: application/vnd.ms-excel
// File Content here
------WebKitFormBoundary2YsMyLR3QAPruTy4--
However, using this code:
WebClient wc = new WebClient();
wc.Credentials = new NetworkCredential(user, pw);
wc.Headers[HttpRequestHeader.Cookie] = "OBBasicAuth=fromDialog";
wc.Headers[HttpRequestHeader.ContentType] = "application/vnd.ms-excel";
wc.UploadFile(baseURL + service + apiVersion + resource, "post", file);
Results in (Fiddler output):
-----------------------8d101dbe85fe96c
Content-Disposition: form-data; name="file"; filename="613022.csv"
Content-Type: application/vnd.ms-excel
// File Content here
-----------------------8d101dbe85fe96c--
Which does not work and the server returns a 503 error. The only difference I see is in the Content-Disposition name. How can I set this or is there a better way to accomplish this?
Perhaps you need "upload" instead of "post" in the method parameters - just guessing.
I believe the problem with your original approach is that WebClient.UploadFile generates the multipart/form-data request in a way that is unexpected by the server, hence the 5xx error code.
After looking around for a bit, I think the answer to this question should give you a starting point to tweak the request according to your needs.

Downloading a Cab file using WebClient gives too few bytes

I need to download a Cab file from a Url into a stream.
using (WebClient client = new WebClient())
{
client.Credentials = CredentialCache.DefaultCredentials;
byte[] fileContents = client.DownloadData("http://localhost/sites/hfsc/FormServerTemplates/HfscInspectionForm.xsn");
using (MemoryStream ms = new MemoryStream(fileContents))
{
FormTemplate = formExtractor.ExtractFormTemplateComponent(ms, "template.xml");
}
}
This is fairly straight forward, however my cab extractor (CabLib) is throwing an exception that it's not a valid cabinet.
I was previously using a SharePoint call to get the byte stream and that was returning 30942 bytes. The stream I get through that method worked correctly with CabLib. The stream I get with the WebClient returns only 28087 bytes.
I have noticed that the responce header content-type is coming back as text/html; charset=utf-8
I'm not too sure why but I think it's what's affecting the data I get back.
I beleive the problem is that SharePoint is passing the xsn to the Forms Server to render as an info path form in HTML for you. You need to stop this from happening. You can do this by adding some query string parameters to the URL request.
These can be found at:
http://msdn.microsoft.com/en-us/library/ms772417.aspx
I suggest you use NoRedirect=true

Categories