We Keep Coding

C# WebClient unable to download csv file from link

Here is my code tries to download the csv file from mentioned url but getting and error.
string remoteUri = "https://www.nseindia.com/api/corporates-corporateActions?index=equities&from_date=30-07-2020&to_date=06-08-2020&csv=true";
string fileName = #"C:\test.csv";
WebClient myWebClient = new WebClient();
myWebClient.DownloadFile(remoteUri,fileName);
Getting an error Fatal Error: Execution time limit was exceeded.
But hitting the above url in browser downloads the csv file

This is because the HTTP server is expecting the following headers in your request:
Accept-Language: fr,fr-FR;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
You can try them in another REST client. I've been able to reproduce your problem with ARC.

There are some issues in that URL from where you are trying to get CSV file.I checked Your code with other CSV URL Its working fine. Bellow is the test CSV URL.
string remoteUri = "http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv";
string fileName = #"Y:\Users Data\Sagar\Test\test.csv";
WebClient myWebClient = new WebClient();
myWebClient.Headers.Add("contentType", "text/csv");
await myWebClient.DownloadFileTaskAsync(remoteUri, fileName);

How do you set the Content-Type header for an HttpClient request with MultipartFormDataContent?

I looked at the MS Source code according to their interpretation the HttpClient itself does not have "Content-Type" only the content should have content-type. Seems logical except when you're dealing with MultipartFormDataContent.
MultipartFormDataContent completely ignores the following code:
string boundary = "--" + GenerateRandomString();
using (var content = new MultipartFormDataContent(boundary))
{
content.Headers.ContentType = new MediaTypeHeaderValue($"multipart/form-data");
content.Headers.ContentType.Parameters.Add(new NameValueHeaderValue("boundry", boundary));
...
}
No "Content-Type" is present in the request.
And also ignores:
string boundary = "--" + GenerateRandomString();
using (var content = new MultipartFormDataContent(boundary))
{
content.Headers.Remove("Content-Type");
content.Headers.TryAddWithoutValidation("Content-Type", "multipart/form-data; boundary=" + boundary);
...
}
Attemting to set it at on the HttpClient
client.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "multipart/form-data; boundary=" + boundary);
throws the following error:
System.InvalidOperationException: 'Misused header name. Make sure request headers are used with HttpRequestMessage, response headers with HttpResponseMessage, and content headers with HttpContent objects.'
I can find plenty of examples of how to do this using StringContent but none with MultipartFormDataContent. MultipartFormDataContent allows setting the Content-Type and Content-Disposition with each field, I need this more at the client level. I need a header that looks something like this:
accept: application/json
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
Authorization: Basic ZXlKbGRDSTZJakUxTWpZNU9UTXpOTnpkMjl5WkNJNklqa3haRGxtWkdKa1lUazRaVEJqWmpsalpUaGxNV1V3TXpOalxuWmpCbE1tVXhJaXdpZFhObGNpSTZJbUZrYldsdUluMD1cbjo=
Cache-Control: no-cache
Many non-Microsoft APIs require the "boundary" tag so it can distinguish the individual fields of data being sent. The validation here on the request seems a little over the top. Even TryAddWithoutValidation doesn't work (maybe a bug?). I realize that it may be possible to interpret RFC7578 in a way that says it shouldn't be required but flat out not allowing it doesn't seem right to me either. Anyone else ever run into this issue and solve it.

Initially I thought this was an HttpClient bug. I added logging to capture the request and the response. That logging was missing headers which lead me to believe that the missing "multi-part/form-data" content header was the issue and the reason the API I'm using kept telling me it couldn't find a required field. It turns out to be an issue with how the API handles the data it's sent when it's multi-part/form-data. After comparing both my HttpWebRequest and the HttpClient request in fiddler I discovered the following difference in the data being sent:
HttpWebRequest
----vekhftkcthxr
Content-Disposition: form-data; name="name";
d30-20180524
HttpClient
----bcgifxyjkmkw
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name=name
d30-20180524
I build the HttpWebRequest manually so I included the quotes and ending semi-colons. The HttpClient request is built for me and does not include the extra quotes and semi-colon. So the API I am using does not play well with request being generated by the HttpClient even though the request is technically correct.
Thanks to Panagiotis Kanavos for showing me my error.

eBay FileTransferService - Please specify a file with a valid format

I'm working with the eBay Large Merchant Services. I have successfully gotten the basic AddFixedPriceItem job to work using .NET HttpClient (finally). However, depending on the content of my AddFixedPriceItem XML file, the response to the uploadFile job would return:
Please specify a File with Valid Format
I'm not able to pinpoint what is causing this error.
Example #1:
I have a file with 2 AddFixedPriceItemRequest requests, which would go up successfully, and a file with 3 AddFixedPriceItemRequest requests which would not. When I would remove some NameValueList tags from the 3rd request, it would work. I don't see anything wrong or different with those NameValueList tags that I removed. If I send up the 3rd request by itself, without removing any tags it works perfectly.
Example #2:
I have 1 AddFixedPriceItemRequest request in a file that contains CDATA information for the Description tag (for HTML values), which would go up successfully, and another file with 2 AddFixedPriceItemRequest requests also containing almost identical CDATA values but would not work. If I remove the CDATA from the added second request, it still does not work. If I also remove the CDATA from the 1st request as well ( which went up successfully by itself the first time around) then it does work.
I'm really confused by this. There seems to be no rhyme and reason as to which a file is considered "valid".
Below is an example of a request that failed: (Stack Overflow doesn't seem to be taking the Content-IDs - but they are there.)
POST http://storage.sandbox.ebay.com/FileTransferService HTTP/1.1
X-EBAY-SOA-SECURITY-TOKEN: xxx
X-EBAY-SOA-SERVICE-NAME: FileTransferService
X-EBAY-SOA-SERVICE-VERSION: 1.0.0
X-EBAY-SOA-OPERATION-NAME: uploadFile
X-EBAY-SOA-OPERATION-FORMAT: XML
Content-Type: multipart/related; boundary="MIME_boundary"; type="application/xop+xml"; start="<0.urn:uuid:9ce221c2-659d-4852-a166-51dcbccf68d9>"; start-info="text/xml"
Host: storage.sandbox.ebay.com
Content-Length: 3041
Expect: 100-continue
--MIME_boundary
Content-Type: application/xop+xml
Content-ID: <0.urn:uuid:9ce221c2-659d-4852-a166-51dcbccf68d9>
Content-Transfer-Encoding: binary
<?xml version="1.0" encoding="utf-8"?><uploadFileRequest xmlns="http://www.ebay.com/marketplace/services"><fileAttachment><Data><xop:Include href="cid:urn:uuid:c6f7fc4d-352b-414f-8638-6c9bdf40519e" xmlns:xop="http://www.w3.org/2004/08/xop/include"/></Data><Size>1710</Size></fileAttachment><fileFormat>zip</fileFormat><fileReferenceId>50008675134</fileReferenceId><taskReferenceId>50008489744</taskReferenceId></uploadFileRequest>
--MIME_boundary
Content-Type: application/octet-stream
Content-Transfer-Encoding: base64
Content-ID: <urn:uuid:c6f7fc4d-352b-414f-8638-6c9bdf40519e>
UEsDBBQAAAAIAMZZ9UhnranV5gUAADYmAAAPACQAYWRkbGlzdGluZ3MueG1sCgAgAAAAAAABABgA4roeimLj0QE1N5fd4t3RAex2PDlG3dEB7Vptc9o4EP4rmny59kOxDZiXjOoZXppL5kjDxaG5+9QR9gJqbcsny2m4X38rgwEbmJ57zDWd8inSrvaRdq1dPRKh/TT4PGSKvXv2Fiyawz38lUKiEvIcBlHy9iKV0SVM2fKSxTy5hP6ozxIYiDAWEUQquXDoNTAfpEM/gEy4iJxuy6ZG3qEuV3AzdExqrFvUyA16vn/Fn8EfS+7BjYJwPfe/n/qdlEKOcNkpm4MD0ceJS42ikD4yGfFoPoInCJxrPl9QoyA6sm69HocOmIK5kMtbFsdo0QsC8QV8R8kUqHFESQci8rlCFPTWMk10fVeC6jRScunoteZtOkilhMjT0iGK8x4dQuJJHmtbZ8h4JAKMIhmhGwqnJMPRda9lN6/e3PX7xP2Cojc9GRJrNYJcBUJIMmJhTHhE7nhA+jJNFuDjXxH9jT7s4tMhT2KmvMUDD+GWPTsYj7KI3kRPGHz0+kEy7zNOeAtqIXzH/W1CjWNKOhIey+YwO23Tosamn8XZjcHjM+4lDn3PQvjAghRGPFGrrtOXLPKpkbVpptyGAr9YJlipS5ZloNvx+xLMTviqIQkfghNhYQZOycMyhiLeNQvEHKLKUP8dZQBS6c+RfaEi3MDtEYNMRpXwrnjEk0UR6NBerBJ/9kwemVKY4UVczLcvlZDep+EUJBEzooOXlNAqQd09gWRBQIaYK5GuIiU0u0luohq5Jn8Qq5M1H3XTzJrHQ2qUUkRLMbX0jnGK9RPzakdHx9xTqYQhKMaDZNOf3I+chVLxpWHwcF6bCzQIa54IjZsQ45kYm+Qydvd07VM8p8YOxqaznUDykMllXhe35VMXQqtud6xt0cxOgj2DsRR+6qm1GxvgrAJg9q5bu+m/ElCtLGZgNtzYWt5EXpD6AH22XM+Sw89YkIAuXscG5LauEt7n8UIoof0vmhV1dDIeOGbLrltNXFGnjo5rifb4oIO/pyxSXC2dFjU2bXoPGN5ojJ56m17S8zyIFfh3q6JdklLj8DC0nqVR3rvFE3TZxxKth+/I8zkeuVrw6C4/dpbJx4a5QS4oCwfUAzwr4m8FOqtkZkTizAlUZg6TBUiolU4fd8Gzc3QgEjVm3O8v13P00yVI5A/H9PnK8kDlAzfRzQWrlAmY2oKtMiXvuSCfMJdWuMmeHLerkPrLWFuAsqpsgwf62M21e2Z7w7VzxFsf/zfDtxfIBy4cs2bumeqRe+ZIqTKewYJMjXb7hqUx9EoC5EPWzKYg2gPYhMfYjzTmZEZssgZFUuA0Oo1Wo40SbFNXMamyeuVY7Q4Kt336wFUApyQ5tRpusRUqnUyw4DTqdpt5zGyD7zeb9S7zwJ/aDctn7VbLtGeYpJOsLq3In3GMoJ6p6ympqzv46kd1kYpEZLCQIjxz1m3cTkZZK0L9lIy1uAfPXPXFclV38ANRVZ15L4mpWs0zUz0z1Z+YqXZOwlQrc5pdmtpldrfV9L0mq0MTZmxarzPPt6yuz5jFWPdMU/9nmtpumKvLR8//lCaKTQM4P7FWoKt5/E7BV78JqxfHwZodkleTEdEq8F+X0OWS5EGrhp4kEE6DJdHZxSWUAvknJNXZtU7EIkyfkzHusFe/Nl+fyXoBT0ifHGLs/SDjDVWhRhDNVQnK/qUiTlZ0NKu4k3zOy24iK2GVAIfIkEpLsloX3/sV/hp0CSxFyqq2rFss8ZIjQS3dthTo7D/FNct+wbcsa3vLam1uWZ2vXLIOzZTVdxaQK2D6nlOaB8+KKJkJGaJf61uAXw1eLYNSWPHExXMlFhJvRZi0ulzLaoXATed4hcNCTO6FCA/UqLsZlolqWxJZFx6lB7BWikpYu1E7yc7Sq9L7age4CHjL5hEo7lVC/SCCA28LdfOpEsoRBx+rgXBfLXAzDDn2VNm7zvHK8J3fDXJS8WM8HGwp0Et5OcAV1a3zb1znl4Of+OWg2dh/Oai36xVeDr7pmrn3I5fZsVseXortrj1tWp1Wt9Xw/DbzZzbmadOr8COXcex/w5x/AFBLAQItABQAAAAIAMZZ9UhnranV5gUAADYmAAAPACQAAAAAAAAAIAAAAAAAAABhZGRsaXN0aW5ncy54bWwKACAAAAAAAAEAGADiuh6KYuPRATU3l93i3dEB7HY8OUbd0QFQSwUGAAAAAAEAAQBhAAAANwYAAAAA
--MIME_boundary--
Here's how I send the request with HttpClient:
byte[] data;
long fileLength;
var uuidRequest = Guid.NewGuid().ToString();
var uuidAttachment = Guid.NewGuid().ToString();
using (FileStream fs = File.OpenRead(filePath))
{
data = new byte[fs.Length];
fs.Read(data, 0, data.Length);
fileLength = fs.Length;
}
var xml =
"<?xml version=\"1.0\" encoding=\"utf-8\"?>" +
"<uploadFileRequest xmlns=\"http://www.ebay.com/marketplace/services\">" +
"<fileAttachment>" +
"<Data><xop:Include href=\"cid:urn:uuid:" + uuidAttachment + "\" xmlns:xop=\"http://www.w3.org/2004/08/xop/include\"/></Data>" +
"<Size>" + fileLength + "</Size>" +
"</fileAttachment>" +
"<fileFormat>zip</fileFormat>" +
"<fileReferenceId>" + fileReferenceID + "</fileReferenceId>" +
"<taskReferenceId>" + jobID + "</taskReferenceId>" +
"</uploadFileRequest>";
var multippartcontent = new MultipartContent("related", "MIME_boundary");
multippartcontent.Headers.Remove("Content-Type");
multippartcontent.Headers.TryAddWithoutValidation("Content-Type", string.Format("multipart/related; boundary=\"MIME_boundary\"; type=\"application/xop+xml\";start=\"<0.urn:uuid:{0}>\";start-info=\"text/xml\"", uuidRequest));
var content = new StringContent(xml);
content.Headers.ContentType = new MediaTypeHeaderValue("application/xop+xml");
content.Headers.Add("Content-ID", string.Format("<0.urn:uuid:{0}>", uuidRequest));
content.Headers.TryAddWithoutValidation("Content-Transfer-Encoding", "binary");
multippartcontent.Add(content);
var attachmentContent = new StringContent(Convert.ToBase64String(data));
attachmentContent.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
attachmentContent.Headers.TryAddWithoutValidation("Content-Transfer-Encoding", "base64");
attachmentContent.Headers.Add("Content-ID", string.Format("<urn:uuid:{0}>", uuidAttachment));
multippartcontent.Add(attachmentContent);
var uploadFileResp = httpclient.PostAsync(ftServiceUrl, multippartcontent).Result.Content.ReadAsStringAsync().Result;
I would note, when I upload this same file that doesn't work for me with the LMS project from eBay (https://ebaydts.com/eBayKBDetails?KBid=1338). That leads me me to think I'm doing something wrong in my code - perhaps something to do with how I'm encoding?
HTTP Response
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
X-EBAY-SOA-SERVICE-METRICS: 455075973
X-EBAY-SOA-REQUEST-ID: 15613449-f630-a471-d2d4-ec64ffff71c7!FileTransferService!10.71.29.45!v3apibulksandboxcore[]
X-EBAY-SOA-SERVICE-VERSION: 1.1.0
X-EBAY-SOA-MESSAGE-PROTOCOL: NONE
X-EBAY-SOA-RESPONSE-DATA-FORMAT: XML
X-EBAY-SOA-OPERATION-NAME: uploadFile
X-EBAY-SOA-SERVICE-NAME: {http://www.ebay.com/marketplace/services}FileTransferService
Content-Type: text/xml;charset=UTF-8
Transfer-Encoding: chunked
Date: Fri, 22 Jul 2016 15:41:19 GMT
1cd
<?xml version='1.0' encoding='UTF-8'?><uploadFileResponse xmlns="http://www.ebay.com/marketplace/services"><ack>Failure</ack><errorMessage><error><errorId>11</errorId><domain>Marketplace</domain><severity>Error</severity><category>Application</category><message>Please specify a File with Valid Format</message><subdomain>FileTransfer</subdomain></error></errorMessage><version>1.1.0</version><timestamp>2016-07-22T15:41:20.118Z</timestamp></uploadFileResponse>
0

I finally got the answers. The "100 continue" message gave me the needed clues. I was spending too much time trying to understand the intermittent operation.
1) Your code never really works. When you run the LMS project a cookie is establish which is setting up default setting for you request. So your code will always work after running the LMS. If you delete cookies from your IE history your code will never work.
2) The "100 continue message" is used for "Chunk Mode" (not stream). The Net library defaults to HTTP 1.1 and most server these days when the see 1.1 will default to chunk mode. The server sends one chunk , but there is no way in the Net library to request a 2nd chunk from server. So the connection times out and fails. The only way around this issue is to set the HTTP request to HTTP 1.0 which doesn't support chunk mode the server will send the response back as a stream.
Try this
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("URL");
request.ProtocolVersion = HttpVersion.Version10;
See also : Set HTTP protocol version in HttpClient
Look at 1st line of request that failed which has 1.1

Where does strange MIME types come from?

I have a web service for uploading files written in C#.
Front-end application is written in Javascript / HTML5 (using https://github.com/blueimp/jQuery-File-Upload)
Recently, I was reviewing server logs and found some strange MIME types for PDF files that where sent by client browser, for example:
application/unknown
application/force-download
application/force-download/n
application/force-download\n
[application/pdf]
Some of them are causing .NET framework throwing exception:
MultipartMemoryStreamProvider streamProvider = new MultipartMemoryStreamProvider();
await Request.Content.ReadAsMultipartAsync(streamProvider);
"Message Error parsing MIME multipart body part header byte 156 of data segment System.Byte[]."
I don't have a clue what to do with that.

Try checking the Content-Type in the request.
Content-Disposition:
form-data;
name="imagefile";
filename="C:\Users\Pictures\sid.png"
Content-Type:
(notice the blank Content-Type, it should be Content-Type: image/png )
// within WebAPI you can use code below to log the request body
string requestBody = await Request.Content.ReadAsStringAsync();

How to call methods after HttpResponse?

I am a bit puzzled at the moment. I have a web application that manipulates a file and then returns the file to a user's browser for download when it's done.
The download part is going well, as I'm using Response.AddHeader and Reponse.BinaryWrite to push the file back to the browser but I am unable to call any further methods after using Response methods.
I suppose I have not worked with HttpReponse enough to know the trick to this. Perhaps I would be better off using another class or generic handler to handle the download?
My code goes something like...
// Methods to be called first
Response.Clear();
Response.ClearContent();
Response.ClearHeaders();
Response.AddHeader("Content-Disposition", string.Format("attachment;filename={0}.pdf", "New_Merged_PDF_" + DateTime.Now.ToString("MMMM-dd-yyyy")));
Response.ContentType = "application/pdf";
Response.BinaryWrite(output.ToArray());
Response.End();
// Methods to be called last (these wont work)
Probably something simple that I'm overlooking but I'm still trying to figure it out.

To add a little color to Servy's explaination; there is an order of operations within the HTTP protocol specification. One of them is that the Headers need to be sent to the client before the Body. This allows the receiver of the response to properly deal with the Body that is sent based on any Headers.
The presence of a message-body in a request is signaled by the
inclusion of a Content-Length or Transfer-Encoding header field in
the request's message-headers.
IETF RFC 2616 - Hypertext Transfer Protocol -- HTTP/1.1
One of the few (if the only, I'm not sure) exception is the Content-Type: multipart/mixed;
IETF RFC 1867 - Form-based File Upload in HTML (although Obsolete, examples are still relevant)
Content-type: multipart/form-data, boundary=AaB03x
--AaB03x
content-disposition: form-data; name="field1"
Joe Blow
--AaB03x
content-disposition: form-data; name="pics"; filename="file1.txt"
Content-Type: text/plain
... contents of file1.txt ...
--AaB03x--