How to download first 200 bytes of a file via HTTP protocol using C#?
I believed it could be done like this:
WebClient wc = new WebClient();
byte[] buffer = new byte[200];
using (var stream = wc.OpenRead(fileName))
{
stream.Read(buffer, 0, 200);
}
but when wc.OpenRead it called it downloads the whole file.
You need to set a Range Header on your WebClient before you invoke the OpenRead method.
See: http://msdn.microsoft.com/en-us/library/system.net.webclient.headers.aspx
Related
I am working on an ASP.NET framework 2.0 application. On a particular page I am providing a link to user. By clicking on this link a window opens with another aspx page. This page actually sends http request to a third-party url which points to a file(like - mirror urls to download file from cloud). The http response is sent back to user on the very first page using response.write from where user click the link.
Now, the problem I am facing is if the file size is low then it works fine. But, if the file is large (i.e., more than 1 GB), then my application waits until whole file is downloaded from the URL. I have tried using response.flush() to send chunk by chunk data to user, but still user is unable to use application because the worker process is busy getting streams of data from third party URL.
Is there any way by which large files can be downloaded asynchronously so that my pop-up window finishes its execution(download will be in progress) and also user can do other activities on application parallely.
Thanks,
Suvodeep
Use WebClient to read the remote file. Instead of downloading you can take the Stream from the WebClient. Put that in while() loop and push the bytes from the WebClient stream in the Response stream. On this way, you will be async downloading and uploading at the same time.
HttpRequest example:
private void WriteFileInDownloadDirectly()
{
//Create a stream for the file
Stream stream = null;
//This controls how many bytes to read at a time and send to the client
int bytesToRead = 10000;
// Buffer to read bytes in chunk size specified above
byte[] buffer = new byte[bytesToRead];
// The number of bytes read
try
{
//Create a WebRequest to get the file
HttpWebRequest fileReq = (HttpWebRequest)HttpWebRequest.Create("Remote File URL");
//Create a response for this request
HttpWebResponse fileResp = (HttpWebResponse)fileReq.GetResponse();
if (fileReq.ContentLength > 0)
fileResp.ContentLength = fileReq.ContentLength;
//Get the Stream returned from the response
stream = fileResp.GetResponseStream();
// prepare the response to the client. resp is the client Response
var resp = HttpContext.Current.Response;
//Indicate the type of data being sent
resp.ContentType = "application/octet-stream";
//Name the file
resp.AddHeader("Content-Disposition", $"attachment; filename=\"{ Path.GetFileName("Local File Path - can be fake") }\"");
resp.AddHeader("Content-Length", fileResp.ContentLength.ToString());
int length;
do
{
// Verify that the client is connected.
if (resp.IsClientConnected)
{
// Read data into the buffer.
length = stream.Read(buffer, 0, bytesToRead);
// and write it out to the response's output stream
resp.OutputStream.Write(buffer, 0, length);
// Flush the data
resp.Flush();
//Clear the buffer
buffer = new byte[bytesToRead];
}
else
{
// cancel the download if client has disconnected
length = -1;
}
} while (length > 0); //Repeat until no data is read
}
finally
{
if (stream != null)
{
//Close the input stream
stream.Close();
}
}
}
WebClient Stream reading:
using (WebClient client = new WebClient())
{
Stream largeFileStream = client.OpenRead("My Address");
}
I have an CSV file in memory that I want to upload to a Web API.
If I save the CSV file to disk and upload it, it gets accepted.
However, I want to avoid the extra work and also make the code cleaner by simply uploading the text I have as a MemoryStream Object (I think that's the correct format?).
The following code works for uploading the file:
string webServiceUrl = "XXX";
string filePath = #"C:\test.csv";
string cred = "YYY";
using (var client = new WebClient()){
client.Headers.Add("Authorization", "Basic " + cred);
byte[] rawResponse = client.UploadFile(webServiceUrl, "POST", filePath);
Console.WriteLine(System.Text.Encoding.ASCII.GetString(rawResponse));
}
How would I do if I had a string with all the contents and I want to upload it in the same way without having to save it down to a file?
WebClient.UploadData or WebClient.UploadString perhaps?
Thank you
EDIT:
I tried what you said but by using a local file (in case there was something wrong with the string), but I get the same error.
Here is what I suppose the code would be using your solution
string webServiceUrl = "XXX";
string file = #"C:\test.csv";
string cred = "YYY";
FileStream fs = new FileStream(file, FileMode.Open, FileAccess.Read);
BinaryReader r = new BinaryReader(fs);
byte[] postArray = r.ReadBytes((int)fs.Length);
using (var client = new WebClient())
{
client.Headers.Add("Authorization", "Basic " + cred);
using (var postStream = client.OpenWrite(webServiceUrl, "POST"))
{
postStream.Write(postArray, 0, postArray.Length);
}
}
Any thoughts?
Use OpenWrite() from the WebClient.
using (var postStream = client.OpenWrite(endpointUrl))
{
postStream.Write(memStreamContent, 0, memStream.Length);
}
As documentation mentioned:
The OpenWrite method returns a writable stream that is used to send data to a resource.
Update
Try to set the position of the MemoryStream to 0 before uploading.
memoryStream.Position = 0;
When you copy the file into the MemoryStream, the pointer is moved to the end of the stream, so when you then try to read it, you're getting a null byte instead of your stream data.
MSDN - CopyTo()
Copying begins at the current position in the current stream, and does not reset the position of the destination stream after the copy operation is complete.
I finally managed to solve it.
First I made a request using CURL that worked.
I analyzed the packet data and made an except copy of the packet.
I did a lot of changes, however, the final change was that using the different functions I found online it never closed the packet with a "Last-Boundary" while CURL did.
So by modifying the function, making sure it properly wrote a Last-Boundary it finally worked.
Also, another crucial thing was to set PreAuthenticate to true, the examples online didn't do that.
So, all in all:
1. Make sure that the packet is properly constructed.
2. Make sure you pre authenticate if you need to authenticate.
webrequest.PreAuthenticate = true;
webrequest.Headers[HttpRequestHeader.Authorization] = string.Format("Basic {0}", cred);
Don't forget to add SSL if using a https (which you probably do if you authenticate):
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Ssl3;
Hope this helps someone.
And thanks for the help earlier!
I want to send a url as query string e.g.
localhost/abc.aspx?url=http:/ /www.site.com/report.pdf
and detect if the above URL returns the PDF file. If it will return PDF then it gets saved automatically otherwise it gives error.
There are some pages that uses Handler to fetch the files so in that case also I want to detect and download the same.
localhost/abc.aspx?url=http:/ /www.site.com/page.aspx?fileId=223344
The above may return a pdf file.
What is best way to capture this?
Thanks
You can download a PDF like this
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uri);
HttpWebResponse response = req.GetResponse();
//check the filetype returned
string contentType = response.ContentType;
if(contentType!=null)
{
splitString = contentType.Split(';');
fileType = splitString[0];
}
//see if its PDF
if(fileType!=null && fileType=="application/pdf"){
Stream stream = response.GetResponseStream();
//save it
using(FileStream fileStream = File.Create(fileFullPath)){
// Initialize the bytes array with the stream length and then fill it with data
byte[] bytesInStream = new byte[stream.Length];
stream.Read(bytesInStream, 0, bytesInStream.Length);
// Use write method to write to the file specified above
fileStream.Write(bytesInStream, 0, bytesInStream.Length);
}
}
response.Close();
The fact that it may come from an .aspx handler doesn't actually matter, it's the mime returned in the server response that is used.
If you are getting a generic mime type, like application/octet-stream then you must use a more heuristical approach.
Assuming you cannot simply use the file extension (eg for .aspx), then you can copy the file to a MemoryStream first (see How to get a MemoryStream from a Stream in .NET?). Once you have a memory stream of the file, you can take a 'cheeky' peek at it (I say cheeky because it's not the correct way to parse a PDF file)
I'm not an expert on PDF format, but I believe reading the first 5 chars with an ASCII reader will yield "%PDF-", so you can identify that with
bool isPDF;
using( StreamReader srAsciiFromStream = new StreamReader(memoryStream,
System.Text.Encoding.ASCII)){
isPDF = srAsciiFromStream.ReadLine().StartsWith("%PDF-");
}
//set the memory stream back to the start so you can save the file
memoryStream.Position = 0;
I am referring to this article to understand file downloads using C#.
Code uses traditional method to read Stream like
((bytesSize = strResponse.Read(downBuffer, 0, downBuffer.Length)) > 0
How can I divide a file to be downloaded into multiple segments, so that I can download separate segments in parallel and merge them?
using (WebClient wcDownload = new WebClient())
{
try
{
// Create a request to the file we are downloading
webRequest = (HttpWebRequest)WebRequest.Create(txtUrl.Text);
// Set default authentication for retrieving the file
webRequest.Credentials = CredentialCache.DefaultCredentials;
// Retrieve the response from the server
webResponse = (HttpWebResponse)webRequest.GetResponse();
// Ask the server for the file size and store it
Int64 fileSize = webResponse.ContentLength;
// Open the URL for download
strResponse = wcDownload.OpenRead(txtUrl.Text);
// Create a new file stream where we will be saving the data (local drive)
strLocal = new FileStream(txtPath.Text, FileMode.Create, FileAccess.Write, FileShare.None);
// It will store the current number of bytes we retrieved from the server
int bytesSize = 0;
// A buffer for storing and writing the data retrieved from the server
byte[] downBuffer = new byte[2048];
// Loop through the buffer until the buffer is empty
while ((bytesSize = strResponse.Read(downBuffer, 0, downBuffer.Length)) > 0)
{
// Write the data from the buffer to the local hard drive
strLocal.Write(downBuffer, 0, bytesSize);
// Invoke the method that updates the form's label and progress bar
this.Invoke(new UpdateProgessCallback(this.UpdateProgress), new object[] { strLocal.Length, fileSize });
}
}
you need several threads to accomplish that.
first you start the first download thread, creating a webclient and getting the file size. then you can start several new thread, which add a download range header.
you need a logic which takes care about the downloaded parts, and creates new download parts when one finished.
http://msdn.microsoft.com/de-de/library/system.net.httpwebrequest.addrange.aspx
I noticed that the WebClient implementation has sometimes a strange behaviour, so I still recommend implementing an own HTTP client if you really want to write a "big" download program.
ps: thanks to user svick
I have tried to implement a REST WCF in order to explore difference between PUT and POST verb. I have uploded a file in a location using the service.
The service implementation is as folowing:
[OperationContract]
[WebInvoke(UriTemplate = "/UploadFile", Method = "POST")]
void UploadFile(Stream fileContents);
public void UploadFile(Stream fileContents)
{
byte[] buffer = new byte[32768];
MemoryStream ms = new MemoryStream();
int bytesRead, totalBytesRead = 0;
do
{
bytesRead = fileContents.Read(buffer, 0, buffer.Length);
totalBytesRead += bytesRead;
ms.Write(buffer, 0, bytesRead);
} while (bytesRead > 0);
using (FileStream fs = File.OpenWrite(#"C:\temp\test.txt"))
{
ms.WriteTo(fs);
}
ms.Close();
}
Client code is as following:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("http://localhost:1922 /EMPRESTService.svc/UploadFile");
request.Method = "POST";
request.ContentType = "text/plain";
byte[] fileToSend = File.ReadAllBytes(#"C:\TEMP\log.txt"); // txtFileName contains the name of the file to upload.
request.ContentLength = fileToSend.Length;
using (Stream requestStream = request.GetRequestStream())
{
// Send the file as body request.
requestStream.Write(fileToSend, 0, fileToSend.Length);
//requestStream.Close();
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
Console.WriteLine("HTTP/{0} {1} {2}", response.ProtocolVersion, (int)response.StatusCode, response.StatusDescription);
Console.ReadLine();
The file is being uploaded and the response status code is being returned as "200 OK". The satus code is same in case of existance or non-existance of the file in the upload location.
I have changed the REST verb to PUT and the status code is same as above.
Could anybody explain, how I can identify the differences between the verbs in this context? I couldn't able to simulate generating continious request fron client code. If the behaviour will differ on doing so, could anybody help me in modifying the client code in ordrr to send continious request in a row ?
POST verb is used when are you creating a new resource (a file in your case) and repeated operations would create multiple resources on the server. This verb would make sense if uploading a file with the same name multiple times creates multiple files on the server.
PUT verb is used when you are updating an existing resource or creating a new resource with a predefined id. Multiple operations would recreate or update the same resource on the server. This verb would make sense if uploading a file with the same name for the second, third... time would overwrite the previously uploaded file.