I have below code in a third party function, it posts file to a webserver. I want to post data in parts, what change should i do in the code. Below code is working and the "request" object contains everything.
private static HttpWebResponse GetRawResponse(HttpWebRequest request)
{
return (HttpWebResponse)request.GetResponse();
}
Also is there a way in which I can find out what is the full name (with path) of the file which is going to be uploaded from the httpwebrequest object.
Thanks.
HttpWebRequest does not spawn multiple HTTP requests. You cannot upload a file in chunks unless you actually create multiple HttpWebRequests. I don't think that would be useful unless you have a server side process to stitch them back together.
If you still want true chunked uploading you might want to look at raw TCP or some other mechanism.
Hope that helps. See this post as well.
Related
I have a pretty big video file I upload to a web service via multipart/form-data.
It takes ~ 30 seconds to arrive and I would prefer not waiting that long simply to access parameters I send along with the file.
My question is simple, can I access parameters sent with the form without waiting for the video payload to be uploaded?
Can this be done using headers or any other methods?
Streaming vs. Buffering
It's about how the webserver is set up. For IIS you can enable Streaming.
Otherwise, by default, IIS will use 'buffering' - the whole request is loaded into memory first (IIS's memory that you can't get to) before your app running in IIS can get it.
Not using IIS? You have to figure out how to get the webserver to do the same thing.
How to stream using IIS:
Streaming large file uploads to ASP.NET MVC
Note the way the file is read in the inner loop:
while ((cbRead = clientRequest.InputStream.Read(rgbBody, 0, rgbBody.Length)) > 0)
{
fileStream.Write(rgbBody, 0, cbRead);
}
Here instead of just saving the data like that question does, you will have to parse any xml/json/etc or whatever contains the file parameters you speak of ... and expect the video to be sent afterwards. You can process them right away if it's a quick process ... then get the rest of the video ... or you can send them to a background thread.
You probably won't be able to parse it just dumping what you have to a json or xml parser, there will be an unclosed tag or } at the top that isn't closed til after the video data is uploaded (however that is done). Or if it's multipart data from a form submission, as you imply, you will have to parse that partial upload yourself, instead of just asking IIS for the post data.
So this will be tricky, you can first start by writing 1k at a time to a log file with a time stamp to prove that you're getting the data as it comes. after that it's just a coding headache.
Getting this to work also means you'll have to have some control over the client and how it sends the data.
That's because you'll at least have to ensure it sends the file parameters FIRST!
Which concerns me, because, if you have control of the client, why can't you take the simple route (as Nobody and Nkosi imply) and use 2 requests? You mention you need one. Why not write js client code to send the parameters first in an XHR and then the file in a second request, using a correlation ID in both to tie them together? (the server could return this from the first request and you could send it in the 2nd).
Obviously, if you're just having a form with some inputs and a file upload and doing submit, then you need one request ;-) But if you have control over the client side you're not stuck with that.
Good luck, there is some advanced programming here, but nothing super high-tech. You will make it work!!!
If you don't have control over the server code, you are probably stuck, if the server app's webserver is buffering, the server app won't get anything, of course, if you wanted to do something with the file parameters first, this really implies you have control of the server side ;-)
I'm trying to get the raw data sent to IIS using a HttpHandler. However, because the request is an "GET"-request without the "Content-Length" header set it reports that there is no data to read (TotalBytes), and the inputstream is empty. Is there any way I can plug into the IIS-pipeline (maybe even before the request is parsed) and just kind of take control over the request and read it's raw data? I don't care if I need to parse headers and stuff like that myself, I just want to get my hands on the actual request and tell IIS to ignore this one. Is that at all possible? Cause right now it looks like I need to do the alternative, which is developing a custom standalone server, and I really don't want to do that.
Most web servers will ignore (and rarely give you access to) the body of a GET request, because the HTTP semantics imply that it is to be ignored anyway. You should consider another method (for example POST or PUT).
See this question and the link in this answer:
HTTP GET with request body
My C# program communicates with a server using a web service, I need the client to download big files from the server and have the option to pause and continue their download, the downloader must also be authorized to download the file.
I had two thoughts on how to do that,
one is to use some 3rd party API like wget to download the files. the problem with that is that I need to learn the API commands and that I'm not certain I can show my download progress in the program, another issue is that I would have to use use bare URLs to get the files from the server which seems ugly and could lead to people just downloading them off the server (I want them to be authorized, although this isn't a real issue since this is just a school project).
My other thought was to create a method on the web service that will get a position in the file and an amount of bytes and return them and the client will piece them together, it seems more complicated but more compelling since the user must be authorized to download the file and I can use it to show the tester some more advanced programming skills ;). The issue with that looks like it might be performance taxing.
What's your opinion? what's the best way to download big files off a server?
Absent the need for authorization and partial downloads, WebClient.DownloadData or WebClient.DownloadDataAsync would be the preferred method of downloading a file from a server.
You could still use WebClient for the authorization by setting the Credentials in your WebClient object instance. If the user isn't authorized to download the file, based on those credentials, the server can return a 404 (Not found) or 403 (Forbidden).
If your server supports HTTP 1.1, the client can start in the middle of the file. To do so, you'll have to create a class that inherits from WebClient and override the GetWebRequest method. That method would then set the headers to do a positional GET.
class MyWebClient : WebClient
{
public int StartDownloadAt { get; set; }
protected override WebRequest GetWebRequest(Uri address)
{
HttpWebRequest req = (HttpWebRequest)base.GetWebRequest(address);
req.AddRange(position_to_start);
}
}
And in the code that uses it:
MyWebClient client = new MyWebClient();
client.StartDownloadAt = 1024 * 2024; // start download 1 megabyte into file.
client.DownloadData(...);
The above is just an example. You'd probably want to make that more robust by having the StartDownLoadAt property reset to 0 when a download is done (or aborted), and not do the AddRange if StartdownloadAt is set to 0. To fully support ranges, you'd probably want properties for start and end range, etc.
And, of course, the client will have to handle stitching the disparate downloaded pieces together after download is complete.
The point is that it should be possible, with a little work, by using the WebClient class.
I am trying to create a webservice that would allow its consumers to download files (can be very huge files). At the server, I have many files that need to be sent back to the consumer, so I am compressing all those file into one big zip file and streaming them back to the user. Right now, my webservice will start compressing the files, when the request comes in, forms the zip files and streams it back.Sometimes compression can take a lot of time, and the request may time out. What can I do to avoid such situations? My solution right now is to, seperate the data into smaller zip files, send a response to consumer saying there would be these many smaller files, and let consumers send request for individual smaller files. So, if i have 1GB zip file, i will break it into 10 smaller zip files, and ask the consumer to request for smaller files in 10 requests. Is this the correct approach? What problems can I be facing? Has anyone dealth with such issues before? I would be glad if you can share your experiences. Also, is it possible to start streaming the zip files without forming them fully?
Treat the request and the delivery as asynchronous operations.
The client can make a request for the file using one method. Another method can let the client know the status of the file packaging (whether they are ready for download yet). A third method can actually download the files.
It may be worth looking at a restful approach. Rather than a soap web service. As OrbMan, suggested an asynch approach may be best.
With REST you could expose a resource as: http://yourlocation/generatefile
Which (when called with a post) returns a http response with a response code of 301 'accepted' and a location header value of location=http://yourlocation/generatefile/id00124 which suggests the location of the data.
You can then poll the http://yourlocation/generatefile/id00124 resource (maybe just header request) to get the status i.e. processing / complete.
When processing is complete. Do a get on the http://yourlocation/generatefile/id00124 to download your file. The response http message should identify you file and the format i.e. encryption and compression types so any consumer knows how to read it.
This is a nice solution to problems which are long running and returns data in formats other than soap anbd general xml.
I hope this helps
I would poll from the calling client as part of the method which gets the file. The client code might flow something like this:
byte[] GetFile()
{
response = request.Post(http://yourlocation/generatefile);
string dataResource = response.Headers["Location"];
bool resourceReady = false;
while(!reasourceReady)
{
resH = request.Header(dataResource);
if(resH.Headers[Status] == "complete")
break;
else
Thread.Sleep(OneSecond); ?? or whetever
}
fileRes = request.Get(dataResource);
return fileRes.ToByteArray();
}
This is only psuedo, but I hope it makes sense...
I know about the WebRequest and the WebResponse objects. The problem is that I do not really want to get the source code of the webpage, I only want to check to see if the link exists or not. The thing is, if I use the GetResponse method, it goes an pull the entire source code of the site.
I am creating a broken link checker with many links. It takes quite a while to check them all. If there a way to to get MINIMAL information from a weblink? Only enough information to see if the link is valid or broken (not the entire source code).
An answer (BESIDES USING ASYNCHRONOUS TRANSFER) would be greatly appreciated!
WebRequest request = HttpWebRequest.Create("http://www.foo.com/");
request.Method = "HEAD"; // Just get the document headers, not the data.
HEAD is similar to GET, only that instead of getting the file contents, we get just the headers.
A standard way of checking the existence of a link is to use a HEAD request, which causes the remote server to send the headers for the requested object, but not the object itself. If you thus requested an object that is not on the server, the server gives you the normal 404 response, but if it does exist, you get a 200 response and no data after the headers. This way very little uninteresting data goes over the wire.