I have a requirement where I need to read large stream or data from a web server.
This is not streaming but reading large data in chunks from the client side.
For this purpose I create an Http Web Request. Following is the sample code..
StreamingObject streamObj = null;
using (HttpWebRequest httpReq = WebRequest.Create(uri))
{
HttpWebResponse response = httpReq.GetRespons();
Stream responseStream = response.GetStream();
streamObj = new StreamingObject(response, responseStream);
}
return streamObj;
In my code I make a Http Web request to given Uri. Then I get the HttpWebResponse and the Response Stream. Create an instance of StreamingObject which is a wrapper class for storing the Http response and the response stream.
I have disposed the Http Web Request. The client is provided with the StreamingObject and the streaming object has one method GetStream which returns the underlying response stream. When the StreamingObject is disposed the Http response and the stream is disposed.
The reason for introducing the StreamingObject was that the Http Response has to be kept open for accessing the underlying stream.
I wish to know whether this is a correct approach or there is simpler way of doing.
I would suggest not disposing of the HttpWebRequest object, because you might very well need it. Just add it to StreamingObject and dispose of it when you dispose of the StreamingObject. For example, I've found that if I want to close the response stream prematurely, the response will hang until all of the data has been read. But calling Abort on the request object will immediately close the stream.
Related
I'm reading a file from user upload and it was working synchronously. I needed to change it in order to immediately send a "received" alert to the user, then read the file asynchronously while the user would periodically poll back to see if the read was finished.
Here is what my code looks like right now:
public FileUpload SaveFile(Stream stream)
{
FileUpload uploadObj = //instantiate the return obj
var task = Task.Run(async () => await ProcessFileAsync(stream));
return upload;
}
public async Task ProcessFileAsync(Stream stream)
{
StreamReader file = new StreamReader(stream);
CsvReader csv = new CsvReader(file, CultureInfo.InvariantCulture);
while (await csv.ReadAsync())
{
//read the file
}
}
the issue I'm having is that by the time I call the csv.ReadAsync() method, the Stream object has been disposed. How do I access the Stream when I want the SaveFile() method to return a value to the user, but the act of returning disposes the Stream object?
The point here is that you're working within the constraints of ASP.NET, which abstracts away a lot of the underlying HTTP stuff.
When you say you want to process a user-uploaded file asynchronously, you want to step out of the normal order of doing things with HTTP and ASP.NET. You see, when a client sends a request with a body (the file), the server receives the request headers and kicks off ASP.NET to tell your application code that there's a new request incoming.
It hasn't even (fully) read the request body at this point. This is why you get a Stream to deal with the request, and not a string or a filename - the data doesn't have to be arrived at the server yet! Just the request headers, informing the web server about the request.
If you return a response at that point, for all HTTP and ASP.NET care, you're done with the request, and you cannot continue reading its body.
Now what you want to do, is to read the request body (the file), and process that after sending a response to the client. You can do that, but then you'll still have to read the request body - because if you return something from your action method before reading the request, the framework will think you're done with it and dispose the request stream. That's what's causing your exception.
If you'd use a string, or model binding, or anything that involves the framework reading the request body, then yes, your code will only execute once the body has been read.
The short-term solution that would appear to get you going, is to read the request stream into a stream that you own, not the framework:
var myStream = new MemoryStream();
await stream.CopyTo(myStream);
Task.Run(async () => await ProcessFileAsync(myStream));
Now you'll have read the entire request body and saved it in memory, so ASP.NET can safely dispose the request stream and send a response to the client.
But don't do this. Starting fire-and-forget tasks from a controller is a bad idea. Keeping uploaded files in memory is a bad idea.
What you actually should do, if you still want to do this out-of-band:
Save the incoming file as an actual, temporary file on your server
Send a response to the client with an identifier (the temporarily generated filename, for example a GUID)
Expose an endpoint that clients can use to request the status using said GUID
Have a background process continuously scan the directory for newly uploaded files and process them
For the latter you could hosted services or third-party tools like Hangfire.
You'll need to either do this if the environment warrants:
var result = task.Result;
//do stuff
...or
public Task<FileUpload> SaveFile(Stream stream)
{
var uploadObj = //instantiate the return obj
await ProcessFileAsync(stream);
return uploadObj;
}
See here for a thorough discussion on fire-and-forget if you go that route:
Web Api - Fire and Forget
I use to send POST request and get response by this way:
response = (HttpWebResponse)request.GetResponse();
But, I just want to send request, I don't care what response is. Size of response package can be up to 500Kb ~ 1Mb, It wastes lots of time. How can I send request and then stop receive response immediately. Thanks so much!
If your only concern is the time it takes to receive the response, and not the bandwidth being used, you could get the response asynchronously.
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.begingetresponse.aspx
The example given is a bit complicated, but the general idea is that your program will not wait for the response to be downloaded when BeginGetResponse is called, like it would if you just called GetResponse. The first method that you pass to BeginGetResponse is the name of a method (called a "callback") that will get called when the response eventually is fully downloaded. This is where you'd put your code to check the HTTP response code, assuming you cared about that. The 2nd parameter is a "state" object that gets passed to your callback method. We'll use this to make sure everything gets cleaned up properly.
It would look something like this:
private void YourMethod()
{
// Set up your request as usual.
request.BeginGetResponse(DownloadComplete, request);
// Code down here runs immediately, without waiting for the response to download
}
private static void DownloadComplete(IAsyncResult ar)
{
var request = (HttpWebRequest)ar.AsyncState;
var response = request.EndGetResponse(ar);
// You can check your response here to make sure everything worked.
}
I assume you are sending a GET request to the server. Change it to a HEAD request.
var request = System.Net.HttpWebRequest.Create("http://...");
request.Method = "HEAD";
request.GetResponse();
This will only return the length of the content. See How to get the file size from http headers for more info.
I have a chunk of javascript code that uses jQuery.post to send some data to a .NET app that's using an HttpListener.
Here's the js:
$.post("http://localhost:8080/catch", { name: "John", time: "2pm" },
function(data) {
alert(data);
});
and the C#:
HttpListenerContext context = listener.GetContext();
HttpListenerRequest request = context.Request;
StreamReader reader = new StreamReader(request.InputStream);
string s2 = reader.ReadToEnd();
Console.WriteLine("Data received:" + s2);
// Obtain a response object.
HttpListenerResponse response = context.Response;
// Construct a response.
string responseString = "<HTML><BODY> Hello world!</BODY></HTML>";
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
// Get a response stream and write the response to it.
response.ContentLength64 = buffer.Length;
System.IO.Stream output = response.OutputStream;
output.Write(buffer, 0, buffer.Length);
// You must close the output stream.
output.Close();
The post request goes out ok, and the .NET app reads in the data ok, but the JS code doesn't seem to get the response. The callback function to the jQuery.post fires, but data is always undefined.For brevity I have omitted some C# above where I set the prefixes to the listener.
Any ideas why I'm not getting my data back client-side?
EDIT: I should add that when I run the JS with HttpFox running I get Http code 200, 'NS_ERROR_DOM_BAD_URI', which I thought had something to do with the "http://localhost:8080/catch" I was targeting, but when I hit that resource in firefox, i get the HTML response just fine and it registers as a GET, 200.
EDIT: I simplified the response to just 'meow', and this is what fiddler is giving me for the full response:
HTTP/1.1 200 OK
Content-Length: 4
Content-Type: text/html
Server: Microsoft-HTTPAPI/2.0
Date: Fri, 15 Apr 2011 12:58:49 GMT
meow
Don't forget about the same origin policy restriction. Unless your javascript is hosted on http://localhost:8080 you won't to be able to send AJAX requests to this URL. A different port number is not allowed either. You will need to host your javascript file on an HTML page served from http://localhost:8080 if you want this to work. Or have your server send JSONP but this works only with GET requests.
Remark: make sure you properly dispose disposable resource on your server by wrapping them in using statements or your server might start leaking network connection handles.
Don't forget to release the resources by closing the response.
Calling Close on the response will force the response to be sent through the underlying socket and will then Dispose all of its disposable objects.
In your example, the Close method is only called on the Output stream. This will send the response through the socket, but will not dispose any resources related to the response, which includes the output stream you referenced.
// Complete async GetContext and reference required objects
HttpListenerContext Context = Listener.EndGetContext(Result);
HttpListenerRequest Request = Context.Request;
HttpListenerResponse Response = Context.Response;
// Process the incoming request here
// Complete the request and release it's resources by call the Close method
Response.Close();
I do not see setting of content-type. Set the content-type to text/html.
response.ContentType = "text/html";
You can simplify the writing code a lot. Just use this:
// Construct a response.
string responseString = "<HTML><BODY> Hello world!</BODY></HTML>";
context.Response.Write(responseString);
No need for the OutputStream or most of that other code. If you do have a reason to use it, note that you actually should not close the OutputStream. When you use Resopnse.OutputStream you're retrieving a reference to it but you're not taking ownership. It's still owned by the Response object and will be closed properly when the Response is disposed at the end of the request.
Code exemple:
HttpWebRequest request =
(HttpWebRequest)HttpWebRequest.Create("http://some.existing.url");
request.Method = "POST";
request.ContentType = "text/xml";
Byte[] documentBytes = GetDocumentBytes ();
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(documentBytes, 0, documentBytes.Length);
requestStream.Flush();
requestStream.Close();
}
When I do request.GetRequestStream (), there's nothing to send in the request. From the name of the method, and the intellisense it shows ("Get System.IO.Stream to use to write request data"), nothing indicates that this line of code will connect to the distant server.
But it seems it does...
Can anyone explain to me what HttpWebRequest.GetRequestStream () exactly does ?
Thanks for your enlightenments.
Getting the request stream does not trigger the post, but closing the stream does. Post data is sent to the server in the following way:
A connection is opened to the host
Send request and headers
Write Post data
Wait for a response.
The act of flushing and closing the stream is the final step, and once the input stream is closed (i.e. the client has sent what it needs to the server), then the server can return a response.
You use GetRequestStream() to synchronously obtain a reference to the upload stream. It is only after you have finished writing to the stream that the actual request is send.
However, I would suggest that you use the BeginGetRequestStream method instead of GetRequestStream. BeginGetRequestStream performs asynchronously and don't lock the current thread while the stream is being obtained. You pass a callback and a context to the BeginGetRequestStream. In the callback, you can call EndGetRequestStream() to finally grab a reference and repeat the writing steps listed above (for synchronous behavior). Example:
context.Request.BeginGetRequestStream(new AsyncCallback(Foo), context);
public void Foo(IAsyncResult asyncResult)
{
Context context = (Context)asyncResult.AsyncState;
try
{
HttpWebRequest request = context.Request;
using (var requestStream = request.EndGetRequestStream(asyncResult))
using (var writer = new StreamWriter(requestStream))
{
// write to the request stream
}
request.BeginGetResponse(new AsyncCallback(ProcessResponse), context);
}
Be very careful with BeginGetRequestStream. It never times out, thus you must add additional logic to your program to recover from situations where GetRequestStream will throw a timeout exception.
In general, threads are cheap. The async Begin/End methods of HttpWebRequest are only worth using if you will have 10,000 or more concurrent requests; because implementing timeouts is very tricky and error-prone. In general, using BeginGetRequestStream is premature optimization unless you need significant performance gains.
I am trying to stream dynamically generated data to a client over HTTP using IIS, and the connection has to remain open for a long period of time, and the server will send periodic status updates to the client while it is performing a time-consuming operation.
This MUST all be handled within ONE request, but I am using a WebClient.OpenRead() stream, which cannot be opened until the headers are sent.
How can I force IIS to send headers to the client, and later send a response body?
This behaviour is normally achievable by setting KeepAlive to true and setting Expect header to "100 and continue". By doing this, server will send the headers with result code 100.
I am not sure if this is possible using WebClient.
Use HttpWebRequest instead to be able to set the values above. In fact WebClient does nothing magical but using GET to get the data. Here is the code for calling OpenRead in Reflector:
try
{
request = this.m_WebRequest = this.GetWebRequest(this.GetUri(address));
Stream responseStream = (this.m_WebResponse = this.GetWebResponse(request)).GetResponseStream();
if (Logging.On)
{
Logging.Exit(Logging.Web, this, "OpenRead", responseStream);
}
stream2 = responseStream;
}
catch (Exception exception)
{
//