I have a need to override the receipt of a raw HTTP request as it is being communicated to an IIS server. I want to know if this is possible.
We have a client who sends huge Web Service calls (tens of Mb) and we want to start acting on portions of those calls as they are being received (in order to get a faster total time of execution for the Web Service call).
Currently, using normal web service methods, our application code is handed the Web Service call after it is totally received.
I realize this isn't the ideal way of handling Web Services, and we're not building our business on this, but we do have an need that we're trying to fill for a limited range of customers.
I have created a handler that implements IHttpHandler, but it appears that at this point in the process pipeline, the Request has been fully received by IIS (which doesn't get us any benefit over our current model). That is, I can read the InputStream directly, but the full request has already been transferred over the wire before I have access to this stream.
I think the answer is that I have to code an ISAPI filter to get this far down, but I don't have the skills to do this in C/C++. Does anyone know if there's another way I can do this without the ISAPI filter route?
An acceptable answer could be, "You have to do this as and ISAPI filter, to do it in C#, check this doc".
You can use a custom HttpModule to hook almost any part of the IIS pipeline. They work in both IIS 6 (under ASP.NET) and are the primary extension mechanism in IIS 7.
There are plenty of examples of building ISAPI filters, but none in C#. I am sure it is possible, but not practical and not without lots dirty tricks.
Your C# investment will hold up well in C++, let me know if you need help. By the way, I recommend you invest in the my standard trio - try to keep up a healthy knowledge of C#, C++ and Java.
I also recommend you consider Apache modules, they may offer more overall flexibility. This is what I would do:
host these web services off of IIS - you never know when IIS will bite you by resetting the application.
Use WCF services, host these from Windows services, use redirection to route the service to the WCF service.
Consider writting a raw sockets application. This one would implement the minimal WS:* protocol to your service and act as a proxy for the real service. When the proxy detects that the inbound message is exceeding a threshold it would begin analyzing the message to extract out what it could process right away.
The result would be standard WCF (through proxy) for smaller messages and non-standard processing for everything else.
Let me know if I can help you build it - this is the kind of thing I like to do...
Oh - and I recall now that WCF is completely configurable. You will be able to provide your own handlers for a variaty of layers and resolve everything from within managed code after all.
Short Version
The solution is to use HttpRequest.GetBufferlessInputStream.
Long Version
The issue is that if you attempt to use:
Request.InputStream
or Request.Form
or Request.Files
you must wait until the whole request has been received before it returns a Stream object. In contrast, the GetBufferlessInputStream method returns the Stream object immediately. You can use the method to begin processing the entity body before the complete contents of the body have been received.
This method can be useful if the request is uploading a large file and you want to begin accessing the file contents before the upload is finished. However, you should only use this method for scenarios where you want to take over all processing of the entity body. This means that you cannot use this method from an .aspx page, because by the time an .aspx page runs, the entity body has already been read.
The only downside, and it is a huge downside, is that you are now reading the Request.InputStream. This means you have to handle the MIME multiparts, and the base64 encoding, yourself.
Example (untested) code
UploadFile.ashx
public class Default : IHttpHandler
{
public void ProcessRequest (HttpContext context)
{
var request = context.Request;
var stm = request.GetBufferlessInputStream(true); //true --> disable web.config limits on request size
if (!stm.CanRead)
throw new Exception("Request input stream is not readable");
//Setup the buffer we'll be shuffling stream data into
int bufferLength = 16 * 8040; //use a multiple of 8040 bytes, because SQL Server uses pages of 8040 bytes. And because i'm saving it into SQL Server.
byte[] buffer = new Byte[bufferLength];
int bytesRead;
bytesRead = stm.Read(buffer, 0, buffer.Length);
while (bytesRead > 0)
{
SavePiece(buffer, bytesRead); //whatever you want to do with it
bytesRead = stm.Read(buffer, 0, buffer.Length);
}
}
private void SavePiece(byte[] buffer, int bufferLength)
{
//It's all going to be multipart mime encoded nonsense.
//Good luck!
}
public bool IsReusable { get { return false;}
}
Bonus Reading
How can I decode a multipart HTTP response?
Are there any multipart/form-data parser in C# - (NO ASP)
https://stackoverflow.com/a/21689347/12597 (example usage of the StreamContent class in .NET)
MSDN: StreamContent Class
Related
I have a pretty big video file I upload to a web service via multipart/form-data.
It takes ~ 30 seconds to arrive and I would prefer not waiting that long simply to access parameters I send along with the file.
My question is simple, can I access parameters sent with the form without waiting for the video payload to be uploaded?
Can this be done using headers or any other methods?
Streaming vs. Buffering
It's about how the webserver is set up. For IIS you can enable Streaming.
Otherwise, by default, IIS will use 'buffering' - the whole request is loaded into memory first (IIS's memory that you can't get to) before your app running in IIS can get it.
Not using IIS? You have to figure out how to get the webserver to do the same thing.
How to stream using IIS:
Streaming large file uploads to ASP.NET MVC
Note the way the file is read in the inner loop:
while ((cbRead = clientRequest.InputStream.Read(rgbBody, 0, rgbBody.Length)) > 0)
{
fileStream.Write(rgbBody, 0, cbRead);
}
Here instead of just saving the data like that question does, you will have to parse any xml/json/etc or whatever contains the file parameters you speak of ... and expect the video to be sent afterwards. You can process them right away if it's a quick process ... then get the rest of the video ... or you can send them to a background thread.
You probably won't be able to parse it just dumping what you have to a json or xml parser, there will be an unclosed tag or } at the top that isn't closed til after the video data is uploaded (however that is done). Or if it's multipart data from a form submission, as you imply, you will have to parse that partial upload yourself, instead of just asking IIS for the post data.
So this will be tricky, you can first start by writing 1k at a time to a log file with a time stamp to prove that you're getting the data as it comes. after that it's just a coding headache.
Getting this to work also means you'll have to have some control over the client and how it sends the data.
That's because you'll at least have to ensure it sends the file parameters FIRST!
Which concerns me, because, if you have control of the client, why can't you take the simple route (as Nobody and Nkosi imply) and use 2 requests? You mention you need one. Why not write js client code to send the parameters first in an XHR and then the file in a second request, using a correlation ID in both to tie them together? (the server could return this from the first request and you could send it in the 2nd).
Obviously, if you're just having a form with some inputs and a file upload and doing submit, then you need one request ;-) But if you have control over the client side you're not stuck with that.
Good luck, there is some advanced programming here, but nothing super high-tech. You will make it work!!!
If you don't have control over the server code, you are probably stuck, if the server app's webserver is buffering, the server app won't get anything, of course, if you wanted to do something with the file parameters first, this really implies you have control of the server side ;-)
I need to create a WCF Service that will have a download file function. This WCF will be consumed by a Delphi application.
The problem: The files that will be downloaded are very large and may cause memory problems on Delphi side. So far, I have this code:
[OperationContract]
byte[] DownloadFile(string filePath);
But this code will cause the client app to hold all data in memory which can be an issue.
I have read that WCF is capable of streaming data as you can read at: How to: Enable Streaming
But I have a question regarding this piece of code cut from MSDN:
[OperationContract]
Stream GetStream(string data);
On the client side I want to pass a TFileStream to the function. By using TFileStream every byte read will go directly to the disk. But the function RETURNS a stream and what I want will not be possible since the stream will not a parameter to the function.
How can I download a file from a WCF service directly to the disk?
I have found that relying on "built-in" streaming capability in WCF when working with other (non-.NET) clients is a big source for strange problems...
Basically we solve this kind of scenario by defining:
[OperationContract]
string DownloadFile(string filePath);
The method generates a HTTP(S) url and returns it...
This way any http-capable client can work with the data in a robust fashion...
BEWARE that this makes the server a bit more complicated since you now need to have some mechanism to generate (and serve HTTP GET on) URLs (security, "globally" unique, only usable for a limited time etc.).
BUT the big advantage is that any client out there (mobile or some strange embedded device or whatever you might encounter) will be able to implement this scheme as long as it has http-support available (Delphi has some very good http-client options).
First of all, I'm not sure whether you can consume a streaming WCF service at all in Delphi 2010. If you can, then it works as follows:
The WCF service must be a streamed service, which means that you need to set the transferMode of the binding to Streamed or StreamedResponse. If you want to pass in a string as parameter, it must be StreamedResponse, otherwise, the parameter must be a stream as well.
Having a streamed service also means that there can be no method that does not return a stream or void. It is, for example, not possible to have the following two methods in the same service when it is a streamed service.
Stream GetStream(string s);
int GetInteger(string s);
Also it is not possible to have:
Stream GetStream(string s);
in a service which is configured to be Streamed, as the parameter would have to be a stream, too.
It is not possible to call the method with a stream which will be "filled", even if you make the method take a Stream parameter - not the real instance of Stream is passed back and forth at that point, but the content is actually copied back and forth.
In Delphi you'd get a stream as a result of the method call. You can then copy the contents of that stream into a TFileStream as you'd do if the source was another stream in Delphi. Code for that can be googled. Basically Adriano has posted something that should work. Basically: Read from the source stream, write to the destination stream until everything was read and written, or you could try something like that:
stream1 := wcfServiceClient.GetTheStream();
try
stream2:= TFileStream.Create('to.txt', fmCreate);
try
stream2.CopyFrom(stream1, stream1.Size);
finally
stream2.Free;
end;
finally
stream1.Free;
end;
Again: This works only under the assumption that you can access a WCF streamed service from Delphi as you'd access it from C# or VB.NET.
Hihi all,
I am able to return stream from my WCF restful json webservice, everything works fine. But when I mixed the stream with another piece of data (both wrap into a custom class), upon consuming the webservice from my client, it gives an error message of "An existing connection was forcibly closed by the remote host".
Any advice how can I achieve the above? What it's required for my webservice is to allow downloading of a file with the file length as an additional piece of information for validation at the client end.
Thanks in advance! :)
There are various restrictions while using Stream in WCF service contracts - as per this MDSN link, only one (output) parameter or return value (of type stream) can be used while streaming.
In another MSDN documentation (this is anyway a good resource, if you want to stream large data using WCF), it has been hinted that one can combine stream and some input/output data by using Message Contract.
For example, see this blog post where author has used explicit message contract to upload both file name & file data. You have to do the similar thing from download perspective.
Finally, if nothing works then you can always push the file length as a custom (or standard such as content-length) HTTP header. If you are hosting in IIS then enable ASP.NET compatibility and use HttpContext.Current.Response to add your custom header.
I'm using a webservice which spits out very large amounts of data in one piece. The response string can be something like 8MB. While not an issue on a desktop PC, an embedded device goes nuts dealing with an 8MB string object.
I wonder if there is a way to get the response as a stream? Currently I'm using the method like below. I tried using a POST request instead, but SOAP is just more convenient (the response is XML and with the POST I have to convert the plain text reply back to valid XML) and I'd like to stick with it. Is it possible to use a different kind of "Invoke" which won't return strings but streams? Any ideas?
[System.Web.Services.Protocols.SoapDocumentMethodAttribute("MyAPI/MyMethod", RequestNamespace="MyAPI", ResponseNamespace="MyAPI", ParameterStyle=System.Web.Services.Protocols.SoapParameterStyle.Wrapped, Use=System.Web.Services.Description.SoapBindingUse.Literal)]
public string MyMethod(string sID)
{
object[] results = this.Invoke("MyMethod", new object[] { sID });
return ((string)(results[0]));
}
If you use the old ASMX web service client infrastructure, then you're stuck with its limitations. One limitation is that there's no simple way to get the response except as deserialized data.
If it were necessary, then you could use a partial class to override the GetWebResponse method to return your own custom WebResponse. This latter would in turn override the GetResponseStream method to call the base version, consume the stream, then to return a stream containing an "empty" web request (otherwise .NET will choke on a stream with no contents).
You might also try something similar by overriding the GetReaderForMessage method. This is passed a SoapClientMessage instance which has a Stream property that you might be able to use. Again, you'll have to set the stream to something that the web service infrastructure can consume.
The better way to do this is with a WCF client. WCF has much more powerful and easy to use extensibility mechanisms.
In fact, you might not even need to extend a WCF client. You might simply be able to configure it to not have this buffering problem at all.
Any web service call is going to return SOAP, isn't it? I don't think a stream could be serialized into a soap packet to be returned from your service. And even if it could, wouldn't the serialized stream be at least as big as the string itself?
I believe the answer is no, there is no concept of a stream for SOAP.
Probably the simplest answer is to have your method:
Parse your response into segments your mobile device can handle
Cache your response in a application variable as a dictionary of these segments
return an arraylist of GUIDs.
You can then have your client request each of these segments separately via their GUIDs, then reassemble the original response when and handle it all the web services return.
ASMX can't do much about this. WCF's BasicHttpBinding can return a Stream to the caller.
http://msdn.microsoft.com/en-us/library/ms733742.aspx
I am trying to create a webservice that would allow its consumers to download files (can be very huge files). At the server, I have many files that need to be sent back to the consumer, so I am compressing all those file into one big zip file and streaming them back to the user. Right now, my webservice will start compressing the files, when the request comes in, forms the zip files and streams it back.Sometimes compression can take a lot of time, and the request may time out. What can I do to avoid such situations? My solution right now is to, seperate the data into smaller zip files, send a response to consumer saying there would be these many smaller files, and let consumers send request for individual smaller files. So, if i have 1GB zip file, i will break it into 10 smaller zip files, and ask the consumer to request for smaller files in 10 requests. Is this the correct approach? What problems can I be facing? Has anyone dealth with such issues before? I would be glad if you can share your experiences. Also, is it possible to start streaming the zip files without forming them fully?
Treat the request and the delivery as asynchronous operations.
The client can make a request for the file using one method. Another method can let the client know the status of the file packaging (whether they are ready for download yet). A third method can actually download the files.
It may be worth looking at a restful approach. Rather than a soap web service. As OrbMan, suggested an asynch approach may be best.
With REST you could expose a resource as: http://yourlocation/generatefile
Which (when called with a post) returns a http response with a response code of 301 'accepted' and a location header value of location=http://yourlocation/generatefile/id00124 which suggests the location of the data.
You can then poll the http://yourlocation/generatefile/id00124 resource (maybe just header request) to get the status i.e. processing / complete.
When processing is complete. Do a get on the http://yourlocation/generatefile/id00124 to download your file. The response http message should identify you file and the format i.e. encryption and compression types so any consumer knows how to read it.
This is a nice solution to problems which are long running and returns data in formats other than soap anbd general xml.
I hope this helps
I would poll from the calling client as part of the method which gets the file. The client code might flow something like this:
byte[] GetFile()
{
response = request.Post(http://yourlocation/generatefile);
string dataResource = response.Headers["Location"];
bool resourceReady = false;
while(!reasourceReady)
{
resH = request.Header(dataResource);
if(resH.Headers[Status] == "complete")
break;
else
Thread.Sleep(OneSecond); ?? or whetever
}
fileRes = request.Get(dataResource);
return fileRes.ToByteArray();
}
This is only psuedo, but I hope it makes sense...