I have a "simple" task. I have an existing project with a web service written in C# which has a method that will send a huge XML file to the client. (This is a backup file of data stored on the server that needs to be sent somewhere else.) This service also had some additional authentication/authorization set up.
And I have an existing Delphi 2007 application for WIN32 which calls the web service to extract the XML data for further processing. It's a legacy system that runs without a .NET installation.
Only problem: the XML file is huge (at least 5 MB) and needs to be sent as a whole. Due to system requirements I cannot just split this up into multiple parts. And I'm not allowed to make major changes to either the C# or the Delphi code. (I can only change the method call on both client and server.) And I'm not allowed to spend more than 8 (work) hours to come up with a better solution or else things will just stay unchanged.
The modification I want to add is to compress the XML data (which reduces it to about 100 KB) and then send it to the client as a binary stream. The Delphi code should then accept this incoming stream and de compress the XML data again. Now, with a minimum of changes to the existing code, how should this be done?
(And yes, I wrote the original client and server in the past and it was never meant to send that much data at once. Unfortunately, the developer who took it over from me had other ideas, made several dumb changes, did more damage and left the company before my steel-tipped boot could connect to his behind so now I need to fix a few things. Fixing this web service has a very low priority compared to the other damage that needs to be restored.)
The server code is based on legacy ASMX stuff, the client code is the result of the Delphi SOAP import with some additional modifications.
The XML is a daily update for the 3000+ users which happens to be huge in it's current design. We're working on this but that takes time. There are more important items that need to be fixed first, but as I said, there's a small amount of time available to fix this problem quickly.
This sounds like a good candidate for an HttpHandler
My good links are on my work computer (I'll add them when I get to work), but you can look to see if it will be a good fit.
-- edit --
Here are the links...
http://www.ddj.com/windows/184416694
http://visualstudiomagazine.com/articles/2006/08/01/create-dedicated-service-handlers.aspx?sc_lang=en&sc_mode=edit
What is the problem with a 5MB file in a soap message? I have written a document server that runs over soap and this server has no problem with large files.
If the size is a problem for you I would just compress and decompress the xml data. This can easily be done with one of the many (free) available components for compression of a TStream descendant.
If you get that kind of compression, merely convert each byte to its hex equivalent, which will only double the size, and send this. Then do the opposite on the other end. Or am I missing something?
I would agree with Brad Bruce, HttpHandler would be fast, and using GZIP or Deflate Compression with I might be wrong... browsers support natively. you can get easy great compression on text based data for cheap cpu time.
System.IO.Compression.GZipStream GZipStream = new System.IO.Compression.GZipStream("Your XML Doc Stream",System.IO.Compression.CompressionMode.Compress)
Related
Background:
I have a PHP server hosting a RESTful API that returns an image when a function /api/GenerateImage is called. I need the fastest and lowest memory using method to achieve this.
Ideas:
I can think of a few ways of achieving this:
Write it entirely in PHP - Image requires a lot of byte manipulation and it feels bad to do this in PHP.
Write a continually running C# program and use an API to generate the image and copy the response stream to the client via PHP - Need to avoid overhead of unnecessary socket connections.
Write a C/C++ command line program that, when called, generates and stores the image in a file, which PHP reads and sends - Need to avoid overhead due to starting a program, writing to a file and then reading to it.
Write it in C/C++ and use a PHP library wrapper to call the function directly from PHP - Seems to be the fastest, but most difficult for a single function.
The two most obvious solutions to me seem absent from your set of options:
Use PHP's GD extension.
Use imagemagick.
Whether these are practical options depends on what exactly you have to do with the image. If they are just static images you don't need anything, just serve them up as files from your web server. The fact you need some code to generate the image implies that the image is dynamically generated.
If neither of these suit, then I would say: your last (4th) solution would be the fastest, but a modified version of your 3rd solution would be simplest: write a command-line program that returns the image data to stdout, and call that from PHP.
I am wondering how should I set up interprocess unified communication in better way than I do now. Client process sends a lot of messages of different sort to the server process. Messages like... I have done some work[what],I started at [time], ended at[time]. or state, progress even command messages.
example message: From:Process1;StartedAt|12:12:12;EndedAt|12:45:56;DoneUnit:51
Server parser split string by semicolon. From first part reads from who was message sent. from second and third part it reads times and from last how much work it did.
When I add another info at the end of message
ex. From:Process1;StartedAt|12:12:12;EndedAt|12:45:56;DoneUnit:51;Source:tableT
I have to rewrite the server parser as well.
Server tries to parse received message using my own parse function. Every message has its own format. So parser know how should message look. But if I change the format on client I have to change it on server as well. It does not seems to be very efficient way.
For that reason I ask you a question.
How should this communication get better or is there any different approach how to store the format for client and server on one place?
I use c# .net 3.5(Must be this version)
Thank you for reply
The obvious solution to your problem would be to not write the parsing code yourself.
If you create a class that can be serialized, you can send the serialized version of the class over the wire and deserialize at the other end. That means the message class can be shared between both applications, and the parsing code is trivial. Depending on your requirements, you can use various serializers: Xml or JSON would be verbose but human-readable, or the binary serializer would be more efficient in terms of bandwidth (but harder to debug or monitor over-the-wire).
While nearly completing a new release, we've ignored the large size of the XML data that our WCF service returns to our silverlight client. Now we're investigating how to shrink the data, so that the results aren't in the 10-100mb range.
Its seems clear that binary serialization is the solution, and it seems easy enough to serialize the data into binary with, for instance, SharpSerializer, but through all of the SO posts about binary serialization and other tutorials I've come across, no one addresses how to send the serialized data across the wire to the Client. I expect I'm missing some obvious but critical piece to the WCF service puzzle.
Hopefully someone can lend me some help. Let me know if I should include more information.
First, try the built-in binary encoding (<binaryMessageEncoding> in config, see http://www.mostlydevelopers.com/blog/post/2009/10/14/Silverlight-3-WCF-Binary-Message-Encoding.aspx and http://www.silverlight.net/learn/data-networking/network-services-(soap,-rest-and-more)/how-do-i-use-binary-encoding-for-wcf-with-silverlight-3 ).
Your data will probably shrink, but please note that the built-in binary encoding was designed to be as fast as possible, not as small as possible.
If that's not enough and you want to use a 3rd=party component to do the serialization to binary data, you can indeed return this data as a byte[] (but you will also need to use <binaryMessageEncoding> above to prevent WCF from base64-encoding the data to make it valid XML). You can also use Stream instead of byte[], this won't give you true streaming behavior on the Silverlight client side but can give you true streaming on the server side.
I have one question about wcf - I want to get large amount of data using wcf service, but I don't want to change config file.
For example - if data is over buffersize or maxReceivedMessageSize I want to split it into packs and send it one by one. On Client side I want to get all of packs and recreate final response.
It is possible?
I don't think there is built-in support for that in WCF. You will have to come up with you own way of splitting data on the server side and combining it on the client side. And you will have to make multiple calls to transfer all data. It's probably a good idea anyway since you might want to show progress bar or something if you transfer a lot of data.
UPDATE:
After some googling I found this. Essentially you can create you custom channel that would do chunking for you. Still, I would probably implement chunking on the application level rather than channel level.
I was curious as to how I would accomplish the following with webservices:
Authenticate a user.
Accept a CSV or XML file.
Process the file and put it into an SQL database.
Someone mentioned in a previous post that I should use a webservice. I can't seem to find any resources that explain how to begin something like this. All the simple examples seem to just show how you can serve XML given a query.
I want to know how to accept stuff and also, how this would differ from an upload control on an authenticated webpage. I don't think I really understand webservices and their benefits.
How would the user sending the XML file interface with my webservice?
If you want to do large file uploads, then a web service may cause some issues, because some web service platforms (including .NET) have default settings limiting the size of the data.
The advantage of a web service is that it does all the mapping of the request to/from XML, so you can return a .NET type, and don't need to muck around with processing request parameters.
However, you may have to put more effort into maintaining state, etc.
For logins, what you can do is have a login function that returns some kind of identifier which can be used to verify the user as valid for that session - one way of doing this being to have columns in your user table for lastActive and sessionGUID, and when they log in you generate a new sessionGUID and return that, and on that and any other valid request they make you update the lastActive, and if there is a request too long after the lastActive time, then you refuse the request... there's any number of similar ways of doing that, but hopefully you get the general idea - you don't want to require the login details each time, but you can generate a temporary identifier and use that.
For accepting an XML file, you'd want to use something like XDocument or XMLReader to read the data that you receive. Assuming you're not talking about the parsing of the XML format that the web service itself uses, you're most likely to be receiving a string and then pushing that into an XDocument and then using the standard XDocument functions to process the data. If the document would be large, then XMLReader should be more efficient.
For reading a CSV file, there are some (free and non-free) CSV readers which help avoid some of the issues you can have, giving you a nice API for processing a string or strings of CSV data. If you know that the source data doesn't have non-structural commas, though, you can just take the string and split it by commas, and then strip any quotes around the values. That tends to get flaky quite fast if there might be addresses or other data that could have commas in, though.
The XML should be able to be passed via the web service just fine - it should be encoded and decoded, so it's then compliant strings being passed out.
As for storing it in a database, there's any number of ways to do that - you can use ADO.NET to store things in a database without further libraries, you can create a database structure in Visual Studio or SQL Server Management Studio and then use SQLMetal or Linq to SQL to generate classes for saving the data, you can use a 3rd party database mapping tool (such as Castle ActiveRecord), or whatever. It depends what you know and how much you're willing to learn. That's really separate to the web service. When you define a web service in .NET you effectively define standard functions with attributes marking them as web services, so the database side is standard .NET database stuff that's not necessarily any different to what you'd do for an ASP.NET website, or even a desktop program.
A web service is not really appropriate for sending an arbitrary file. It can be done, but if that's your only reason for creating the web service, you might as well just stick to HTTP.
If the file has a specific format or specific contents then you might want to create a web service for that. The purpose of an ASMX or WCF web service is to provide discoverability and strong typing to the data (among other things, but I'm sticking to the basics for the moment). From the perspective of the client, instead of trying to create some ugly XML or CSV blob and chuck it over HTTP, you use an actual service proxy with POCO classes:
MyService service = new MyService();
MyData data = new MyData() { ID = 3, Name = "Test", Date = DateTime.Now };
service.Save(data);
Visual Studio (and equivalent tools in Java and some other platforms) will take care of generating the proxy for you, so really all you have to do is write the above code.
But if you're just trying to send any data, this won't get you anywhere, because you can't generate a proxy for raw XML. Well, you can, but it would just be an XmlDocument and that accomplishes nothing in terms of usability, type safety or discoverability.
Don't get confused by the "XML" in "XML Web Service". It's not a tool for sending around vanilla XML. Rather, XML refers to the format of the message, as it is transmitted over the wire, as opposed to a POST string (id=3&name=Test&date=2010-01-24) or a binary RPC call as used in .NET Remoting.
In terms of authentication, if you do decide to use WCF, you just have to use the right binding. A WCF proxy is normally configured by default to use wsHttpBinding, which uses integrated Windows authentication to secure the messages. Again, assuming you use Visual Studio, this is all done pretty much automatically for you unless you decide to change the defaults.