While nearly completing a new release, we've ignored the large size of the XML data that our WCF service returns to our silverlight client. Now we're investigating how to shrink the data, so that the results aren't in the 10-100mb range.
Its seems clear that binary serialization is the solution, and it seems easy enough to serialize the data into binary with, for instance, SharpSerializer, but through all of the SO posts about binary serialization and other tutorials I've come across, no one addresses how to send the serialized data across the wire to the Client. I expect I'm missing some obvious but critical piece to the WCF service puzzle.
Hopefully someone can lend me some help. Let me know if I should include more information.
First, try the built-in binary encoding (<binaryMessageEncoding> in config, see http://www.mostlydevelopers.com/blog/post/2009/10/14/Silverlight-3-WCF-Binary-Message-Encoding.aspx and http://www.silverlight.net/learn/data-networking/network-services-(soap,-rest-and-more)/how-do-i-use-binary-encoding-for-wcf-with-silverlight-3 ).
Your data will probably shrink, but please note that the built-in binary encoding was designed to be as fast as possible, not as small as possible.
If that's not enough and you want to use a 3rd=party component to do the serialization to binary data, you can indeed return this data as a byte[] (but you will also need to use <binaryMessageEncoding> above to prevent WCF from base64-encoding the data to make it valid XML). You can also use Stream instead of byte[], this won't give you true streaming behavior on the Silverlight client side but can give you true streaming on the server side.
Related
Should data manipulation once data is returned either with TCP or HTTP be received as byte arrays or is it an O.K. practice to receive it as a string? I've been trying to find some professional projects on github to get my answer, but have had no luck. Some examples of HTTPClient from Microsoft on MSDN usually make use of the GetByteArrayAsync(website) method, instead of GetStringAsync(website). Is there any reason why they would use GetByteArrayAsync instead of GetStringAsync, which would make data manipulation much easier right off the bat? Are there any advantages to using GetByteArrayAsync first instead?
What moves "through the wire" are bytes, not strings.
They might be text, but can be pictures, or a zip file.
At TCP/HTTP level this is unknown, and it does not matter.
That decision belongs with a higher level.
HTTP has a bit more info than TCP, so you might have a mimetype to help you decide what those bytes are.
Even if you know it is some kind of text, you will need to know the character set. You might get that info in the HTTP header, or in the document itself, or there might be a standard saying what the encoding is.
Only then you will be able to convert to a string.
First of all, what I do at the moment:
I sniff a asyncron serial bus with 9 bit protocol and send the data to the PC. At the PC side I receive the data as an endless string, that looks like that: .12_80E886.02_80E894.13. The Software of the PC-side is written with winforms with C#. Now I have the problem that I haven´t a clearly start you can see it in the stream example. The reason for that is, that I start the sniff somewhere in the protocol.
What I want to do:
I think I can use startindex = IndexOf("_"), and set them now as new start. I have to evaluate sign´s in the stream the stream is build: _(timestamp in milliseconds).(addressbyte databyte). The only what I want to display in my RichTextBox is the databyte, also I need a data management method for the timestamp. Because I have in the GUI the function that I can see the time beetween two or more databyte´s, for that I think I make a sql database. The addressbyte need I to collor the byte with an one as address in a special collor.
Question:
How can I evaluate the stream so that i have alternately timestamp,
addressbyte and than databyte as single substring?
The reason why I want them so, is that, I think I can make an easy if elseif else block to realize all what I want to do.
When someone has an better suggestion for my project pls write it as comment.
With friendly wishes sniffi
I think you're trying to solve two problems at the same time. It would be better to separate them and solve them individually.
There is the issue of transporting the data, for this you are using streams. That is a valid solution. There is sending and receiving the data (bits) over the stream.
You have the problem of transforming these bits (after receiving them) into actual objects (dates, strings, etc..). For that you an use a simple parser, tokenizer, a local script that can get the correct parts from the data and convert it, or you can use a serialization framework (like DataContracts).
If you have simple data, I would opt for using a single method that can parse the data. For more complex scenarios I would look into serialization.
Also be ware that you will need to validate your inputs, since you cannot assume that there is always a trusted (non compromised) piece of software that is sending the bits to you.
I think string is bad choice. Propably data is send as bytes. Sniff rather bytes than string. And you need protocol description to understand data.
You need to read bytes form bus and interpret it.
I'm just starting out really with WCF and Web Services in general. I have a pretty firm understanding on the purposes behind them and how they work, but I was wondering what the capabilities with them are if you wished to return something other than Text; such as a straight HTML form, or an image.
I've done some simple googling but alas all I can find is how to handle data passed from a form, rather than how to return a form.
I hope someone could give me a good starting point on what I should be looking at. I looked at a thread stating to look at Streaming with WCF but that may seem a bit excessive and was wondering if someone could give me some general advice and input.
Many thanks,
Ronald.
WCF services can return any object that the runtime can seralize. We return custom objects in our services with no issues, provided it's a .NET Client consuming them. Other languages may have to work harder to de-serialize complex objects.
(Meaning you have to write more code because non-Visual Studio IDE's probably won't know how to auto-generate the required client code.)
It probably depends on the actual binding but, for the sake of simplicity, assume that you bind your WCF service over http. Then, everything you pass to and from the service should somehow be translated to string. Simple types, ints, doules, strings, are easily convertible. Compound types - also, as they consist of simple types. When it comes to specific types like images or html forms, you always try to find a way to convert them at one side and convert back on the other side. In many cases the serializer can do it for you, for example if you return byte[], the data will be encoded as base64 string. If the serializer fails for some reason, you have to find your own way to pass your specific types. Please also remember that for WCF, it is you to select a particular serializer:
http://nirajrules.wordpress.com/2009/08/26/wcf-serializers-xmlserializer-vs-datacontratserializer-vs-netdatacontractserializer/
WCF is designed to build web API using standard or custom protocols. If you use the default configuration, WCF will output objects serialized using SOAP, but JSON is available too, for instance.
WCF is certainly able to output plain HTML, but it isn't designed with this goal in mind. It is meant to be used for communication between processes.
I am looking for the protobuf-net equivalent to the C++ API Message::ByteSize to find out the serialized message length in bytes.
I haven't played with the C++ API, so you'll have to give me a bit more context / information. What does this method do? Perhaps a sample usage?
If you are consuming data from a stream, there are "WithLengthPrefix" versions to automate limiting to discreet messages, or I believe the method to just read the next length from the stream is on the public API.
If you want to get a length in place of serializing, then currently I suspect the easiest option might be to serialize to a dummy stream and track the length. Oddly enough, an early version of protobuf-net did have "get the length without doing the work" methods, but after discussion on the protobuf-net I removed these. The data serialized is still tracked, obviously. However, because the API is different than the binary data length for objects is not available "for free".
If you clarify what the use-case is, I'm sure we can make it easily available (if it isn't already).
Re the comment; that is what I suspected. Because protobuf-net defers the binary translation to the last moment (because it is dealing with regular .NET types, not some self-generated code) there is no automatic way of getting this value without doing the work. I could add a mechanism to let you get this value by writing to Stream.Null? but if you need the data anyway you might benefit from just writing to MemoryStream and checking the .Length in advance of copying the data.
I have a "simple" task. I have an existing project with a web service written in C# which has a method that will send a huge XML file to the client. (This is a backup file of data stored on the server that needs to be sent somewhere else.) This service also had some additional authentication/authorization set up.
And I have an existing Delphi 2007 application for WIN32 which calls the web service to extract the XML data for further processing. It's a legacy system that runs without a .NET installation.
Only problem: the XML file is huge (at least 5 MB) and needs to be sent as a whole. Due to system requirements I cannot just split this up into multiple parts. And I'm not allowed to make major changes to either the C# or the Delphi code. (I can only change the method call on both client and server.) And I'm not allowed to spend more than 8 (work) hours to come up with a better solution or else things will just stay unchanged.
The modification I want to add is to compress the XML data (which reduces it to about 100 KB) and then send it to the client as a binary stream. The Delphi code should then accept this incoming stream and de compress the XML data again. Now, with a minimum of changes to the existing code, how should this be done?
(And yes, I wrote the original client and server in the past and it was never meant to send that much data at once. Unfortunately, the developer who took it over from me had other ideas, made several dumb changes, did more damage and left the company before my steel-tipped boot could connect to his behind so now I need to fix a few things. Fixing this web service has a very low priority compared to the other damage that needs to be restored.)
The server code is based on legacy ASMX stuff, the client code is the result of the Delphi SOAP import with some additional modifications.
The XML is a daily update for the 3000+ users which happens to be huge in it's current design. We're working on this but that takes time. There are more important items that need to be fixed first, but as I said, there's a small amount of time available to fix this problem quickly.
This sounds like a good candidate for an HttpHandler
My good links are on my work computer (I'll add them when I get to work), but you can look to see if it will be a good fit.
-- edit --
Here are the links...
http://www.ddj.com/windows/184416694
http://visualstudiomagazine.com/articles/2006/08/01/create-dedicated-service-handlers.aspx?sc_lang=en&sc_mode=edit
What is the problem with a 5MB file in a soap message? I have written a document server that runs over soap and this server has no problem with large files.
If the size is a problem for you I would just compress and decompress the xml data. This can easily be done with one of the many (free) available components for compression of a TStream descendant.
If you get that kind of compression, merely convert each byte to its hex equivalent, which will only double the size, and send this. Then do the opposite on the other end. Or am I missing something?
I would agree with Brad Bruce, HttpHandler would be fast, and using GZIP or Deflate Compression with I might be wrong... browsers support natively. you can get easy great compression on text based data for cheap cpu time.
System.IO.Compression.GZipStream GZipStream = new System.IO.Compression.GZipStream("Your XML Doc Stream",System.IO.Compression.CompressionMode.Compress)