I want to sending by webservice/ WCF big files like 2gb psd.
Is WCF message streaming the best way to cope with this ??
Odds are, a client-server design which exchanges huge amounts of data like 2GB files indicates a problem with the design. Consider these alternatives:
Don't send 2GB across the wire, you'll tie up the client during the upload, you might lose the file in transit, etc etc etc. Maybe send a URL to your service instead, so the service can download the file and handle any problems it encounters on the server side.
For huge amounts of data, client-server might be a totally inappropriate way to process your data. You might be better moving processing to the client side instead of the server.
I would use something like msmq transport if you want to send something that large via WCF, that way you can ensure delivery.
Related
We experience a lack of performance during iteration across remote private MSMQ queue. We tried to use both API methods - MessageQueue.GetAllMessages() and MessageQueue.GetEnumerator2() and see the same results.
It seems that the problem is in Message Queuing Service, because it always uses only up to 15% of CPU (single core). For example, if we iterate across local queue - we use 100% of CPU and can load 1 million messages in 2 seconds, but for remote queues it takes 30 seconds to load only 10K! Network connection is 100MBPs.
Is there a way to increase MSMQ performance for remote queues and force it to use 100% of CPU or Network?
MSMQ is optimised to go as fast as it can - it's not going slow just to irritate you.
Performance will be poor on remote queues. This is not the best way to use MSMQ. High performance is obtained through the "send remote, read local" model.
Remote access uses RPC which will be slow over a LAN. If you looked at a network trace, you would see all the back-and-forth communication. Binding to the remote RPC service and querying to find where MSMQ is listening; binding to the remote MSMQ RPC listener; requesting messages from the listener; etc etc.
This may or may not be relevant in your scenario, but it's a way to improve overall performance for MSMQ.
If you're sending messages that wrap a consistent type - a serialized class, for example - buffer them before sending and send one message containing an array or collection of items.
I was working with some serialized class and sending a large volume of messages. I tested and found that if I sent them in batches of 50 instead of individually then the size of queue was reduced by 75%. I didn't spend much time optimizing from there. It depends on the size of your messages. But this gets rid of much of the overhead incurred in sending individual messages.
Try using the TCP connection syntax and use an explicit numerical IP address 123.123.123.123. See if this affects your performance. If it does then think security.
You use the terms GetMessage but also talk about loading so I am confused about if you want performance on Message Receive “GetOne” or Load into the queue operations.
For core production code I always operate one at a time on the messages so I am never trying to GetAllMessages or EnumerateAllMessages except in specific management functions.
I need to transfer 1 GB using a web service. I think to transfer piecewise using msmq. Maybe there is a way to take it easy?
If you CAN break the data up in smaller chunks, then do. Web services aren't designed to transport that much data in one go, so even though it's possible, it's gonna be a bumpy ride.
But the world doesn't work in an efficient way, so here's what you do:
write the data as binary to a local file.
2. Create a streamwriter that writes to your webservice using a streamreader to read from the file.
3. If anything happens, catch the exception and try to resume from where your file pointer is.
4. If you can modify the webservice, have it read the data and write to a binary file, catching any errors and trying to write any new data on resume to the file at the current pointer.
The trick is going to be to figure out how to tell the service you're trying to resume an interrupted request.
If this isn't clear, I'll try to expand some more.
I need to transfer 1 GB using a web service. I think to transfer piecewise using msmq.
I want to transport people with a car. I Think of using a plane.
Get it? Either web service, or MSMQ. They do not magically mix.
THAT SAID: Web service, alrge data = bad idea. Even JSON has overhead. STreaming, non streaming? That is a LOT of open variables, and in most cases the web service here makes relatively little sense.
Up (sent to service) or down (to the service)? More questions - I would not really want a 1gb upload to a web service.
If you have to, splice the data and make an api to ask for all "parts" and then get part by part - that also allows a progress bar to be shown. Your software MUST handle re-requests for parts due to failures which MAY happen in transit.
I would seriously consider not using a web service here if the data is binary and just go with a REST api, at least for downloads. Likely also for uploads. Lots depends on all the stuff you did not even know how to ask for or did not bother to describe.
You can make some service to first creat buffer in destination next split data and send it through service, then finalize it.
Currently I'm having a NetTcpBinding with the default Buffered Transport Mode. I'm trying to determine what the best solution is for my request that builds a class that ends up being a large amount of data.
Basically I have a request that comes into WCF to grab a bunch of reporting information and return it to the client. As long as this is under the 64k MaxMessageRecievedSize it is fine, but if it goes over then I end up having an exception.
Should I switch to a Streaming TransferMode and then stream a file back to the client instead since the data could be small or large amounts? Or is it ok to increase the MaxMessageRecievedSize? It seems like a bad idea to increase the MaxMessageRecievedSize because if we have multiple connections to the service all fetching large amounts of data we could potentially have memory issues?
Any thoughts on how I could go about achieving this?
For the sizes you mention, you should switch to streaming. There's a detailed document that discusses this at http://msdn.microsoft.com/en-us/library/ms733742.aspx
I am working on a project where i need to connect with multiple clients and every client is streaming live screen capturing to server. Server show that.
What would be the best approach for that.
Thank You
You can use WCF in streaming mode for the video, but I doubt it is a good solution.
I think that going for pure sockets is better, to get the performance required. Showing a live video stream is also not really a limited operation (which is what WCF is built for), but rather something ongoing.
My suggestiion is to:
Use a pure TCP socket for the video stream for a start.
If that gives problems, you can switch to UDP. It is better to skip over any lost packages for live video, but with UDP you have to track package ordering etc. yourself.
If you need control operations, use a separate WCF service for that.
I'm thinking like the methods games like Counter Sstrike, WoW etc uses. In CS you often have just like 50 ping, is there any way to send information to an online MySQL database at that speed?
Currently I'm using an online PHP script which my program requests, but this is really slow, because the program first has to send headers and post-information to it, and then retrieve the result as an ordinary webpage.
There really have to be any easier, faster way of doing this? I've heard about TCP/IP, is this what I should use here? Is it possible for it to connect to the database in a faster way than indirectly via the PHP script?
TCP/IP is made up of three protocols:
TCP
UDP
ICMP
ICMP is what you are using when you ping another computer on a network.
Games, like CounterStrike, don't care about what you previously did. So there's no requirement for completeness, to be able to reconstruct what you did (which is why competitors have to tape what they are doing). This is what UDP is used for - there's no guarantee that data is delivered or received. Which is why lag can be such a problem - you're already dead, you just didn't know it.
TCP guarantees that data is sent and received. Slower than UDP.
There are numerous things to be aware of to have a fast connection - less hops, etc.
Client-to-server for latency-critical stuff? Use non-blocking UDP.
For reliable stuff that can be a little slower, if you use TCP make sure you do so in a non-blocking fashion (select(), non-blocking send, etc.).
The big reason to use UDP is if you have time-sensitive data - if the position of a critter gets dropped, you're better off ignoring it and sending the next position packet rather than re-sending the last one.
And I don't think any high-performance game has each and every call resolve to a call to the database. It's more common to (if a database is even used) persist data occasionally, or at important events.
You're not going to implement Counterstrike or anything similar on top of http.
Most games like the ones you cite use UDP for this (one of the TCP/IP suite of protocols.) UDP is chosen over TCP for this application since it's lighter weight allowing for better performance and TCP's reliability features aren't necessary.
Keep in mind though, those games have standalone clients and servers usually written in C or C++. If your application is browser-based and you're trying to do this over HTTP then use a long-lived connection and strip back the headers as much as possible, including cookies. The Tornado framework may be of interest to you there. You may also want to look into HTML5 WebSockets however widespread support is still a fair way off.
If you are targeting a browser-based plugin like Flash, Java, SilverLight then you may be able to use UDP but I don't know enough about those platforms to confirm.
Edit:
Also worth mentioning: once your networking code and protocol is sufficiently optimized there are still things you can do to improve the experience for players with high pings.