I'm looking into using NServiceBus for fault tolerant communications between a central server and many remote located PCs.
I'm running the GateWay sample in the newest (3.2) release, and all works well - with a trial commercial license the performance seems great, sending to 3 remote PCs. But - to test the fault tolerance, if I disconnect one of the PCs, I notice that although the other sites continue to receive messages from the server, the performance suffers greatly - the site can be waiting up to 30 seconds to recieve a message - presumably because the server is busy dealing with retries for the site that is disconnected.
Is there a simple configuration-type answer to this?
I think the problem here was that I hadn't specified the number of worker threads on the server. I changed this to 5 worker threads, and the issue has now gone away.
Related
I have messages that I want to send from multiple WPF client applications to a service that can be processed some time after being sent.
Because of expected intermittent connectivity issues between client and server and necessary down time for the service, I'm inclined to create a WCF service with a queued endpoint. This has worked well for me in the past when the client machines were actually other servers and few in number.
I'm concerned about doing this with many client machines primarily because I think it will be difficult to monitor so many outgoing queues to confirm that no traffic is being trapped on the client machines.
Has anyone tried doing this before?
If so, would you recommend it? Why or why not?
Even if you haven't done it, can you think of other pitfalls beside the operational issue of monitoring all those outbound queues?
Your question may be better worded as:
Should a system be rolled out with many nodes all using MSMQ?
If so this is the essence of messaging and is what such systems are designed for irrespective of whether they are JMS, Apache MQ, Websphere, SonicMQ, or MSMQ.
Also, "traffic is being trapped on the client machines" - how do you define trapped? Remember, the application may be quite happy for the message to be sitting locally for days before being forwarded to the remote host. Messaging systems have timeouts generally for both reaching the destination and for the destination to process it.
I think you will be fine.
We have an application (a meta-search engine) that must make 50 - 250 outbound HTTP connections in response to a user action, frequently.
The way we do this is by creating a bunch of HttpWebRequests and running them asynchronously using Action.BeginInvoke. This obviously uses the ThreadPool to launch the web requests, which run synchronously on their own thread. Note that it is currently this way as this was originally a .NET 2.0 app and there was no TPL to speak of.
Using ETW (our event sources combined with the .NET framework and kernal ones) and NetMon is that while the thread pool can start 200 threads running our code in about 300ms (so, no threadpool exhaustion issues here), it takes up a variable amount of time, sometimes up to 10 - 15 seconds for the Windows kernel to make all the TCP connections that have been queued up.
This is very obvious in NetMon - you see around 60 - 100 TCP connections open (SYN) immediately (the number varies, but it's never more then around 120), then the rest trickle in over a period of time. It's as if the connections are being queued somewhere, but I don't know where and I don't know how to tune this to we can perform more concurrent outgoing connections. Perfmon Outbound Connection Queue stays at 0 but in the Connections Established counter you can see an initial spike of connections then a gradual increase as the rest filter through.
It does appear that latency to the endpoints to which we are connecting play a part, as running the code close to the endpoints that it connects to doesn't show the problem as significantly.
I've taken comprehensive ETW traces but there is no decent documentation on many of the Microsoft providers, which would be a help I'm sure.
Any advice to work around this or advice on tuning windows for a large amount of outgoing connections would be great. The platform is Win7 (dev) and Win2k8R2 (prod).
It looks like slow DNS queries are the culprit here. Looking at the ETW provider "Microsoft-Windows-Networking-Correlation", I can trace the network call from inception to connection and note that many connections are taking > 1 second at the DNS resolver (Microsoft-Windows-RPC).
It appears our local DNS server is slow/can't handle the load we are throwing at it and isn't caching aggressively. Production wasn't showing as severe symptoms as the prod DNS servers do everything right.
I am required to create a high performance application where I will be getting 500 socket messages from my socket clients simultaneously. Based on my logs i could see that my dual core system is processing 80 messages at a time.
I am using Async sockets (.BeginRecieve) and i have set NoDelay to true
From the logs from my clients and my server i could see that the message i wrote from my client is read by my server after 3-4 sec.
My service time of my application should be lot lesser.
First, you should post your current code so any potential bugs can be identified.
Second, if you're on .NET 3.5, you might want to look at the SocketAsyncEventArgs enhancements.
Start looking at your resource usages:
CPU usage - both on the overall system, as well as your specific process.
Memory usage - same as above.
Networking statistics.
Once you identify where the bottleneck is, both the community and yourself will have an easier time looking at what to focus on next in improving the performance.
A review of your code may also be necessary - but this may be more appropriate for https://codereview.stackexchange.com/.
When you do a socket.listen, what is your backlog set to? I can't speak to .net 4.0, but with 2.0 I have seen a problem where once your backlog is filled up (too many connection attempts too fast) then some of the sockets will get a TCP accept and then a TCP Reset. The Client then may or may not attempt to reconnect later again. This causes a connection bottleneck rather than a data throughput or a processing bottleneck.
I am in the initial phase of investigating a message queueing solution for C# and I'd appreciate any experience, lessons learned, war stories, etc. I also have a couple of specific questions about MSMQ's suitability for our configuration.
Briefly, we have a distributed architecture: a variety of server-side events generate work items that are retrieved by our deployed clients. Those clients connect to the server, retrieve the work, and process it, then go back to "waiting for work to do."
A couple of relevant details about these clients:
We have a few hundreds of client installations today, and need to be ready for growth to tens of thousands in the next 18 months.
The clients never submit tasks to the server -- they are "pull only."
Payloads to the clients are <= 1K each.
We don't need heavy-weight authentication nor encryption of the traffic (though that's a fine bonus)
Our clients run on a variety of MS operating systems, >= WinXP-SP1. Some are parts of windows active directories, or windows domains, or ad hoc workgroups.
Mostly, the clients are idle. We want to efficiently "wait for work" then respond to the work as quickly as possible (i.e., we want clients to receive the work item ASAP after it's queued)
Occasionally, our clients disappear from the internet for a time: their machines are shut off for a day, or overnight, etc. We want work items to arrive when they're back online. Put another way, we do need message reliability.
We control all of the client and server code, but not the client environments (though our installers do install prerequisite software like .NET 3.5, if it's not there already)
So, given the above - will MSMQ work "naturally" for us? I've not found a clear answer to how (or if) MSMQ handles clients listening for messages when they aren't in the domain/active directory and when they are connecting over the internet. So far, my reading on MSMQ feels pretty "enterprise-centric" - is our non-enterprise requirement going to be a problem with MSMQ?
What other solutions have you used in the past in similar setups?
And, of course, what other questions should I be asking? ;-)
Thanks!
RabbitMQ sounds like the solution for you.
RabbitMQ
RabbitMQ .NET/WCF Library
If you are looking at a central server on the Internet with queues holding messages waiting for remote clients to read them then MSMQ is not the product for you (much as it pains me to say that).
MSMQ cannot pull messages over the Internet using HTTP.
You would have to open up port 135 and use the RPC protocol and that is not necessarily a great idea on the Internet.
Cheers
John Breakwell
I am using SOAP in C# .Net 3.5 to consume a web service, from a video game company. I am having lots of SOAP Exceptions with the error "Operation Timed Out"
While one process is timing out, others fly by with no problems. I would like to rule out a problem on my end, but I have no idea where to begin. My timeout is 5 minutes. For every 5,000 requests, maybe 500 fail.
Anyone have some advice for diagnosing web services failures? The web service owner will probably give no support to helping me on this, as it's a free service.
Thanks
I've had to do a lot of debugging connecting to a SOAP Service using PHP and timeouts are the worst problem. Normally the problem is the 'client' doesn't have a high enough timeout and bombs after something like 30s.
I test making the calls using SoapUI. I keep using a higher client-side timeout using that until I find something that works. Once I find that out I use the newly found time to my client and re-test.
Your only solution may be to make sure your 'clients' have a high enough timeout that will work for everything. 5 minutes should be fine for most of your server-side timeouts.
OK this is a huge question and there is a lot that it could be.
Have you tackled HTTP two connection limit? http://madskristensen.net/post/Optimize-HTTP-requests-and-web-service-calls.aspx
Have you got enough IO threads to cater for the load? Use the performance monitoring to check this for your App Pool - I think there is a IO threads counter. A quick google turned this up - http://www.guidanceshare.com/wiki/ASP.NET_2.0_Performance_Guidelines_-_Threading
Are you exhausting your bandwidth? Use performance monitoring again to check the usage of your network card.
This is a really hard subject to broach textually, as it so dependent on environment but I hop these might help.
This also looks interesting - http://software.intel.com/en-us/articles/how-to-tune-the-machineconfig-file-on-the-aspnet-platform/