We Keep Coding

Best practice for application monitoring

I am trying to create a monitoring application for our operations department to be proactive when dealing with systems that are encountering problems. I created an app that does the job but it has some draw backs:
Each copy of the app running serves individual pings to the systems, when 1 ping would suffice.
I have 3 different api's for getting the status of our systems depending whether its hosted IIS, WCF or desktop.
To fix the first issue i was going to create a database which an interim service(app)(monitor) would make the pings, then the app would query the database for updates. After thinking about this I realized the second issue and decided it is a future problem.
So my thought was to, rather than have the interim application pinging the systems, simply have each system have one interface in which it posts it status to the database every x time. But then I ran into a problem with the WCF and IIS services we have. These services can sit for days without anyone actually using the service. How would I make these services continue to post its data?
My questions are:
Is it better to have data REQUESTED or PUSHED in this type of situation?
If REQUESTED, what is a suggested practice for maintaining a single API across mulitple platforms(IIS, WCF, Desktop)?
If PUSHED, how would you handle the case of the Web services which are instance based and not continuously running?

For web services, one solution might be to implement a health-check end point , something that you can simply call like: webservice/isServiceUp?
I prefer that this information is PULLED. If a service / web Service/ Application is down, then you can't possibly rely on it to write something to the DB... it would be possible but highly risky and unreliable.
In a real world situation, it is a little more complicated than that because something might happen between your service host and the consumer (DNS problem for example), in which case, you would want to consider the case of not getting anything back from the isServiceUp (no true no false, just a 400 lvl error)...
Consider using your load balancer for checking on APPS / web services and proactively switching to a different IP in case of issues... it is a possibility.

WCF Scalability with Session

we are evaluating a new project which will have a .NET Server which is available in the internet. We have access to the server but the hosting is done by a 3rd party company.
We are evaluating using WCF on the .NET Server. (I have no professional experience with WCF and just reading into the topic). The WCF service will talk to a SQL Server to perform its duties.
Here is the scenario:
Multiple client machines running our own ActionScript software will connect to that .NET Server.
Clients might be online 24/7 and should periodically poll our server to tell the server that they are there.
A client needs to be able to login, and only if the login has worked the other calls will be allowed and at some point it logs out. So we need to "remember" the state with a particual client...
Highest expected load is around 1000 Clients, of which 500 will only do polling while the other 500 will be "active". "Active" means a maximum of 1 call each minute, no heavy payload in each call, neither in the request nor in the response, just 1-3 database accesses per call.
We already tested some "HelloWorld" with ActionScript and WCF using BasicHttp(s)Binding.
But because we need session handling we were thinking about taking using the wsHttpBinding binding because it can provide us WCF Sessions.
So far so good, but then I stumbled upon the fact that it should
However:
I find that in my Oreilly WCF Services 3rd edition book (Page 177) it is written
and even Microsoft is writing to be careful using that:
http://msdn.microsoft.com/en-us/magazine/cc163590.aspx
"A service configured for private sessions cannot typically support more than a few dozen (or perhaps up to a few hundred) outstanding clients due to the cost associated with each such dedicated service instance."
So because we need to identify the state with each client, we could of course implement our own "Session Handling" on top of stateless HttpBindingBinding, and make a call to that SessionHandling class each time when my WCF methods get called, but I am reluctant to do anything like that, it looks to me like thousands of people should already have faced the same problem.
So, my question now is:
Do you think wsHttpBinding on my server could handle the payload?
How "bad" is it really to go with wsHttpBinding on WCF? Does anybody already have experience with this? Can I use it? What would you use?
Final Remarks:
I am not limited to WCF if we dont like it, we just shall do an evaluation.
From the companies point of view it would also be fine to go for a protobuf-RPC or XML-RPC solution over TCP and the ActionScript clients implementing that. (just examples!) So no need for hosting WCF in IIS on the server as long as the coding part is comfortable (enough) for the programmers on both sides and the ADMINISTRATION on the deployed server is not too much either. With just making some TCP-ports based communication I am a bit afraid what it would mean for the administration in regards to firewall and stuff. Payload is not an issue, client processing power is also not an issue. The only thing I am concerned about is scalability of the server and security.
Thanks in advance for any suggestions!

I would not be concerned with scalability. You can always add a server or two to your farm in case of issues.
I would rather be concerned with your architecture and the need to store anything in session - are you sure about that?
Note that you don't need ws binding to support sessions, basic binding supports sessions as well.

Azure: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full

We are migrating our website to azure and sometimes have an problem now with the following exception:
An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full
We developed a search engine for long-distance-buses in germany and make thousands of requests to external services per day:
MongoDB
Azure Caches
Azure Storages (Blob)
External Apis
I guess, it are about 250.000 calls per day. More or less all of time are fast an very short. We have a timeout for all external apis with 8sec or less to prevent hanging requests.
The external apis are mostly accessed with the following class: http://pastebin.com/evNUVMXp
It worked fine so far, but everything was running on one machine only. At the moment I have 2 vms for mongodb and a website for 3 large instances running.
Any ideas what I can do to solve the problem?
Thank you for your help:
Sebastian

Main problem, I had, was with the with CosmosDB client instances. It is recommended that it is used as a static within application.
So, I would suggest to introduce data client to be used as a Singleton.
So usage of it in application would be then like:
DataClientFactory<EntityData>.Instance.DocumentDbRepository
Where DataClientFactory, implements Singleton patteren, and DocumentDbRepository implements repository patteren around CosmosDB client API (or your data client, that you are using).
CosmosDB client (and as seems also some others) seems to be designed to be quite aggressive with parallel requests.

You can try to set
ServicePointManager.DefaultConnectionLimit
to a higher number like 100 for example and test that out.
Here is the documentation:
ServicePoint Manager Class [MSDN]

I believe the server that hosts the database(s) needs to increase the limit of dynamic ports it can allow.
The solution is described here.
Best of luck to you.

Disconnected Architecture With .NET

I'm working with an n-Tier application using WinForm and WCF
Engine Service (Windows Service) => WCF Service => Windows Form Client Application
The problem is that the WinForm Client Application need to be 100% available for work even if Engine Service is down.
So how can I make a disconnected architecture in order to make my winform application always available ?
Thanks.

Typically you implement a queue that's internal to your application.
The queue will forward the requests to the web service. In the event the web service is down, it stays queued. The queue mechanism should check every so often to see if the web service is alive, and when it is then forward everything it has stored up.
Alternatively, you can go direct to the web service, then simply post it to the queue in the event of initial failure. However, the queue will still need to check on the web service every so often.
EDIT:
Just to clarify, yes all of the business logic would need to be available client side. Otherwise you would need to provide a "verify" mechanism when the client connects back up.
However, this isn't a bad thing. As you should be placing the business logic in it's own assembly(ies) anyway.

Have a look at Smart Client Factory: http://msdn.microsoft.com/en-us/library/aa480482.aspx
Just to highlight the goals (this is sniped from the above link):
They have a rich user interface that
takes advantage of the power of the
Microsoft Windows desktop.
They connect to multiple back-end
systems to exchange data with them.
They present information coming from
multiple and diverse sources through
an integrated user interface, so the
data looks like it came from one
back-end system.
They take advantage of local storage
and processing resources to enable
operation during periods of no
network connectivity or intermittent
network connectivity.
They are easily deployed and
configured.
Edit
I'm going ansewr this with the usual CYA statement of it really depends. Let me give you some examples. Take an application which will watch the filesystem for files to be generated in any number of different formats (DB2, Flatfile, xml). The application will then import the files, displaying to the user a unified view of the document. And allow him to place e-commerce orders.
In this app, you could choose to detect the files zip them up and upload to the server do the transforms (applying business logic like normalization of data etc). But then what happens if the internet connection is down. Now the user has to wait for his connection before he can place his e-Commerce order.
A better solution would be to run the business rules in the client transforming the files. Now let's say, you had some business logic which would based on the order determine additional rules such as a salesman to route it to or pricing discounts...These might make sense to sit on the server.
The question you will need to ask is what functionality do I need to make my application function when the server is not there. Anything thing which falls within this category will need to be client side.
I've also never used Click Once deployment we had to roll our own updater which is a tale for another thread, but you should be able to send down updates preety easily. You could also code your business logic in an assembly, that you load from a URL, so while it runs client side it can be updated easily.

You can do all your processing off line, and use some thing like Microsoft Sync Framework to sync the data between the client and the server.
Assuming both server and client are .net, you can use same code base to do the data validation both on the server and the client. This way you will have a single code base that will serve both server and client.
You can use frameworks like CSLA.NET to simplify this validation process.

Communication between server and client for WinForms

I have 50+ kiosk style computers that I want to be able to get a status update, from a single computer, on demand as opposed to an interval. These computers are on a LAN in respect to the computer requesting the status.
I researched WCF however it looks like I'll need IIS installed and I would rather not install IIS on 50+ Windows XP boxes -- so I think that eliminates using a webservice unless it's possible to have a WinForm host a webservice?
I also researched using System.Net.Sockets and even got a barely functional prototype going however I feel I'm not skilled enough to make it a solid and reliable system. Given this path, I would need to learn more about socket programming and threading.
These boxes are running .NET 3.5 SP1, so I have complete flexibility in the .NET version however I'd like to stick to C#.
What is the best way to implement this? Should I just bite the bullet and learn Sockets more or does .NET have a better way of handling this?
edit:
I was going to go with a two way communication until I realized that all I needed was a one way communication.
edit 2:
I was avoiding the traditional server/client and going with an inverse because I wanted to avoid consuming too much bandwidth and wasn't sure what kind of overhead I was talking about. I was also hoping to have more control of the individual kiosks. After looking at it, I think I can still have that with WCF and connect by IP (which I wasn't aware I could connect by IP, I was thinking I would have to add 50 webservices or something).

WCF does not have to be hosted within IIS, it can be hosted within your Winform, as a console application or as windows service.
You can have each computer host its service within the winform, and write a program in your own computer to call each computer's service to get the status information.
Another way of doing it is to host one service in your own computer, and make the 50+ computers to call the service once their status were updated, you can use a database for the service to persist the status data of each node within the network. This option is easier to maintain and scalable.
P.S.
WCF aims to replace .net remoting, the alternatives can be net.tcp binding or net.pipe

Unless you have plans to scale this to several thousand clients I don't think WCF performance will even be a fringe issue. You can easily host WCF services from windows services or Winforms applications, and you'll find getting something working with WCF will be fairly simple once you get the key concepts.
I've deployed something similar with around 100-150 clients with great success.
There's plenty of resources out on the web to get you started - here's one to get you going:
http://msdn.microsoft.com/en-us/library/aa480190.aspx

Whether you use a web service or WCF on your central server, you only need to install and configure IIS on the server (and not on the 50+ clients).
What you're trying to do is a little unclear from the question, but if the clients need to call the server (to get a server status, for example), then they just call a method on the webservice running on the server.
If instead you need to have the server call the clients from time to time, then you'll need to have each client call a sign-in method on the server webservice each time the client starts up. The sign-in method would take a delegate method from the client as a parameter. The server would then call this delegate when it needed information from the client.
Setting up each client with its own web service would represent an inversion of the traditional (one server, multiple clients) client/server architecture, and as you've already noted this would be impractical.

Do not use remoting.
If you want robustness and scalability you end up ruling out everything but what are essentially stateless remote procedure calls. Since this is exactly the capability of web services, and web services are simpler and easier to build, remoting is an essentially pointless technology.
Callbacks with remote delegates are on the performance/reliability forbidden list, so if you were thinking of using remoting for that, think again.
Use web services.
I know you don't want to be polling, but I don't think you need to. Since you say all your units are on a single network segment then I suggest UDP for broadcast change notifications, essentially setting a dirty flag, and allowing the application to (re-)fetch on demand. It's still not reliable but it's easy and very fast because it's broadcast.
As others have said you don't need IIS, you can self-host. See ServiceHost class for details on how to do this.

I'd suggest using .NET Remoting. It's quite easy to implement and doesn't require anything else.

For me its is better to learn networking.. or the manual way of socket communication.. web services are mush slower because it contains metadata..
your clients and the servers can transform to multithreaded application. just imitate the request and response architecture. it is much easy to implement a network application like this..

If you just need a status update, you can use much simpler solution, such as simple tcp server/client messaging or like orrsella said, remoting. WCF is kinda overkill here.
One note though, if all your 50+ kiosk is connected via internet, then you might need use VPN or have an open port on each kiosk(which is a security risk) so that your server can retrieve status update from each kiosk.
We had a similiar situation, but the status is send to our server periodically, so we only have 1 port to protect/secure. The frequency of the update is configurable as to accomodate slower clients.

As someone who implemented something like this with over 500+ clients and growing:
Message Queing is the way to go.
We have gone from an internal developed TCP server and client to WCF polling and ended up with Message queing. It's the only guaranteed way to get data to and from clients and servers over the internet. As a bonus, many of these solutions have an extensive framework makeing it trivial to implement publish-subscribe, Send-one-way, point-to-point sending, Request-reply. Some of these are possible with WCF but it will involve crying, shouting, whimpering and long nights not to mention gallons of coffee.
A couple of important remarks:
Letting a process poll the clients instead of the other way around = Bad idea.. it is not scalable at all and you will soon be running in to trouble when the process is take too long to complete.. Not to mention having to handle all the ip addresses ( do you have access to all clients on the required ports ? What happpens when the ip changes etc..)
what we have done: The clients sends status updates to a central message queue on a regular interval ( you can easily implement live updates in the UI), it also listens on it's own queue for a GetStatusRequest message. if it receives this, it answers ( has a timeout).. this way, we can see overal status of all clients at all times and get a specific status of a specific client when needed.
Concerning bandwidth: kiosk usually show images/video etc.. 1Kb or less status messages will not be the big overhead.
I CANNOT stress enough that the current design you present will have a very intensive development cycle AND will not scale or extend well ( trust me, we have learned this lesson). Next to this, building a good client/server protocol for this type of stuff is a hard job that will be totally useless afterwards if you make a design error ( migrating a protocol is not easy)
We have built our solution ontop of ActiveMQ ( using NMS library c#) and are currently extending Simple Service Bus for our internal workings.
We only use WCF for the communication between our winforms app and the centralized service(s)