I am trying to create a monitoring application for our operations department to be proactive when dealing with systems that are encountering problems. I created an app that does the job but it has some draw backs:
Each copy of the app running serves individual pings to the systems, when 1 ping would suffice.
I have 3 different api's for getting the status of our systems depending whether its hosted IIS, WCF or desktop.
To fix the first issue i was going to create a database which an interim service(app)(monitor) would make the pings, then the app would query the database for updates. After thinking about this I realized the second issue and decided it is a future problem.
So my thought was to, rather than have the interim application pinging the systems, simply have each system have one interface in which it posts it status to the database every x time. But then I ran into a problem with the WCF and IIS services we have. These services can sit for days without anyone actually using the service. How would I make these services continue to post its data?
My questions are:
Is it better to have data REQUESTED or PUSHED in this type of situation?
If REQUESTED, what is a suggested practice for maintaining a single API across mulitple platforms(IIS, WCF, Desktop)?
If PUSHED, how would you handle the case of the Web services which are instance based and not continuously running?
For web services, one solution might be to implement a health-check end point , something that you can simply call like: webservice/isServiceUp?
I prefer that this information is PULLED. If a service / web Service/ Application is down, then you can't possibly rely on it to write something to the DB... it would be possible but highly risky and unreliable.
In a real world situation, it is a little more complicated than that because something might happen between your service host and the consumer (DNS problem for example), in which case, you would want to consider the case of not getting anything back from the isServiceUp (no true no false, just a 400 lvl error)...
Consider using your load balancer for checking on APPS / web services and proactively switching to a different IP in case of issues... it is a possibility.
Related
I'm facing an issue in a High Available Windows Services I developed with a master/slave setup.
Context:
The services itself synchronize data to two endpoints. One endpoint is synced to a local database, and one is an external. The database that is local is duplicated on both machines, so both master and slave need to sync this. The external endpoint only needs to be synced once.
The Master will by default sync to the external service, the slave will take over when the master is down. When the master goes back up and the slave is still synchronizing to the external, master will ask slave to finish a portion of the work, and then tell the master is done so he can continue the remaining work.
All this needs to happen asynchronously, I do not want the program to stop and wait for the other to respond (like the slave still handling the data).
I already implemented all the logic for this.
Setup:
Two Windows services running on two different machines.
Currently the communication is done over Named Pipes.
The problem:
Named pipes isn't reliable enough for the throughput that is being done. It also often crashes, and isn't made for reconnecting/closing and reopening a lot of times. I also face the problem that it just 'hangs' a lot when sending/receiving messages. Retrying sometimes works but I think I shouldn't be retrying. I need to have a reliable communication between the two instances.
Solutions:
I've been looking for an alternative to Named Pipes, but can't seem to find a solution of which I'm convinced that would work. Mostly because a lot of the technologies are for communication between a service and a client over http.
WCF over MSMQ is also not what I need, because I only want communication to happen when both are online. WCF in general is also more focused on one endpoint receiving data and sending a response. I need bidirectional communication, so both instances need to be able to receive and send messages at any time.
I think my best option is SignalR, but I'm also not convinced.
Have you looked at MassTransit over RabbitMQ?
We have been using them together very successfully both for intra-service and client/service communication for a few years now.
At my work we have some old 'services' which are actually Console applications which run 24/7 on our servers. Finally we decided to replace it with Windows Services, but now we can see from each console window when there are errors or certain events (like too many requests etc.)
Now the idea came up to let every windows service write in a database every minute (so we can see if the service hasn't stopped), and also write in the database when there's certain events (like the many requests).
Now I've been thinking about hosting a WCF Service in each of the windows services instead, because I think there's more possibilities (and I don't really like the idea of our databases having data inserted from 10+ services every minute) and it's still possible to write an application which sends requests to the WCF services every x amount of time.
Are there any downsides of WCF vs the approach of writing in a database? In a database of course it's easier to keep the data until it's been read, but that's only one (small) issue I can think of.
Thanks in advance!
I would have go with the follow options:
As you suggested, create WCF endpoint and verify you services are on. You will still need to implement error escalation. I like to use ELMAH
Using a watchdog How can I verify if a Windows Service is running
For the events, log services activities using a logger such as log4net or Nlog. Then append it to DB/Email/File. My favorite is to manage logs using ELK (Elastic/Logstash/Kibana) or Splunk.
we are evaluating a new project which will have a .NET Server which is available in the internet. We have access to the server but the hosting is done by a 3rd party company.
We are evaluating using WCF on the .NET Server. (I have no professional experience with WCF and just reading into the topic). The WCF service will talk to a SQL Server to perform its duties.
Here is the scenario:
Multiple client machines running our own ActionScript software will connect to that .NET Server.
Clients might be online 24/7 and should periodically poll our server to tell the server that they are there.
A client needs to be able to login, and only if the login has worked the other calls will be allowed and at some point it logs out. So we need to "remember" the state with a particual client...
Highest expected load is around 1000 Clients, of which 500 will only do polling while the other 500 will be "active". "Active" means a maximum of 1 call each minute, no heavy payload in each call, neither in the request nor in the response, just 1-3 database accesses per call.
We already tested some "HelloWorld" with ActionScript and WCF using BasicHttp(s)Binding.
But because we need session handling we were thinking about taking using the wsHttpBinding binding because it can provide us WCF Sessions.
So far so good, but then I stumbled upon the fact that it should
However:
I find that in my Oreilly WCF Services 3rd edition book (Page 177) it is written
and even Microsoft is writing to be careful using that:
http://msdn.microsoft.com/en-us/magazine/cc163590.aspx
"A service configured for private sessions cannot typically support more than a few dozen (or perhaps up to a few hundred) outstanding clients due to the cost associated with each such dedicated service instance."
So because we need to identify the state with each client, we could of course implement our own "Session Handling" on top of stateless HttpBindingBinding, and make a call to that SessionHandling class each time when my WCF methods get called, but I am reluctant to do anything like that, it looks to me like thousands of people should already have faced the same problem.
So, my question now is:
Do you think wsHttpBinding on my server could handle the payload?
How "bad" is it really to go with wsHttpBinding on WCF? Does anybody already have experience with this? Can I use it? What would you use?
Final Remarks:
I am not limited to WCF if we dont like it, we just shall do an evaluation.
From the companies point of view it would also be fine to go for a protobuf-RPC or XML-RPC solution over TCP and the ActionScript clients implementing that. (just examples!) So no need for hosting WCF in IIS on the server as long as the coding part is comfortable (enough) for the programmers on both sides and the ADMINISTRATION on the deployed server is not too much either. With just making some TCP-ports based communication I am a bit afraid what it would mean for the administration in regards to firewall and stuff. Payload is not an issue, client processing power is also not an issue. The only thing I am concerned about is scalability of the server and security.
Thanks in advance for any suggestions!
I would not be concerned with scalability. You can always add a server or two to your farm in case of issues.
I would rather be concerned with your architecture and the need to store anything in session - are you sure about that?
Note that you don't need ws binding to support sessions, basic binding supports sessions as well.
I am working on a project in which a WCF service will be consumed by iOS apps. The number of hits expected on the webserver at any given point in time is around 900-1000. Every request may take 1-2 seconds to complete. The same number of requests are expected on every second 24/7.
This is what my plan:
Write WCF RESTful service (the instance context mode will be percall).
Request/Response will be in Json.
There are some information that needs to be persisted in the server - this information is actually received from another remote system - which is shared among all the requests. Since using a database may not be a good idea (response time is very important - 2 seconds is the max the customer can wait), would it be good to keep it in server memory (say a static Dictionary - assume this dictionary will be a collection of 150000 objects - each object consists of 5-7 string types and their keys). I know, this is volatile!
Each request will spawn a new thread (by using Threading.Timers) to do some cleanup - this thread will do some database read/write as well.
Now, if there is a load balancer introduced sometime later, the in-memory stored objects cannot be shared between requests routed through another node - any ideas?
I hope you gurus could help me by throwing your comments/suggestions on the entire architecture, WCF throttling, object state persistence etc. Please provide some pointers on the required Hardware as well. We plan to use Windows 2008 Enterprise Edition server, IIS and SQL Server 2008 Std edition database.
Adding more t #3:
As I said, we get some information to the service from a remote system. On the web server where the the WCF is hosted, a client of the remote system will be installed and WCF references one of this client dlls to get the information, in the form of a hashtable(that method returns a hashtable - around 150000 objects will be there in this collection). Would you suggest writing this information to the database, and the iOS requests (on every second) which reach the service retrieves this information from the database directly? Would it perform better than consuming directly from this hashtable if this is made static?
Since you are using Windows Server 2008 I would definitely use the Windows Server App Fabric Cache to store your state:
http://msdn.microsoft.com/en-us/library/ff383813.aspx
It is free to use, well supported and integrated and is (more or less) API compatible with the Windows Azure App Fabric Cache if you every shift your service to Azure. In our company (disclaimer: not my team) we used to use MemCache but changed to the App Fabirc Cache and don't regret it.
Let me throw some comments/suggestions based on my experience in serving a similar amount or request under the WCF framework, 3.5 back in the days.
I don't agree to #3. Using a database here is the right thing to do. To address response time, implement caching and possibly cache dependency in order to keep the data synchronized across all instances (assuming that you are load balanced)(also see App Fabric suggested above/below). In real world scenarios, data changes, often, and you must minimize the impact.
We used Barracuda hardware and software to handle scalability as far as I can tell.
Consider indexing keys/values with Lucene if applicable. Lucene delivers extremely good performances when it comes to read/write. Do not use it to store your entire data, read on it. A life saver if used correctly. Note that it could be complicated to implement on a load balanced environment.
Basically, caching might be the only necessary change to your architecture.
I have 50+ kiosk style computers that I want to be able to get a status update, from a single computer, on demand as opposed to an interval. These computers are on a LAN in respect to the computer requesting the status.
I researched WCF however it looks like I'll need IIS installed and I would rather not install IIS on 50+ Windows XP boxes -- so I think that eliminates using a webservice unless it's possible to have a WinForm host a webservice?
I also researched using System.Net.Sockets and even got a barely functional prototype going however I feel I'm not skilled enough to make it a solid and reliable system. Given this path, I would need to learn more about socket programming and threading.
These boxes are running .NET 3.5 SP1, so I have complete flexibility in the .NET version however I'd like to stick to C#.
What is the best way to implement this? Should I just bite the bullet and learn Sockets more or does .NET have a better way of handling this?
edit:
I was going to go with a two way communication until I realized that all I needed was a one way communication.
edit 2:
I was avoiding the traditional server/client and going with an inverse because I wanted to avoid consuming too much bandwidth and wasn't sure what kind of overhead I was talking about. I was also hoping to have more control of the individual kiosks. After looking at it, I think I can still have that with WCF and connect by IP (which I wasn't aware I could connect by IP, I was thinking I would have to add 50 webservices or something).
WCF does not have to be hosted within IIS, it can be hosted within your Winform, as a console application or as windows service.
You can have each computer host its service within the winform, and write a program in your own computer to call each computer's service to get the status information.
Another way of doing it is to host one service in your own computer, and make the 50+ computers to call the service once their status were updated, you can use a database for the service to persist the status data of each node within the network. This option is easier to maintain and scalable.
P.S.
WCF aims to replace .net remoting, the alternatives can be net.tcp binding or net.pipe
Unless you have plans to scale this to several thousand clients I don't think WCF performance will even be a fringe issue. You can easily host WCF services from windows services or Winforms applications, and you'll find getting something working with WCF will be fairly simple once you get the key concepts.
I've deployed something similar with around 100-150 clients with great success.
There's plenty of resources out on the web to get you started - here's one to get you going:
http://msdn.microsoft.com/en-us/library/aa480190.aspx
Whether you use a web service or WCF on your central server, you only need to install and configure IIS on the server (and not on the 50+ clients).
What you're trying to do is a little unclear from the question, but if the clients need to call the server (to get a server status, for example), then they just call a method on the webservice running on the server.
If instead you need to have the server call the clients from time to time, then you'll need to have each client call a sign-in method on the server webservice each time the client starts up. The sign-in method would take a delegate method from the client as a parameter. The server would then call this delegate when it needed information from the client.
Setting up each client with its own web service would represent an inversion of the traditional (one server, multiple clients) client/server architecture, and as you've already noted this would be impractical.
Do not use remoting.
If you want robustness and scalability you end up ruling out everything but what are essentially stateless remote procedure calls. Since this is exactly the capability of web services, and web services are simpler and easier to build, remoting is an essentially pointless technology.
Callbacks with remote delegates are on the performance/reliability forbidden list, so if you were thinking of using remoting for that, think again.
Use web services.
I know you don't want to be polling, but I don't think you need to. Since you say all your units are on a single network segment then I suggest UDP for broadcast change notifications, essentially setting a dirty flag, and allowing the application to (re-)fetch on demand. It's still not reliable but it's easy and very fast because it's broadcast.
As others have said you don't need IIS, you can self-host. See ServiceHost class for details on how to do this.
I'd suggest using .NET Remoting. It's quite easy to implement and doesn't require anything else.
For me its is better to learn networking.. or the manual way of socket communication.. web services are mush slower because it contains metadata..
your clients and the servers can transform to multithreaded application. just imitate the request and response architecture. it is much easy to implement a network application like this..
If you just need a status update, you can use much simpler solution, such as simple tcp server/client messaging or like orrsella said, remoting. WCF is kinda overkill here.
One note though, if all your 50+ kiosk is connected via internet, then you might need use VPN or have an open port on each kiosk(which is a security risk) so that your server can retrieve status update from each kiosk.
We had a similiar situation, but the status is send to our server periodically, so we only have 1 port to protect/secure. The frequency of the update is configurable as to accomodate slower clients.
As someone who implemented something like this with over 500+ clients and growing:
Message Queing is the way to go.
We have gone from an internal developed TCP server and client to WCF polling and ended up with Message queing. It's the only guaranteed way to get data to and from clients and servers over the internet. As a bonus, many of these solutions have an extensive framework makeing it trivial to implement publish-subscribe, Send-one-way, point-to-point sending, Request-reply. Some of these are possible with WCF but it will involve crying, shouting, whimpering and long nights not to mention gallons of coffee.
A couple of important remarks:
Letting a process poll the clients instead of the other way around = Bad idea.. it is not scalable at all and you will soon be running in to trouble when the process is take too long to complete.. Not to mention having to handle all the ip addresses ( do you have access to all clients on the required ports ? What happpens when the ip changes etc..)
what we have done: The clients sends status updates to a central message queue on a regular interval ( you can easily implement live updates in the UI), it also listens on it's own queue for a GetStatusRequest message. if it receives this, it answers ( has a timeout).. this way, we can see overal status of all clients at all times and get a specific status of a specific client when needed.
Concerning bandwidth: kiosk usually show images/video etc.. 1Kb or less status messages will not be the big overhead.
I CANNOT stress enough that the current design you present will have a very intensive development cycle AND will not scale or extend well ( trust me, we have learned this lesson). Next to this, building a good client/server protocol for this type of stuff is a hard job that will be totally useless afterwards if you make a design error ( migrating a protocol is not easy)
We have built our solution ontop of ActiveMQ ( using NMS library c#) and are currently extending Simple Service Bus for our internal workings.
We only use WCF for the communication between our winforms app and the centralized service(s)