We Keep Coding

WCF service health monitoring

I just implemented a WCF service and I am currently looking at service monitoring options. Our server team that currently hosts only java services wants us to have instances running all the time, so it can gather data in that instance during its lifetime and they said they will use one of our operations with webmon to get statistical information. But we are using per call and I dont think that will work under this architecture.
I am wondering if there is a way to get the statistics of how an operation in the service did in certain amount of time and provide an another operation for webmon to use that gives an integer value about its performace in certain time period, webmon, then decides weather to alert the admin or not.
I was considering parsing of log files to get statistics but that might be an expensive operation if done every 15 mins.
If not what are my options for detailed automatic health monitoring of wcf applications?

My company very recently agreed to open-source (under the GPL License) the tool that we use internally to monitor our live web services and for producing availability and response time reports. It's called ServiceMon and it may meet your needs.
It runs on Windows as a standalone application and works by following a simple script of operations that dictate the services to be monitored. For example, to check a web page contains a particular value, in a similar manner to webmon, you'd use this line:
http-get "http://www.google.com" must-contain "I'm Feeling Lucky"
The frequency at which it executes the script operations can be easily configured as can the order which it processes them.
In addition to monitoring web pages and web services we use ServiceMon to track availability statistics of each service and to produce response time statistics.
ServiceMon is written using a plugin architecture so you can use .NET to add new types of monitoring operations. So, for example, if your web service uses funky authentication you can fairly easily plug this in to the utility.
Full documentation and download instructions here
I hope you find it useful and I'd love to hear your thoughts
Disclaimer: I developed ServiceMon so I may be a little bit biased :)

how many webservices

I have a web service that looks like this:
public class TheService : System.Web.Services.WebService
{
[WebMethod(EnableSession = true)]
public string GetData(string Param1, string Param2) { ... }
}
In other words, it's contained in one class and in there, I have one public method and there is another private method that does a read to the database.
The issue I'm facing is in terms of scalability. I'm building a web app that should work for 1,000 daily users and each user will do about 300-500 calls a day to the web service and so that's about 300,000 to 500,000 requests per day. I need to add 9 more calls to the web service. Some of these calls will involve database writes.
My question is this: am I better off creating 9 separate web services or continue with the one service I have and add the other methods. Or may be something different and better. I'm planning to deploy the application on Azure so I'm not really concerned about hardware, just the application side of things.

I wouldn't base my decision off the volume, or for performance/scalability reasons. You won't get much if any performance benefit from keeping them lumped together or separating them. Any grouping or filtering that can be done while the services are grouped one way can also be done with the services grouped the other way. The ability to partition between servers will be the same, too.
Design
Instead I would focus on trying to make your code understandable and maintainable. Group your services how they make the most sense architecturally within your program. Keep them logically grouped how they make the most sense to be grouped, from a problem-domain perspective (as opposed to a solution domain perspective).
Since you're free to group them how you want, I recommend you read up on SOLID, which is a set of guiding principles for creating software architecture.
One of the principles listed that is particularly important is the Interface Segregation Principle, which can be defined by the notion that "many client specific interfaces are better than one general purpose interface."
Performance and scalability
Since you mentioned performance and scalability being a concern, I recommend you follow this plan:
Determine how long you can wait until you can patch/maintain the software
Determine your expected load, including both average and peak load-per-time (you've determined the average), and how much you expect this traffic to grow over time (specifically over the period you can go without patching/maintaining the software)
Create a model describing exactly which calls will be done and in which ratios (per time and per server)
Create automation that mirrors these models as closely as you can. Try to model both average and peak traffic, and surpassing your highest scale traffic
Profile your code, DB, network traffic, and disk traffic while running this automation
Determine the bottlenecks, and if they are within acceptable tolerance
Optimize your bottlenecks (as required), and repeat from the profiling step forward
The next release of your software, repeat from the top to add scenarios/load/automation
Perform regression testing using your existing tests, altered to fit the new scale

Splitting the web methods into several web services won't help you here; load balancing will.

The number of web services will not have any affect on scalability of the app.
Finding your bottlenecks will help scalability. If you're bottleneck is the DB, you may need to find ways to tune your queries, partition your data across more stores, etc... If you're bottleneck is CPU on the web services (web roles in azure), then adding more than one web role to your cluster will help. Azure supports that.
But, simply don't start adding roles. Understand where your bottlenecks are. Measure, profile and tune.
Azure has devfabric and IIS locally to help you profile locally as well.

Splitting the web-services into multiple web roles because of physical constraints and not necessarily due to logical layout may be worth considering because:
Using Azure you can scale out your Roles independently of one another. This means that IF different web methods need to scale in different patterns (ie: your first web method has the biggest volume in the mornings and after lunch and your other two web methods have the biggest volume in the evening and during the night), and the last 2 web methods are usually flat throughout the day, it very well maybe worth it to split your methods across Roles by scalability constraints and not by logical constraints.
By increasing/decreasing the servers allocated to each method independently you maybe able to fine-tune your optimal power vs. need with a much greater precision.
HTH

Actually, creating separate Web Services, as Igorek suggested, will provide much more granular scale-out. In that scenario, you can deploy different Web Services to different Roles, each role getting its own set of instances (along with the option to create different instance sizes per role). Windows Azure will load-balance across all the instances of a Role.
So from a granularity standpoint:
Least granular: Combine all methods into a single Web Service, hosted on a single Role. As you scale out to multiple instances, all service method requests are load-balanced across all instances. Because you're combining everything into one Role, you will find this to be optimized for cost: You can run all Web Services code in a single instance (really 2 instances to give yourself SLA).
More granular: Create separate Web Services, each with their own methods, and host on the same Role (allows you to exercise SOLID principles, as Merlyn described). Same basic performance characteristics as the first option, as all requests are still load-balanced across the same set of instances.
Most granular: Create separate Web Services, each with their own methods, and host each Web Service endpoint on a separate Role, allowing for independent VM sizing and scale-out of each Web Service endpoint. This option has a higher runtime cost to it, as you now have a minimum of one instance per Web Service endpoint (again, 2 instances in a real world, live application).

I am not sure about exact your case, but moving expensive (from CPU/DB point of view) tasks to separate Worker Role usually are good solution for Azure. In that case you will have one WebRole with services that will receive requests (it will be light weight, so you sjould not have many Instances for it) and create tasks for Worker Roles and one or few Worker Roles that will process that tasks - #1 Worker Roles can be created per kind of task (to group similar actions like reading/writing data to DB) or #2 one Worker Role can handle any type of task. I don't see any benefits in #2, because to get the same behavior you can just create one WebRole with many instances and handle all there. So you will have ability to control processing time by adding/removing Worker Roles.
As other people suggested - using Azure platform by itself will not make app scalable, especially if you are using SQL Azure, you will need to implement sharding or add many DBes to avoid one big DB for all requests.
I don't know if that's related to this questing, but just to let you know - Azure is dropping connections which are not active during 60 sec (I did not find some way to increase that timeout, you can Google this problem). This may be an issue is you are porting web-services to Azure and your responses can reach 60 seconds. One way to avoid it is keeping connection active, which is pretty simple if clients know about this "feature".

Windows Workflow Foundation WF4 - Workflow hosting

For a client the system we're creating must support the following:
- It must be possible to run multiple workflows, and multiple instances of the same workflows with a different context (different data/business objects).
- Some workflows will be long-running, involve multiple users/client session and waiting for external user input. So the workflows must be able to be persisted and respond to some signal from a client app. And it also means that the execution of workflows must be done on a server app (right?).
- I want to be able to run all kinds of workflows on the server app, and I do not want to have to re-deploy the server app when a workflow changes.
My first thought was Workflow Services. After a lot of research I concluded that this is not the right path since Workflow Services basically gives the possibility to execute activities at a remote location from a workflow started in a client app. Is this correct? Or can I use Workflow Services for the scenario above? Most examples and/or tutorials are basically a ReceiveSignal/Send combination with some logic in between.
Basically I want to initiate (from a client app) the start of a workflow with a specific context (in the workflow server app).
What is the best approach?
Any help is very much appreciated!

As for your requirements:
It must be possible to run multiple workflows, and multiple instances of the same workflows with a different context (different data/business objects).
This is no problem with WF.
Some workflows will be long-running,
involve multiple users/client session
and waiting for external user input.
So the workflows must be able to be
persisted and respond to some signal
from a client app. And it also means
that the execution of workflows must
be done on a server app (right?).
WF is designed for long running tasks that can interact with outside influences. However, that doesn't say its easy to accomplish; there is no universal solution which you can hook into. You will probably have to design custom Activities that interact with Workflow Extensions which handle moving user input into the workflow. Same with exposing the workflow to the outside, although WF4 does come with a host of WCF activities which could be used to accomplish this.
I want to be able to run all kinds of
workflows on the server app, and I do
not want to have to re-deploy the
server app when a workflow changes.
This is harder to accomplish. You must, at a minimum, separate the workflows from the server code. The simplest route is to store your workflow as xaml and load it at runtime from, say, a database.
Other options are to use some kind of dependency injection framework (such as Structure Map or Unity) which loads the workflow assembly at runtime. If the workflows change, you can drop the new assembly on the server, change your config and restart. Alternatively, you can isolate your workflow assemblies within their own AppDomain, load them at runtime and throw away the domain when you must reload a new version. Which one you do depends on your requirements; I'm actually doing the third option as I have to load many different versions of workflow assemblies at runtime, run them concurrently, and they often have embedded resources thus preventing me from going the XAML route.
My first thought was Workflow
Services. After a lot of research I
concluded that this is not the right
path since Workflow Services basically
gives the possibility to execute
activities at a remote location from a
workflow started in a client app. Is
this correct?
I'm hosting my workflows within a standard Windows service application. I have to manage and maintain the WCF frontend which the client uses to interact with my workflows. As far as I can tell, Workflow Services seems like a viable option for stable workflows, if I understand it correctly. Hosting an application in AppFabric is also a good option, and I believe simpler than using a Windows Service. But no matter what the host is, you have two options--your workflows define your service contract, or you must define the service contract and handle all execution and communication with workflows you are managing.
The first is a good option for stable workflows with a simple facade. It doesn't seem like a good option for you, as you must load workflows dynamically as they change; that requires logic outside of the workflow to handle not only communications from the client ("here's a new version of workflow X!") but also managing the workflow lifespan.
It seems like you will have to find some kind of host application (IIS WebService application, Windows Service, AppFabric, Azure), define your WCF service and bring it online, handle calls from clients, communicate these calls to your workflow running code, which then must load and execute these calls and return the result up the chain.
I can't help but notice that you seem slightly ill-prepared for the journey that awaits you. I would strongly suggest creating a prototype that slices neatly through the centermost meat of your requirements. A hosted application (I'd suggest AppFabric) with a WCF frontend that loads a XAML-based workflow that processes client calls. Once you have the simple version nailed down, you can widen the scope to encompass all your requirements.

WCF for a shared data access

I have a little experience with WCF and would like to get your opinion/suggestion on how the following problem can be solved:
A web service needs to be accessible from multiple clients simultaneously and service needs to return a result from a shared data set. The concrete project I'm working on has to store a list of IP addresses/ranges. This list will be queried by a bunch of web servers for a validation purposes and we speak of a couple of thousand or more queries per minute.
My initial draft approach was to use Windows service as a WCF host with service contract implementing class that is decorated with ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple) that has a list object and a custom locking for accessing it. So basically I have a WCF service singleton with a list = shared data -> multiple clients. What I do not like about it is that data and communication layers are merged into one and performance wise this doesn't feel "right".
What I really really (- want is Windows service running an instance of IP list holding container class object, a second service running WCF service contract implementation and a way the latter querying the former in a nice way with a minimal blocking. Using another WCF channel would not really take me far away from the initial draft implementation or would it?
What approach would you take? Project is still in a very early stage so complete design re-do is not out of question.
All ideas are appreciated. Thanks!
UPDATE: The data set will be changed dynamically. Web service will have a separate method to add IP or IP range and on top of that there will be a scheduled task that will trigger data cleanup every 10-15 minutes according to some rules.
UPDATE 2: a separate benchmark project will be kicked up that should use MySQL as a data backend (instead on in-memory list).

It depends how far it has to scale. If a single server will suffice, then fine; keep it conveniently in memory (as long as you can recreate the data if the server gets restarted). If the data-volume is low, then simple blocking (lock) should work fine to synchronize the data, or for higher throughput a ReaderWriterLockSlim. I would probably not store it directly in the WCF class instance, though.
I would avoid anything involving sessions (if/when this ties into the WCF life-cycle); this is rarely helpful to simple services.
For distributed load (over multiple servers) I would give consideration to a separate dedicated backend. A database or memcached / AppFabric / etc would be worth consideration.

High availability & scalability for C#

I've got a C# service that currently runs single-instance on a PC. I'd like to split this component so that it runs on multiple PCs. Each PC should be assigned a certain part of the work. If one PC fails, its work should be moved to a backup machine.
Data synchronization can be done by the DB, so that should not be much of an issue. My current idea is to use some kind of load balancer that splits and sends the incoming requests to the array of PCs and makes sure the work is actually processed.
How would I implement such a functionality? I'm not sure if I'm asking the right question. If my understanding of how this goal should be achieved is wrong, please give me a hint.
Edit:
I wonder if the idea given above (load balancer splitswork packages to PCs and checks for result) is feasible at all. If there is some kind of already implemented solution so this seemingly common problem, I'd love to use that solution.
Availability is a critical requirement.

I'd recommend looking at a Pull model of load-sharing, rather than a Push model. When pushing work, the coordinating server(s)/load-balancer must be aware of all the servers that are currently running in your system so that it knows where to forward requests; this must either be set in config or dynamically set (such as in the Publisher-Subscriber model), then constantly checked to detect if any servers have gone offline. Whilst it's entirely feasible, it can complicate the scaling-out of your application.
With a Pull architecture, you have a central work queue (hosted in MSMQ, Sql Server Service Broker or similar) and each processing service pulls work off that queue. Expose a WCF service to accept external requests and place work onto the queue, safe in the knowledge that some server will do the work, even though you don't know exactly which one. This has the added benefits that each server monitors it's own workload and picks up work as-and-when it is ready, and you can easily add or remove servers to/from this model without any change in config.
This architecture is supported by NServiceBus and the communication between Windows Azure Web & Worker roles.

From what you said each PC will require a full copy of your service -
Each PC should be assigned a certain
part of the work. If one PC fails, its
work should be moved to a backup
machine
Otherwise you won't be able to move its work to another PC.
I would be tempted to have a central server which farms out work to individual PCs. This means that you would need some form of communication between each machine and and keep a record back on the central server of what work has been assigned where.
You'll also need each machine to measure it's cpu loading and reject work if it is too busy.
A multi-threaded approach to the service would make good use of those multiple processor cores that are ubiquitoius nowadays.

How about using a server and multi-threading your processing? Or even multi-threading on a PC as you can get many cores on a standard desktop now.
This obviously doesn't deal with the machine going down, but could give you much more performance for less investment.

you can check windows clustering, and you have to handle set of issues that depends on the behaviour of the service (you can put more details about the service itself so I can answer)

This depends on how you wanted to split your workload, this usually done by
Splitting the same workload by multiple services
Means same service being installed on
different servers and will do the
same job. Assume your service is reading huge data from the db servers and processing them to produce huge client specific datafiles and finally this datafile is been sent to the clients. In this approach all your services installed in diff servers will do the same work but they split the work to increaese the performance.
Splitting the part of the workload by multiple services
In this approach each service will be assigned to the indivitual jobs and works on different goals. in above example one serivce is responsible for reading data from db and generating huge data files and another service is configured only to read the data file and send it to clients.
I have implemented the 2nd approach in one of my work. Because this let me isolate and debug the errors in case of any failures.

The usual approach for load balancer is to split service requests evenly between all service instances.
For each work item (request) you can store relative information in database. Then each service should also have at least one background thread checking database for abandoned work items.

I would suggest that you publish your service through WCF (Windows Communication Foundation).
Then implement a "central" client application which can keep track of available providers of your service and dish out work. The central app will act as scheduler and load balancer of the tasks to be performed.
Check out Juwal Lövy's book on WCF ("Programming WCF Services") for a good introduction on this topic.

You can have a look at NGrid : http://ngrid.sourceforge.net/
or Alchemi : http://www.gridbus.org/~alchemi/index.html
both are grid computing framework with load balancers that will get you started in no time.
Cheers,
Florian