We Keep Coding

Real time activity monitoring for a Windows service

I have a Windows service that currently outputs logging activity to either a text file or a database (depending on the activity). What I would like to do is to have a way to run another process (probably an executable) that can connect to that service and receive activity updates from that Windows service using like a publish/subcribe approach.
In theory, I guess this can be done by hosting a socket connection on the Windows service and pushing activity data as it happens. I wonder though if there is s better approach? Is there maybe a framework that can do all this for me easily? Or maybe I should use a MQ product to broadcast the application activity?
I am using C# .net version 4.5

There are several ways:
Socket is very good two-way communication
WCF is also another option, also two way communication support
Database - if you need to keep the history of the signals, you probably use one table where server/host inserts into a table and client reads from table using SQL Dependency. You can read new signals without timer or waiting, almost in real time.
Another good option is SignalR.
I have been using all technologies except WCF.

how do I filter Azure Service Bus Queue messages based on a message property?

I am using Azure Service Bus Queue to send emails out from my app. I have many different customers that send out emails via my app and each message gets a property that identifies that customer: CustomerID
I need to write an admin area for my customers to look at the pending message in the queue and more importantly see the deadletter queue. I do not want them to see everyones deadletters so I want to filter the messages based on the property CompanyID.
How do I accomplish this?
I read about topics and subscriptions but I add 10+ customers a week at least and this would not be a reasonable solution for me.

Queues do not support Filtering. You can write admin clients that get all messages and filter on the client end but consider Topics/Subscriptions because you can easily add up to 2000 Subscriptions per Topic and then filter messages in these by Customer etc. For things that you want to Query repeatedly on an approach as the one mentioned above, where you have a daemon parse the queue and update a table and then each customer runs queries on that status table would work better.

Queues are generally not a good fit for querying and advanced filtering scenarios. Peeking through a large queue when a customer checks for status would defeat the whole purpose of using a service bus.
My suggestion is to store the status of started tasks in Azure table storage. Once worker role processes or fails processing a message in the queue, it could simply update the status in the table storage.

You may use a polling mechanism utilizing receiveMessages() API in PEEK_LOCK mode in order to achieve filtering over queues and maintain a subscription to a flux stream of messages that pass your filter.
I have attempted to write a sample implementation in this project - Find GitHub Repo Here
Also, you can have a look at this article to understand its usage and scope.

What is the best way to communicate between a WCF service and separate threads?

The wording of the question doesn't necessarily do the issue justice...
I've got a client UI sitting on a local box with and a background windows service to support it while it performs background functions.
The client UI is just the presentation layer and the windows service does all the hard hitting action... so there needs to be communication between the two of them. After spending a while on google and reading best practices, I decided to make the service layer using WCF and named pipes.
The client UI is the WCF client and the windows service acts as the WCF host (hosting locally only) to support the client.
So this works fine, as it should. The client UI can pass data to the WCF host. But my question is, how do I make that data useful? I've got a couple engines running on the windows service/WCF host but the WCF host is completely unaware of the existence of any background engines. I need the client's communications requests to be able to interact with those engines.
Does anybody have any idea of a good design pattern or methodology on how to approach facilitating communication between a WCF host and running threads?

I think that your best bet is to have some static properties or methods that can be used to interchange data between the service threads/processes and the WCF service.
Alternatively, the way that we approach this is through the use of a database where the client or wcf service queues up requests for the service to respond to and the service, when it is available, updates the database with the responses to those requests. The client then polls the database (through WCF) on a regular basis to retrieve the results of any outstanding requests.
For example, if the client needs a report generated, we fire off a request through WCF and WCF creates a report generation request in the database.
The service responsible for generating reports regularly polls this table and, when it finds a new entry, it spins off a new thread/process that generates the report.
When the report has completed (either successfully or in failure), the service updates the database table with the result.
Meanwhile, the client asks the WCF service on a regular basis if any of the submitted reports have completed yet. The WCF service in turn polls the table for any requests that have been completed, but not been delivered to the client yet, gathers the information from them, and returns them to the client.
This mechanism allows us to do a couple of things:
1) We can scale the number of services processing these requests across multiple physical/virtual machines as the workload increases.
2) A given service can support numerous clients.
3) Through the WCF interface, we can extend this support to any client platform that we choose to support (web, win, tablet, phone, etc).
Forgot to mention:
Just because we elect to use a database doesn't mean that you have to in order to implement this pattern. You can easily implement the same functionality by creating a static request collection that the WCF service and worker service access in much the same way that we use the database.
You will just need to be very careful about properly obtaining and releasing locks on the static properties to avoid cross-thread collisions or deadlocks.

How can I signal a c# console app from SQL Server?

Rather than poll against some tables, I'd like to signal a waiting c# app that there are new rows to be processed in a table, maybe via a trigger. Is there some way for the database to signal to a console app, or am I stuck polling the table looking for new rows?

Take a look at Query Notifications (SQL Server 2005+).
Microsoft SQL Server 2005 introduces
query notifications, new functionality
that allows an application to request
a notification from SQL Server when
the results of a query change. Query
notifications allow programmers to
design applications that query the
database only when there is a change
to information that the application
has previously retrieved.
There is an example here of how to write a simple form app to register a query for notification: http://msdn.microsoft.com/en-us/library/a52dhwx7(VS.80).aspx.
This does require the Service Broker to be enabled on the database.
You should take a look at the notes in the Remarks section of the MSDN SqlDependency documentation to make sure it is the right choice for your scenario

Check if SqlCacheDependency can be of any use...
http://www.asp.net/data-access/tutorials/using-sql-cache-dependencies-cs

If it is SQL Server 2008, You can use Event-Based Activation using Service Broker as well.

I have used WCF with SQLCLR to get data from another process into SQL, worked pretty good apart from some minor quirks to set it up. So you can just call other processes from SQl this way. It's usually quite hard to deploy to customers though, DBA don't like this sort of stuff.
GJ

Check out SQL Service Broker. Using a Queue and the WAITFOR syntax I have changed a custom polling service into a blocking/signal service. You could also look at the event-activation. Either way this would allow for a transactional way to call your external program in a non-polling async way that will not slow down triggers and database locks.

Is this the correct approach to poll the database?

I am creating a WCF service (CALLER) for Azure. The service(CALLER) calls async methods of another third party service(EXTN). The third party service calls the callback methods of another WCF service (LISTNER) hosted by me on Azure. CALLER enter the service details in the databsae with status = PENDING.
In the callback service (LISTNER) I am updating the status of the request as COMPLETED/FAILED in the database.
But I want the CALLER should be notified when status is updated in the SQL Azure db.
I am thinking of creating a worker thread which will poll the database periodically to check the status update and notify the CALLER about this.
Is there any other better / efficient alternative to this approach?

The features you're looking for are implemented in the AppFabric service bus.

Not really. There is another way (not sure it works on azure) by using a the integrated SQL message queueing (queue on updates via trigger), and your thread could continously poll then (there is a way to have a the read WAIT for an etnry in teh queue, so you issue one and it waits), but besides that...
...no, not from the database level.
I have a similar application and I handle it by a ntification trigger OUTSIDE The database (i.e. notifications are sent from the business logic that values change).

Another option is to use Queues and have the caller poll for notification messages from the listener. The Service Bus can be used, by having the Caller subscribe to event notifications sent from the Listener. In your scenario though it doesn't provide much more than the Queues do - if you are behind the firewall, the Service Bus uses polling as well.
Queues are probably the most efficient way to send notifications - that's why they were created in the first place. The Service Bus is used to create semi-permanent connections between different services by providing a lot more features than simple message passing. That makes it a bit less flexible, requires a bit more programming. Its billing model (charge per SB connection) reflect this too. You are not expected to use a lot of SB connections.