Let's say you have this console application:
static void Main(string[] args)
{
var httpClient = new HttpClient()
{ BaseAddress = new Uri("http://www.timesofmalta.com") };
var responseTask = httpClient.GetAsync("/");
}
Since the task is not awaited, the program reaches its end, finds no other foreground threads executing, and exits before any response is received. That's pretty clear because this is a console application.
Now let's say you have a WCF application, where a request similarly causes a task to be spawned, but does not await it. Let's say this task is long-running, and is fire-and-forget rather than anything like an HTTP GET.
In such a case, what happens to that task? Does the thread just die as in the console application, bringing down the task with it? Can this cause code occurring later in the task to not be executed?
It depends on how your WCF is hosted. Whenever the application exits, its threads will be torn down, and any outstanding asynchronous operations are simply dropped.
Note that if WCF is hosted in ASP.NET, then fire-and-forget is dangerous; ASP.NET will recycle your app periodically just to keep things clean, and at that time your fire-and-forget operation can disappear. ASP.NET provides APIs to register work like this (if you absolutely must do it in-process).
If you're running on another host, you'll have to take care of registering with that host, using whatever technique is available.
Or, you can introduce a proper distributed architecture: the WCF endpoint merely serializes a description of the work to be done into a reliable queue (Azure queue / MSMQ / WebSphereMQ / etc), and an independent background worker processes that work (Azure webjob / Azure worker role / Win32 service / etc). This is a more complex setup but fixes the "lost work" problem you can get if you try to have your WCF app do it all in-process.
Related
I have a Service Fabric cluster hosting an 'Orchestrator'-type service which spins up and shuts down other Stateful services to do work, using FabricClient.ServiceManagementClient's CreateServiceAsync and DeleteServiceAsync methods.
The work involves processing messages which are stored for a short time within a ReliableConcurrentQueue.
I'm trying to handle the graceful shutdown of these services via the CancellationToken by ensuring that the queue is completely drained of messages before the service is deleted, but have found that the service's access to the ReliableConcurrentQueue is revoked once the CancellationToken is cancelled.
For example, calling StateManager.GetOrAddAsync<T>() from a callback registered with the CancellationToken, results in a FabricNotReadableException, containing the message "Primary state manager is currently not readable".
Reading around, it seems this is expected behaviour:
"In Service Fabric, when a Primary is demoted, one of the first things
that happens is that write access to the underlying state is revoked."
https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-lifecycle
Also, the answers to this question suggest that FabricNotReadableException is often a transient issue, and affected calls can be retried. This doesn't seem to be the case in this example; multiple retries at various frequencies/delays all seem to fail the same way.
Is there a way to guarantee that everything in the queue is processed using the combination of Stateful services, Reliable Collections and CancellationTokens? Or should I be looking into storage outside of what Service Fabric can provide?
Consider performing the queue item processing inside RunAsync.
Stopping / changing the role of a service causes the CancellationToken passed to RunAsync to be cancelled.
Once that happens, you need to make sure that you only exit that method when the queue depth is 0.
Also, once this cancellation is requested, you should probably stop allowing new items to be enqueued.
I've been trying to learn about WCF services and hosts. I made a simple host program to host my simple service. It works fine, but I don't understand how the host program can continue completing unrelated tasks after opening the service. Does the service run on a separate thread that opens behind the scenes? Or when my client calls the service, does that pause the host program? I don't see that documented anywhere.
namespace MyHostProgram
{
class Program
{
static void Main(string[] args)
{
var host = new ServiceHost(typeof(MyServices.Service1));
host.Open();
while (true)
{
Console.Writeline("Doing other tasks in host program");
}
host.Close();
}
}
}
Note that I am not asking if adding another thread will speed things up like WCF Service and Threading, I'm asking what the default behavior is.
When you call the Open function of the ServiceHost class, it creates and opens the listener for the service on the configured endpoints. It does this asynchronously and the control is given back to the calling thread.
So the answers to your questions are:
Does the service run on a separate thread that opens behind the scenes?
Yes
when my client calls the service, does that pause the host program?
No
All that means is Open() spawns up a new thread that contains the transport receive-loop or registers for asynchronous callbacks (depending on the binding).
You may consider reading on multi-threading and asynchronous programming to better grasp this.
Hope this helps!
I currently have one Service Fabric application that is composed of multiple Services. What I'm trying to achieve is a Queuing mechanism so one Service can publish a message to a queue, and another Service can receive messages from the same queue.
The following doesn't work (for the Listener service, there is nothing to dequeue):
PublisherService:
protected override async Task RunAsync(CancellationToken cancellationToken)
{
var myQueue = await StateManager.GetOrAddAsync<IReliableQueue<string>>("fooQueue");
while (true)
{
cancellationToken.ThrowIfCancellationRequested();
using (var tx = this.StateManager.CreateTransaction())
{
// Put some message in the queue
await myQueue.EnqueueAsync(tx, "Foobar");
await tx.CommitAsync();
}
await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken);
}
}
ListenerService:
protected override async Task RunAsync(CancellationToken cancellationToken)
{
var myQueue = await StateManager.GetOrAddAsync<IReliableQueue<string>>("fooQueue");
while (true)
{
cancellationToken.ThrowIfCancellationRequested();
using (var tx = this.StateManager.CreateTransaction())
{
var result = await myQueue.TryDequeueAsync(tx);
if (result.HasValue)
{
ServiceEventSource.Current.ServiceMessage(this.Context, "New message receieved: {0}", result.Value.ToString());
}
await tx.CommitAsync();
}
await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken);
}
}
It looks like the scope of a queue is limited to a single Service. This doesn't appear to be a limitation specified in the documentation.
So my questions are:
is this actually some undocumented limitation?
or is there something wrong in the code above?
how could I achieve the scenario above (one service adds messages to a queue, another service retrieves messages from the same queue)?
Obviously I could use an Azure Service Bus, but I can't for several reasons:
in my actual real-world scenario, I will have several queues (variable number) so it would require creating Service Bus Queues on demand (which is not exactly a fast operation)
adds a dependency to another Azure service (so increases the failure probability for the whole system)
costs more
more complex deployment
etc.
ReliableQueues are local to a service yes, because its intent is to store state for that particular service. That state is replicated to other instances. It is like a normal System.Collections.Generic.Queue<T> in .Net.
For a low cost solution maybe you can use Azure Storage Queues. Yes, it adds a dependency but it has a high availability. It is a tradeoff that only you can decide to accept or not.
On the other hand, think out of the box:
Create a stateful service with multiple ReliableQueues and expose methods other services can call using stand remoting communication like:
class QueuingService
{
void AddToQueue<T>(string queuename, T input) { .. }
void DeQueue(string queuename) { .. }
}
This creates of course a dependency but it has all the safety mechanisms Service Fabric provides and does not cost you much. But then again, you are building a poor mans service bus/azure storage queue yourself.
About the docs, no it does not says so with many words that a reliable queue is tied to 1 service but it depends on how you interpret this
Service Fabric offers a stateful programming model available to .NET developers via Reliable Collections. Specifically, Service Fabric provides reliable dictionary and reliable queue classes. When you use these classes, your state (my interpretation: The state of the service) is partitioned (for scalability), replicated (for availability), and transacted within a partition (for ACID semantics).
Check out the Priority Queue Service, which was created for this purpose.
If you add a fault-handling retry pattern to all of your calling code, you should not need a queue in between your calls, see https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-communication
Relevant part from the link is here:
An exception handler is responsible for determining what action to take when an exception occurs. Exceptions are categorized into retryable and non retryable.
Non retryable exceptions simply get rethrown back to the caller.
retryable exceptions are further categorized into transient and non-transient.
Transient exceptions are those that can simply be retried without re-resolving the service endpoint address. These will include transient network problems or service error responses other than those that indicate the service endpoint address does not exist.
Non-transient exceptions are those that require the service endpoint address to be re-resolved. These include exceptions that indicate the service endpoint could not be reached, indicating the service has moved to a different node.
I have a Windows Service that takes the name of a bunch of files and do operations on them (zip/unzip, updating db etc). The operations can take time depending on size and number of files sent to the service.
(1) The module that is sending a request to this service waits until the files are processed. I want to know if there is a way to provide a callback in the service that will notify the calling module when it is finished processing the files. Please note that multiple modules can call the service at a time to process files so the service will need to provide some kind of a TaskId I guess.
(2) If a service method is called and is running and another call is made to the same service, then how will that call be processed(I think there is only one thread asociated with the service). I have seen that when the service is taking time in processing a method, the threads associated with the service begin to increase.
WCF does indeed offer duplex bindings which allow you to specify a callback contract, so that the service can call back to the calling client to notify.
However, in my opinion, this mechanism is rather flaky and not really to be recommended.
In such a case, when the call causes a fairly long running operation to happen, I would do something like this:
If you want to stick to HTTP/NetTcp bindings, I would:
drop off the request with the service, and then "let go" - this would be a one-way call, you just drop off what you want to have done, and then your client is done
have a status call that the client could call after a given time to find out whether or not the results of the request are ready by now
if they are, there should be a third service call to retrieve the results
So in your case, you could drop off the request to zip some files. The service would go off and do its work and store the resulting ZIP in a temporary location. Then later on the client could check to see whether the ZIP is ready, and if so, retrieve it.
This works even better over a message queue (MSMQ) which is present in every Windows server machine (but not a lot of people seem to know about it or use it):
your client drops off the request on a request queue
the service listens on that request queue and fetches request after request and does it works
the service can then post the results to a result queue, on which your callers in turn are listening
Check out how to do all of this efficiently by reading the excellent MSDN article Foudnations: Build a queue WCF Response Service - highly recommended!
A message-queue based systems tends to be much more stable and less error-prone that a duplex-/callback-contract based system, in my opinion.
(1) The simplest way to achieve this is with a taskId as you note, and then have another method called IsTaskComplete with which client can check whether the task has been completed.
(2) Additional calls made to the service will start new threads.
edit: the default service behaviour is to start new threads per call. The configurable property is Instance Context Mode, and can be set to PerCall, PerSession, or Shareable.
The question has a solution, but I'm using a WCF duplex service to get the result of a long operation, and even though I found a problem that has cost me several hours to solve (and that's why I searched this question earlier), now it works perfectly, and I believe it is a simple solution within the WCF duplex service framework.
What is the problem with a long operation? The main problem is blocking the client interface while the server performs the operation, and with the WCF duplex service we can use a call back to the client to avoid the blockage (It is an old method to avoid blocking but it can easily be transformed into the async/await framework using a TaskCompletionSource).
In short, the solution uses a method to start the operation asynchronously on the server and returns immediately. When the results are ready, the server returns them by means of the client call back.
First, you have to follow any standard guide to create WCF duplex services and clients, and I found these two useful:
msdn duplex service
Codeproject Article WCF Duplex Service
Then follow these steps adding your own code:
Define the call back interface with an event manager method to send results from the server and receive them in the client.
public interface ILongOperationCallBack
{
[OperationContract(IsOneWay = true)]
void OnResultsSend(....);
}
Define the Service Interface with a method to pass the parameters needed by the long operation (refer the previous ILongOperationCallBack interface in the CallBackContractAttribute)
[ServiceContract(CallbackContract=typeof(ILongOperationCallBack))]
public interface ILongOperationService
{
[OperationContract]
bool StartLongOperation(...);
}
In the Service class that implements the Service Interface, first get the proxy of the client call back and save it in a class field, then start the long operation work asynchronously and return the bool value immediately. When the long operation work is finished send the results to the client using the client call back proxy field.
public class LongOperationService:ILongOperationService
{
ILongOperationCallBack clientCallBackProxy;
public ILongOperationCallBack ClientCallBackProxy
{
get
{
return OperationContext.Current.GetCallbackChannel<ITrialServiceCallBack>());
}
}
public bool StartLongOperation(....)
{
if(!server.IsBusy)
{
//set server busy state
//**Important get the client call back proxy here and save it in a class field.**
this.clientCallBackProxy=ClientCallBackProxy;
//start long operation in any asynchronous way
......LongOperationWorkAsync(....)
return true; //return inmediately
}
else return false;
}
private void LongOperationWorkAsync(.....)
{
.... do work...
//send results when finished using the cached client call back proxy
this.clientCallBackProxy.SendResults(....);
//clear server busy state
}
....
}
In the client create a class that implements ILongOperationCallBack to receive results and add a method to start the long operation in the server (the start method and the event manager don't need to be in the same class)
public class LongOperationManager: ILongOperationCallBack
{
public busy StartLongOperation(ILongOperationService server, ....)
{
//here you can make the method async using a TaskCompletionSource
if(server.StartLongOperation(...)) Console.WriteLine("long oper started");
else Console.Writeline("Long Operation Server is busy")
}
public void OnResultsSend(.....)
{
... use long operation results..
//Complete the TaskCompletionSource if you used one
}
}
NOTES:
I use the bool return in the StartLongOperation method to indicate that the server is Busy as opposed to down, but it is only necessary when the long operation can't be concurrent as in my actual application, and maybe there are best ways in WCF to achieve non concurrency (to discover if the server is down, add a Try/Catch block as usual).
The important quote that I didn't see documented is the need to cache the call back client proxy in the StartLongOperation method. My problem was that I was trying to get the the proxy in the working method (yes, all the examples use the call back client proxy in the service method, but it isn't explicity stated in the documentation, and in the case of a long operation we must delay the call back until the operation ends).
Do not get and cache twice the call back Proxy after a service method has returned and before the next one.
Disclaimer: I haven't added code to control errors, etc.
I want to create a simple client-server example in WCF. I did some testing with callbacks, and it works fine so far. I played around a little bit with the following interface:
[ServiceContract(SessionMode = SessionMode.Required, CallbackContract = typeof(IStringCallback))]
public interface ISubscribeableService
{
[OperationContract]
void ExecuteStringCallBack(string value);
[OperationContract]
ServerInformation Subscribe(ClientInformation c);
[OperationContract]
ServerInformation Unsubscribe(ClientInformation c);
}
Its a simple example. a little bit adjusted. You can ask the server to "execute a string callback" in which case the server reversed the string and calls all subscribed client callbacks.
Now, here comes the question: If I want to implement a system where all clients "register" with the server, and the server can "ask" the clients if they are still alive, would you implement this with callbacks (so instead of this "stringcallback" a kind of TellTheClientThatIAmStillHereCallback). By checking the communication state on the callback I can also "know" if a client is dead. Something similar to this:
Subscribers.ForEach(delegate(IStringCallback callback)
{
if (((ICommunicationObject)callback).State == CommunicationState.Opened)
{
callback.StringCallbackFunction(new string(retVal));
}
else
{
Subscribers.Remove(callback);
}
});
My problem, put in another way:
The server might have 3 clients
Client A dies (I pull the plug of the laptop)
The server dies and comes back online
A new client comes up
So basically, would you use callbacks to verify the "still living state" of clients, or would you use polling and keep track "how long I havent heard of a client"...
You can detect most changes to the connection state via the Closed, Closing, and Faulted events of ICommunicationObject. You can hook them at the same time that you set up the callback. This is definitely better than polling.
IIRC, the Faulted event will only fire after you actually try to use the callback (unsuccessfully). So if the Client just disappears - for example, a hard reboot or power-off - then you won't be notified right away. But do you need to be? And if so, why?
A WCF callback might fail at any time, and you always need to keep this in the back of your mind. Even if both the client and server are fine, you might still end up with a faulted channel due to an exception or a network outage. Or maybe the client went offline sometime between your last poll and your current operation. The point is, as long as you code your callback operations defensively (which is good practice anyway), then hooking the events above is usually enough for most designs. If an error occurs for any reason - including a client failing to respond - the Faulted event will kick in and run your cleanup code.
This is what I would refer to as the passive/lazy approach and requires less coding and network chatter than polling or keep-alive approaches.
If you enable reliable sessions, WCF internally maintains a keep-alive control mechanism. It regularly checks, via hidden infrastructure test messages, if the other end is still there. The time interval of these checks can be influenced via the ReliableSession.InactivityTimeout property. If you set the property to, say, 20 seconds, then the ICommunicationObject.Faulted event will be raised about 20 to 30 (maximum) seconds after a service breakdown has occurred on the other side.
If you want to be sure that client applications always remain "auto-connected", even after temporary service breakdowns, you may want to use a worker thread (from the thread pool) that repeatedly tries to create a new proxy instance on the client side, and calls a session-initiating operation, after the Faulted event has been raised there.
As a second approach, since you are implementing a worker thread mechanism anyway, you might also ignore the Faulted event and let the worker thread loop during the whole lifetime of the client application. You let the thread repeatedly check the proxy state, and try to do its repair work whenever the state is faulted.
Using the first or the second approach, you can implement a service bus architecture (mediator pattern), guaranteeing that all client application instances are constantly ready to receive "spontaneous" service messages whenever the service is running.
Of course, this only works if the reliable session "as such" is configured correctly to begin with (using a session-capable binding, and applying the ServiceContractAttribute.SessionMode, ServiceBehaviorAttribute.InstanceContextMode, OperationContractAttribute.IsInitiating, and OperationContractAttribute.IsTerminating properties in meaningful ways).
I had a similar situation using WCF and callbacks. I did not want to use polling, but I was using a "reilable" protocol, so if a client died, then it would hang the server until it timed out and crashed.
I do not know if this is the most correct or elegant solution, but what I did was create a class in the service to represent the client proxy. Each instance of this class contained a reference to the client proxy, and would execute the callback function whenever the server set the "message" property of the class. By doing this, when a client disconnected, the individual wrapper class would get the timeout excetpion, and remove itself from the server's list of listeners, but the service would not have to wait for it. This doesn't actually answer your question about determining if the client is alive, but it is another way of structuring the service to addrss the issue. If you needed to know when a client died, you would be able to pick up when the client wrapper removed itself from the listener list.
I have not tried to use WCF callbacks over the wire but i have used them for interprocess communication. I was having a problem where call of the calls that were being sent were ending up on the same thread and making the service dead lock when there were calls that were dependant on the same thread.
This may apply to the problem that you are currently have so here is what I had to do to fix the problem.
Put this attribute onto the server and client of the WCF server implemetation class
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple)]
public class WCFServerClass
The ConcurrencyMode.Multiple makes each call process on its own thread which should help you with the server locking up when a client dies until it timesout.
I also made sure to use a Thread Pool on the client side to make sure that there were no threading issues on the client side