I'm trying to build a file download actor, using Akka.net. It should send messages on download completion but also report download progress.
In .NET there are classes supporting asynchronous operations using more than one event. For example WebClient.DownloadFileAsync has two events: DownloadProgressChanged and DownloadFileCompleted.
Preferably, one would use the task based async version and use the .PipeTo extension method. But, I can't see how that would work with an async method exposing two events. As is the case with WebClient.DownloadFileAsync. Even with WebClient.DownloadFileTaskAsync you still need to handle DownloadProgressChanged using an event handler.
The only way I found to use this was to hook up two event handlers upon creation of my actor. Then in the handlers, I messages to Self and the Sender. For this, I must refer to some private fields of the actor from inside the event handlers. This feels wrong to me, but I cannot see another way out.
Is there a safer way to use multiple event handlers in an Actor?
Currently, my solution looks like this (_client is a WebClient instance created in the constructor of the actor):
public void HandleStartDownload(StartDownload message)
{
_self = Self;
_downloadRequestor = Sender;
_uri = message.Uri;
_guid = message.Guid;
_tempPath = Path.GetTempFileName();
_client.DownloadFileAsync(_uri, _tempPath);
}
private void Client_DownloadFileCompleted(object sender, System.ComponentModel.AsyncCompletedEventArgs e)
{
var completedMessage = new DownloadCompletedInternal(_guid, _tempPath);
_downloadRequestor.Tell(completedMessage);
_self.Tell(completedMessage);
}
private void Client_DownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e)
{
var progressedMessage = new DownloadProgressed(_guid, e.ProgressPercentage);
_downloadRequestor.Tell(progressedMessage);
_self.Tell(progressedMessage);
}
So when the download starts, some fields are set. Additionally, I make sure I Become a state where further StartDownload messages are stashed, until the DownloadCompleted message is received by Self:
public void Ready()
{
Receive<StartDownload>(message => {
HandleStartDownload(message);
Become(Downloading);
});
}
public void Downloading()
{
Receive<StartDownload>(message => {
Stash.Stash();
});
Receive<DownloadCompleted>(message => {
Become(Ready);
Stash.UnstashAll();
});
}
For reference, here's the entire Actor, but I think the important stuff is in this post directly: https://gist.github.com/AaronLenoir/4ce5480ecea580d5d283c5d08e8e71b5
I must refer to some private fields of the actor from inside the event
handlers. This feels wrong to me, but I cannot see another way out.
Is there a safer way to use multiple event handlers in an Actor?
There's nothing inherently wrong with an actor having internal state, and members that are part of that state raising events which are handled within the actor. No more wrong than this would be if taking an OO approach.
The only real concern is if that internal state gets mixed between multiple file download requests, but I think your current code is sound.
A possibly more palatable approach may be to look at the FileDownloadActor as a single use actor, fire it up, download the file, tell the result to the sender and then kill the actor. Starting up actors is a cheap operation, and this completely sidesteps the possibility of sharing the internal state between multiple download requests.
Unless of course you specifically need to queue downloads to run sequentially as your current code does - but the queue could be managed by another actor altogether and still treat the download actors as temporary.
I don't know if that is your case, but I see people treating Actors as micro services when they are simply objects. Remember Actors have internal state.
Now think about scalability, you can't scale messages to one Actor in a distributed Actor System. The messages you're sending to one Actor will be executed in the node executing that Actor.
If you want to execute download operations in parallel (for example), you do as Patrick said and create one Actor per download operation and that Actor can be executed in any available node.
Related
I have the following Actor where I am trying to restart and resend the failing message back to the actor :
public class BuildActor : ReceivePersistentActor
{
public override string PersistenceId => "asdad3333";
private readonly IActorRef _nextActorRef;
public BuildActor(IActorRef nextActorRef)
{
_nextActorRef = nextActorRef;
Command<Workload>(x => Build(x));
RecoverAny(workload =>
{
Console.WriteLine("Recovering");
});
}
public void Build(Workload Workload)
{
var context = Context;
var self = Self;
Persist(Workload, async x =>
{
//after this line executes
//application goes into break mode
//does not execute PreStart or Recover
var workload = await BuildTask(Workload);
_nextActorRef.Tell(workload);
context.Stop(self);
});
}
private Task<Workload> BuildTask(Workload Workload)
{
//works as expected if method made synchronous
return Task.Run(() =>
{
//simulate exception
if (Workload.ShowException)
{
throw new Exception();
}
return Workload;
});
}
protected override void PreRestart(Exception reason, object message)
{
if (message is Workload workload)
{
Console.WriteLine("Prestart");
workload.ShowException = false;
Self.Tell(message);
}
}
}
Inside the success handler of Persist I am trying to simulate an exception being thrown but on exception the application goes in to break mode and PreRestart hook is not invoked. But if I make BuildTask method synchronous by removing Task.Run then on exception both PreRestart and Recover<T> methods are invoked.
I would really appreciated if someone can point to me what should be the recommended pattern for this and where I am going wrong.
Most probably, Akka.Persistence is not the good solution for your problem here.
Akka.Persistence uses eventsourcing principles for storing actor's state. Few key points important in this context:
What you're sending to actor, is a command. It describes a job, you want to be done. Executing that command may result in doing some actual processing and eventually may lead to persist actor's linear state change history in form of the events.
In Akka.NET Persist method is used only to store events - they describe the fact, that something has happened: because of that, they cannot be denied and they cannot fail (a thing that you're doing in your Persist callback).
When an actor restarts at any point in time, it will always try to recreate its own state by replaying all events Persisted up to the last known point in time. For this reason it's important that Recover method should only focus on replaying actor's state (it can be called multiple times over the same event) and never result in side effects (example of side effect is sending an email). Any exception thrown there will mean, that actor state is irrecoverably corrupted and that actor will be killed.
If you want to resend the message to your actor, you could:
Put a reliable message queue (i.e. RabbitMQ or Azure Service Bus) or log (Kafka or Event Hub) in front of your actor processing pipeline. This is actually the most reasonable scenario in many cases.
Use at-least-once delivery semantics from Akka.Persistence - but IMHO only if for some reason you cannot use 1st solution.
The most simplistic and unreliable option (since messages are residing only in memory and never persisted) is dead letter queue. Every unhandled message is send there. You can subscribe to it and filter the incoming data to detect which messages should be send again to their recipients.
We are using the following method in a Stateful Service on Service-Fabric. The service has partitions. Sometimes we get a FabricNotReadableException from this peace of code.
public async Task HandleEvent(EventHandlerMessage message)
{
var queue = await StateManager.GetOrAddAsync<IReliableQueue<EventHandlerMessage>>(EventHandlerServiceConstants.EventHandlerQueueName);
using(ITransaction tx = StateManager.CreateTransaction())
{
await queue.EnqueueAsync(tx, message);
await tx.CommitAsync();
}
}
Does that mean that the partition is down and is being moved? Of that we hit a secondary partition? Because there is also a FabricNotPrimaryException that is being raised in some cases.
I have seen the MSDN link (https://msdn.microsoft.com/en-us/library/azure/system.fabric.fabricnotreadableexception.aspx). But what does
Represents an exception that is thrown when a partition cannot accept reads.
mean? What happened that a partition cannot accept a read?
Under the covers Service Fabric has several states that can impact whether a given replica can safely serve reads and writes. They are:
Granted (you can think of this as normal operation)
Not Primary
No Write Quorum (again mainly impacting writes)
Reconfiguration Pending
FabricNotPrimaryException which you mention can be thrown whenever a write is attempted on a replica which is not currently the Primary, and maps to the NotPrimary state.
FabricNotReadableException maps to the other states (you don't really need to worry or differentiate between them), and can happen in a variety of cases. One example is if the replica you are trying to perform the read on is a "Standby" replica (a replica which was down and which has been recovered, but there are already enough active replicas in the replica set). Another example is if the replica is a Primary but is being closed (say due to an upgrade or because it reported fault), or if it is currently undergoing a reconfiguration (say for example that another replica is being added). All of these conditions will result in the replica not being able to satisfy writes for a small amount of time due to certain safety checks and atomic changes that Service Fabric needs to handle under the hood.
You can consider FabricNotReadableException retriable. If you see it, just try the call again and eventually it will resolve into either NotPrimary or Granted. If you get FabricNotPrimary exception, generally this should be thrown back to the client (or the client in some way notified) that it needs to re-resolve in order to find the current Primary (the default communication stacks that Service Fabric ships take care of watching for non-retriable exceptions and re-resolving on your behalf).
There are two current known issues with FabricNotReadableException.
FabricNotReadableException should have two variants. The first should be explicitly retriable (FabricTransientNotReadableException) and the second should be FabricNotReadableException. The first version (Transient) is the most common and is probably what you are running into, certainly what you would run into in the majority of cases. The second (non-transient) would be returned in the case where you end up talking to a Standby replica. Talking to a standby won't happen with the out of the box transports and retry logic, but if you have your own it is possible to run into it.
The other issue is that today the FabricNotReadableException should be deriving from FabricTransientException, making it easier to determine what the correct behavior is.
Posted as an answer (to asnider's comment - Mar 16 at 17:42) because it was too long for comments! :)
I am also stuck in this catch 22. My svc starts and immediately receives messages. I want to encapsulate the service startup in OpenAsync and set up some ReliableDictionary values, then start receiving message. However, at this point the Fabric is not Readable and I need to split this "startup" between OpenAsync and RunAsync :(
RunAsync in my service and OpenAsync in my client also seem to have different Cancellation tokens, so I need to work around how to deal with this too. It just all feels a bit messy. I have a number of ideas on how to tidy this up in my code but has anyone come up with an elegant solution?
It would be nice if ICommunicationClient had a RunAsync interface that was called when the Fabric becomes ready/readable and cancelled when the Fabric shuts down the replica - this would seriously simplify my life. :)
I was running into the same problem. My listener was starting up before the main thread of the service. I queued the list of listeners needing to be started, and then activated them all early on in the main thread. As a result, all messages coming in were able to be handled and placed into the appropriate reliable storage. My simple solution (this is a service bus listener):
public Task<string> OpenAsync (CancellationToken cancellationToken)
{
string uri;
Start ();
uri = "<your endpoint here>";
return Task.FromResult (uri);
}
public static object lockOperations = new object ();
public static bool operationsStarted = false;
public static List<ClientAuthorizationBusCommunicationListener> pendingStarts = new List<ClientAuthorizationBusCommunicationListener> ();
public static void StartOperations ()
{
lock (lockOperations)
{
if (!operationsStarted)
{
foreach (ClientAuthorizationBusCommunicationListener listener in pendingStarts)
{
listener.DoStart ();
}
operationsStarted = true;
}
}
}
private static void QueueStart (ClientAuthorizationBusCommunicationListener listener)
{
lock (lockOperations)
{
if (operationsStarted)
{
listener.DoStart ();
}
else
{
pendingStarts.Add (listener);
}
}
}
private void Start ()
{
QueueStart (this);
}
private void DoStart ()
{
ServiceBus.WatchStatusChanges (HandleStatusMessage,
this.clientId,
out this.subscription);
}
========================
In the main thread, you call the function to start listener operations:
protected override async Task RunAsync (CancellationToken cancellationToken)
{
ClientAuthorizationBusCommunicationListener.StartOperations ();
...
This problem likely manifested itself here as the bus in question already had messages and started firing the second the listener was created. Trying to access anything in state manager was throwing the exception you were asking about.
I'm working on a really big project developed by two teams, one (mainly) for the database, and one (where I am) mainly for the GUI and helper classes as an interface between GUI and DB.
Obviously, there are errors in communication, and - of course - we can't assume 100Mbit bandwidth & super-fast server computer.
Language is C# .NET, target "framework" is WPF and Silverlight.
When a user clicks a button, the GUI asks the DB (through helper classes) for information. Let's say... pizza types. The server should answer "{Funghi,Frutti di mare,Prosciutto}". When DB sends his answer, we receive a "database.Ready" event and fill our datagrid.
BUT if the user clicks the button while we haven't received the answer yet, the GUI sends an another request to the database. And the whole system tries to serve the user.
But it can't, because when the second request is sent, the first is disposed when we want to read it. So NullReferenceExceptions occur.
I've solved this by implementing kind of a semaphore which closes when user input occurs and opens when the Ready event (the functions it calls) finishes working.
Problem:
If I don't receive the Ready event, no user input is allowed, but this is wrong.
Question:
Is there a common (or at least, working) solution to stop waiting for the Ready event and...
1) re-sending the request a few times, hoping we receive our pizza types?
AND/OR
2) Dropping the request, tell the user that database failed to send the answer, and re-open the semaphore?
I can't post code here as this code is the property of a corporation, I'd rather like to have theoretical solutions, which are okay for professionals too.
Sorry for the long post, and thank you for your answers!
I assume that you are already using a background thread to dispatch the query to the database and wait for it's response. You can use the Task API that was introduced in .NET 4.0 to cancel such a request. For that, you pass in a CancellationToken that signals the status to the executing task. You can obtain a CancellationToken via a CancellationTokenSource as shown in the following code:
public partial class MainWindow : Window
{
private readonly CancellationTokenSource _cancellationTokenSource = new CancellationTokenSource();
public MainWindow()
{
InitializeComponent();
}
private void Button_CallDatabase(object sender, RoutedEventArgs e)
{
Task.Factory.StartNew(CallDatabase, _cancellationTokenSource.Token);
}
private void Button_OnNavigate(object sender, RoutedEventArgs e)
{
// If you navigate, you can cancel the background task and thus
// it will not execute any further
_cancellationTokenSource.Cancel();
}
private void CallDatabase()
{
// This simulates a DB call
for (var i = 0; i < 50; i++)
{
Thread.Sleep(100);
}
// Check if cancellation was requested
if (_cancellationTokenSource.Token.IsCancellationRequested)
{
Debug.WriteLine("Request cancelled");
return;
}
Debug.WriteLine("Update Controls with DB infos.");
}
}
Note that this example is simplified, you can and should use this in another component (e.g. view model).
If you still want to use the Ready event, you could also just unregister from it when you navigate away, so that no further actions will be performed when it is raised.
I am fairly new to Rx and am having trouble finding a solution to my problem. I am using Rx to commence a download through a client library. Currently it looks like:
private void DownloadStuff(string descriptor, Action<Stuff> stuffAction)
{
this.stuffDownloader.GetStuffObservable(descriptor).Subscribe(x => stuffAction(x))
}
Where stuffDownloader is a wrapper around download logic defined in the client library. But I encountered a problem where I call DownloadStuff too much, causing many downloads, and overwhelming the system. Now what I would like to do is
private void DownloadStuff(string descriptor, Action<Stuff> stuffAction)
{
this.stuffDownloader.GetStuffObservable(descriptor)
.SlowSubscribe(TimeSpan.FromMilliSeconds(50))
.Subscribe(x => stuffAction(x))
}
Where SlowSubscribe is some combination of Rx actions to only subscribe on some interval.
Normally I would just put these DownloadStuff calls on a queue and pull them off on an interval, but I've been trying to do more through Rx lately. Three solutions occur to me:
This functionality exists and can be done all on the subscription side.
This is possible but the infrastructure of the downloader is incorrect and should change (i.e. stuffDownloader needs to behave differently)
This shouldn't be done with Rx, do it another way.
It occurs to me #2 is possible by passing an IObservable of descriptors to the client library and somehow slowing how the descriptors get onto the Observable.
You could in theory use Rx to treat your requests as events. This way you could leverage the serializing nature of Rx to queue up downloads.
I would think that you network layer (or stuffDownloader) would do this for you, but if you want to join me for a hack....this is what I have come up with (Yeehaw!!)
1.
Dont pass an Action, use Rx!! You are basically losing the error handling here and setting yourself up for weird unhandled exceptions.
private void DownloadStuff(string descriptor, Action<Stuff> stuffAction)
becomes
private IObservable<Stuff> DownloadStuff(string descriptor)
2.
Now we just have one method calling another. Seems pointless. Throw away the abstraction.
3.
Fix the underlying. To me the stuffDownloader is not doing it's job. Update the interface to take an IScheduler. Now you can pass it a dedicated EventLoopScheduler to enforce the serialization of the work
public IObservable<Stuff> GetStuffObservable(string descriptor, IScheduler scheduler)
4.
Fix the implementation?
As you want to serialize your requests (hmmmm....) you can just make the call synchronous.
private Stuff GetSync(string description)
{
var request = (HttpWebRequest)WebRequest.Create("http://se300328:90/");
var response =request.GetResponse();
var stuff = MapToStuff(response);
return stuff;
}
Now you just call that in you other method
public IObservable<Stuff> GetStuffObservable(string descriptor, ISchedulerLongRunning scheduler)
{
return Observable.Create<Stuff>(o=>
{
try
{
var stuff = GetStuff(description);
o.OnNext(stuff);
o.OnCompleted();
}
catch(Exception ex)
{
o.OnError(ex);
}
return Disposable.Empty(); //If you want to be sync, you cant cancel!
})
.SubscribeOn(scheduler);
}
However, having done all of this, I am sure this is not what you really want. I would expect that there is a problem somewhere else in the system.
Another alternative is to consider using the Merge operator and it's max concurent feature?
I have a web service with two methods:
RetrieveFirstLevelOptions() and RetrieveSecondLevelOptions(int levelOneOption).
The GUI contains two comboBoxes: FirstLevelOptionsCombo and SecondLevelOptionsCombo.
I am having trouble with creating a control flow for the initialization stage when I have to make a request to RetrieveFirstLevelOptions() and then, once the results are in, call RetrieveSecondLevelOptions(default levelOneOption = 0).
The problem is that since everything happens asynchronously I don't know what the best approach is to allow this behaviour take place once and only once, at the beginning.
An option I would love to have is to attach a second event handler to the RetieveFirstLevelOptionsCompleted event and have it remove itself after running only once. But it looks like such a behaviour is impossible to get.
Another option would be to have a boolean flag to indicate whether in Initialization phase and if it is, then the handler for RetrieveFirstLevelOptionsCompleted would execute some additional logic. However this idea makes it look like all event handlers would have to check for state information and do different things depending on the current state. This seems to be bad design because the control flow seems to be descentralized.
I want to have a centralized control flow mechanism that will make the decisions in one spot. But how can this be done when everything is executed asynchronously?
"An option I would love to have is to attach a second event handler to the RetieveFirstLevelOptionsCompleted event and have it remove itself after running only once. But it looks like such a behaviour is impossible to get."
Is there some reason this isn't working:
class Example
{
SomeWebService myService;
Example()
{
// do stuff
myService.RetrieveFirstLevelOptionsCompleted += MyHandlerMethod;
}
void MyHandlerMethod(object sender, RetrieveFirstLevelOptionsCompletedEventArgs e)
{
// do stuff
myService.RetrieveFirstLevelOptionsCompleted -= MyHandlerMethod;
// potentially attach next event handler for all subsequent calls
myService.RetrieveFirstLevelOptionsCompleted += MyHandlerMethod2;
}
}
The pattern that I usually use in a situation like this is to create a wrapper around the Async web service proxy method that accepts a callback method. The callback method then gets passed to the RetrieveFirstLevelOptionsAsync() method like so:
public void RetrieveFirstLevelOptions(Action callback)
{
client.RetrieveFirstLevelOptionsAsync(callback);
}
void client_RetrieveFirstLevelOptionsCompleted(object sender, AsyncCompletedEventArgs e)
{
var callback = e.UserState as Action;
if (callback != null)
{
callback();
}
}
So when you call RetrieveFirstLevelOptions(), you just pass the callback that you want to run only once, and you don't ever have to worry about it getting called multiple times. Presumably you'd put your call to RetrieveSecondLevelOptions() within that callback.