I'm using RavenDB as denormalized read model populated from domain events. I found problem, when two events (let's call them Created and Updated) are denormalized in the same time, loading document to be updated by Updated event occurs before saving changes made by Created event. I've came up with solution based on Changes API to wait for document creation:
public static T WaitAndLoad<T>(this IDocumentSession #this, ValueType id)
where T : class
{
var fullId = #this.Advanced.DocumentStore.Conventions.FindFullDocumentKeyFromNonStringIdentifier(id, typeof(T), false);
var ev = new ManualResetEvent(false);
var cancelation = new CancellationTokenSource();
#this.Advanced.DocumentStore
.Changes()
.ForDocument(fullId)
.Subscribe(change =>
{
if (change.Type == DocumentChangeTypes.Put)
{
ev.Set();
}
}, cancelation.Token);
try
{
var existing = #this.Load<T>(id);
if (existing != null)
{
return existing;
}
ev.WaitOne();
return #this.Load<T>(id);
}
finally
{
cancelation.Cancel();
}
}
Unfortunately second call to Load returns null because Id of document is already in knownMissingIds field in InMemoryDocumentSessionOperations and no request to server is made.
It there any other way to wait until document is created?
Well, I'm not sure what mechanism you are using for event processing, but I have been in a similar situation with something like NServiceBus. I don't think this exactly a RavenDB problem. You would probably have the same issue if you were writing to a SQL Server database.
The generalized problem is, Create and Update events are fired off, but they are received and processed in the wrong order. What to do?
Well the general advice is that your event handlers should be idempotent, and should retry when failed. So if Update is received first, it will throw an exception and be scheduled for retry. Then Create comes through, then Update retries and all is good.
Specifically blocking and waiting in the handler for the Update event is not advised, because if you had several of these then they could block all worker threads and the Create events would never come through.
Related
I have a synchronous method that calls a method which collates a bunch of data on a custom object and stores it on a table entry on a Firebird database, located on a server.
On the server, a monitoring process keeps watching the first table for new entries using a database event (a table trigger raises an event which is captured by the monitor). When this event is raised, that data is sent to a third-party black-box service to be processed with the use of a proprietary library, that takes between near-naught and 1 minute to reply.
The third-party service replies with some data which is entered on a second table on the database. This second table has another trigger that the client's program monitors. The client's program must either wait until the third-party replies some data, or it times out (the same 1 minute).
I'm currently delving into the world of database events and I've reached an impasse:
Currently I have a key press that runs a synchronous method, which according to an application setting either runs another synchronous method, which runs flawlessly, or another method that inserts an entry on a Firebird database. This database is monitored by another process, which reads that entry, do some stuff, and inserts the new data on another table.
Back on the main program, what I currently have is the method has an event handler which is triggered when the new data is inserted. However, as it is an event, the rest of the method runs its course, ending prematurely, before the event handler has the chance to read the new data.
In pseudo code:
MainWindow_KeyDown(object sender, EventArgs e)
{
if (e.Key == X)
{
MakeADecision()
}
}
MakeADecision()
{
if (Properties.Settings.Default.MySetting) Console.Write(DoLocalStuff());
else Console.Write(DoRemoteStuff());
}
string DoRemoteStuff()
{
using (OldDataTableAdapter)
using (NewDataTableAdapter)
{
OldDataTableAdapter.Insert(OldData);
var revent = new FBRemoteEvent(MyConnectionString);
revent.RemoteEventCounts += (sender, e) =>
{
NewDataTableAdapter.Fill(NewDataDataTable);
NewData = NewDataDataTable[0].MYCOLUMN;
};
revent.QueueEvents("MY_FB_EVENT");
}
return NewData;
}
As you can see, the issue here is that DoRemoteStuff reaches its return before the event can be triggered. I tried turning DoRemoteStuff() into an async method, but I don't know how to use events with async methods. Can anyone please help me with this? Any tips or hints on how to work with async methods?
A possible solution would be to use a TaskCompletionSource so you can convert your method to an async method. This is based on Is it possible to await an event instead of another async method?.
MakeADecision()
{
if (Properties.Settings.Default.MySetting)
{
Console.Write(DoLocalStuff());
}
else
{
// Consider making MakeADecision async as well
NewData = DoRemoteStuff().Result;
Console.Write(NewData);
}
}
async Task<string> DoRemoteStuff()
{
Task<string> task;
using (OldDataTableAdapter)
{
OldDataTableAdapter.Insert(OldData);
task = WaitForEvent(MyConnectionString);
}
return await task;
}
private async Task<string> WaitForEvent(string connectionString)
{
var taskCompletionSource = new TaskCompletionSource<string>();
var revent = new FbRemoteEvent(connectionString);
revent.RemoteEventCounts += (sender, e) =>
{
using (NewDataTableAdapter)
{
NewDataTableAdapter.Fill(NewDataDataTable);
string newData = NewDataDataTable[0].MYCOLUMN;
taskCompletionSource.SetResult(newData);
}
sender.Dispose();
};
revent.QueueEvents("MY_FB_EVENT");
return await taskCompletionSource.Task;
}
Some things to point out:
You need to explicitly dispose the event to avoid a memory leak
The using for NewDataTableAdapter belongs within the event handler
The MakeADecision method seems like a candidate to be made async as well
A word of warning, my C# is a bit rusty (and I have never done much with async), so I'm not sure if this is the idiomatic way of doing it. I also did not test the code as written above (I wrote and tested a simpler version, but I may have introduced bugs while transforming your code to a similar solution).
This solution may also have the possibility of a race condition between inserting the new data triggering the event and registering for the event (unless the Dispose at the end of the using block is what commits the data), consider moving the WaitForEvent before inserting. Also consider the possibility of receiving the event from an update done for/by another change.
I have the following Actor where I am trying to restart and resend the failing message back to the actor :
public class BuildActor : ReceivePersistentActor
{
public override string PersistenceId => "asdad3333";
private readonly IActorRef _nextActorRef;
public BuildActor(IActorRef nextActorRef)
{
_nextActorRef = nextActorRef;
Command<Workload>(x => Build(x));
RecoverAny(workload =>
{
Console.WriteLine("Recovering");
});
}
public void Build(Workload Workload)
{
var context = Context;
var self = Self;
Persist(Workload, async x =>
{
//after this line executes
//application goes into break mode
//does not execute PreStart or Recover
var workload = await BuildTask(Workload);
_nextActorRef.Tell(workload);
context.Stop(self);
});
}
private Task<Workload> BuildTask(Workload Workload)
{
//works as expected if method made synchronous
return Task.Run(() =>
{
//simulate exception
if (Workload.ShowException)
{
throw new Exception();
}
return Workload;
});
}
protected override void PreRestart(Exception reason, object message)
{
if (message is Workload workload)
{
Console.WriteLine("Prestart");
workload.ShowException = false;
Self.Tell(message);
}
}
}
Inside the success handler of Persist I am trying to simulate an exception being thrown but on exception the application goes in to break mode and PreRestart hook is not invoked. But if I make BuildTask method synchronous by removing Task.Run then on exception both PreRestart and Recover<T> methods are invoked.
I would really appreciated if someone can point to me what should be the recommended pattern for this and where I am going wrong.
Most probably, Akka.Persistence is not the good solution for your problem here.
Akka.Persistence uses eventsourcing principles for storing actor's state. Few key points important in this context:
What you're sending to actor, is a command. It describes a job, you want to be done. Executing that command may result in doing some actual processing and eventually may lead to persist actor's linear state change history in form of the events.
In Akka.NET Persist method is used only to store events - they describe the fact, that something has happened: because of that, they cannot be denied and they cannot fail (a thing that you're doing in your Persist callback).
When an actor restarts at any point in time, it will always try to recreate its own state by replaying all events Persisted up to the last known point in time. For this reason it's important that Recover method should only focus on replaying actor's state (it can be called multiple times over the same event) and never result in side effects (example of side effect is sending an email). Any exception thrown there will mean, that actor state is irrecoverably corrupted and that actor will be killed.
If you want to resend the message to your actor, you could:
Put a reliable message queue (i.e. RabbitMQ or Azure Service Bus) or log (Kafka or Event Hub) in front of your actor processing pipeline. This is actually the most reasonable scenario in many cases.
Use at-least-once delivery semantics from Akka.Persistence - but IMHO only if for some reason you cannot use 1st solution.
The most simplistic and unreliable option (since messages are residing only in memory and never persisted) is dead letter queue. Every unhandled message is send there. You can subscribe to it and filter the incoming data to detect which messages should be send again to their recipients.
I have an MVC3/.NET 4 application which uses Entity Framework (4.3.1 Code First)
I have wrapped EF into a Repository/UnitOfWork pattern as described here…
http://www.asp.net/mvc/tutorials/getting-started-with-ef-using-mvc/implementing-the-repository-and-unit-of-work-patterns-in-an-asp-net-mvc-application
Typically, as it explains in the article, when I require the creation of a new record I’ve been doing this…
public ActionResult Create(Course course)
{
unitOfWork.CourseRepository.Add(course);
unitOfWork.Save();
return RedirectToAction("Index");
}
However, when more than simply saving a record to a database is required I wrap the logic into what I’ve called an IService. For example…
private ICourseService courseService;
public ActionResult Create(Course course)
{
courseService.ProcessNewCourse(course);
return RedirectToAction("Index");
}
In one of my services I have something like the following…
public void ProcessNewCourse(Course course)
{
// Save the course to the database…
unitOfWork.CourseRepository.Add(course);
unitOfWork.Save();
// Generate a PDF that email some people about the new course being created, which requires more use of the unitOfWork…
var someInformation = unitOfWork.AnotherRepository.GetStuff();
var myPdfCreator = new PdfCreator();
IEnumerable<People> people = unitOfWork.PeopleRepository.GetAllThatWantNotifiying(course);
foreach(var person in people)
{
var message = “Hi ” + person.FullName;
var attachment = myPdfCreator.CreatePdf();
etc...
smtpClient.Send();
}
}
The above isn’t the actual code (my app has nothing to do with courses, I’m using view models, and I have separated the PDF creation and email message out into other classes) but the gist of what is going on is as above!
My problem is that the generation of the PDF and emailing it out is taking some time. The user just needs to know that the record has been saved to the database so I thought I would put the code below the unitOfWork.Save(); into an asynchronous method. The user can then be redirected and the server can happily take its time processing the emails, and attachments and whatever else I require it to do post save.
This is where I’m struggling.
I’ve tried a few things, the current being the following in ICourseService…
public class CourseService : ICourseService
{
private delegate void NotifyDelegate(Course course);
private NotifyDelegate notifyDelegate;
public CourseService()
{
notifyDelegate = new NotifyDelegate(this.Notify);
}
public void ProcessNewCourse(Course course)
{
// Save the course to the database…
unitOfWork.CourseRepository.Add(course);
unitOfWork.Save();
notifyDelegate.BeginInvoke(course);
}
private void Notify(Course course)
{
// All the stuff under unitOfWork.Save(); moved here.
}
}
My Questions/Problems
I’m randomly getting the error: "There is already an open DataReader associated with this Command which must be closed first." in the Notify() method.
Is it something to do with the fact that I’m trying to share the unitOrWork and therefore a dbContext across threads?
If so, can someone be kind enough to explain why this is a problem?
Should I be giving a new instance of unitOfWork to the Notify method?
Am I using the right patterns/classes to invoke the method asynchronously? Or should I be using something along the lines of....
new System.Threading.Tasks.Task(() => { Notify(course); }).Start();
I must say I've become very confused with the terms asynchronous, parallel, and concurrent!!
Any links to articles (c# async for idiots) would be appreciated!!
Many thanks.
UPDATE:
A little more digging got me to this SO page: https://stackoverflow.com/a/5491978/192999 which says...
"Be aware though that EF contexts are not thread safe, i.e. you cannot use the same context in more than one thread."
...so am I trying to achieve the impossible? Does this mean I should be creating a new IUnitOfWork instance for my new thread?
You could create a polling background thread that does the lengthy operation separately from your main flow. This thread could scan the database for new items (or items marked to process). This solution is pretty simple and ensures that jobs get done even if you application crashes (it will be picked up when the polling thread is started again).
You could also use a Synchronised Queue if it's not terrible if the request is 'lost', in the case your application crashes after the doc is requested and before it's generated/sent.
One thing is almost sure - as rikitikitik said - you will need to use a new unit of work, which means a separate transaction.
You could also look at Best threading queue example / best practice .
I'm using EventStore and things appear to be working, the event is stored and dispatched by my in-memory broker, the event is processed by my read model. But the "dispatched" flag in the EventStore Commits table is not getting set for some reason, so each time I restart my app it replays all of the events. I'm using SQL Server 2012 as the store. No errors are occurring. Any idea why this flag would not be getting set?
My code:
private static IStoreEvents BuildEventStore(IBroker broker)
{
return Wireup.Init()
.UsingSqlPersistence(Constants.EventStoreConnectionStringName)
.InitializeStorageEngine()
.UsingJsonSerialization()
.Compress()
.UsingAsynchronousDispatchScheduler()
.DispatchTo(new DelegateMessageDispatcher(c => DispatchCommit(broker, c)))
.Build();
}
private static void DispatchCommit(IBroker broker, Commit commit)
{
foreach (var #event in commit.Events)
{
if(!(#event.Body is IDomainEvent))
{
throw new InvalidOperationException("An event was published that is not an IDomainEvent: " + #event.Body.GetType());
}
broker.Publish((IDomainEvent)#event.Body);
}
}
UsingAsynchronousDispatchScheduler wires in a dispatcher that does its own thing out of the mainline of your processing. For example, if it has issues writing, the command processing thread is not going to hear about it.
One way of making life simplerdifferent is to change it to synchronous so your command processing chain gets to hear about the exceptions (remember that a dispatch exception doesn't roll back the fact that the command got processed and the event has now happened though).
But really you need to USL to see what the processor wired in by UsingAsynchronousDispatchScheduler does wrt reporting problems so that your monitoring can pick up the issue.
In a Windows Form window, multiple events can trigger an asynchronous method. This method downloads a file and caches it. My problem is that I want that method to be executed once. In other words, I want to prevent the file to be downloaded multiple times.
If the method downloading the file is triggered twice, I want the second call to wait for the file (or wait for the first method to be done).
Does someone have an idea on how to achieve that?
UPDATE: I am simply trying to prevent unnecessary downloads. In my case, when a client put its mouse over an item in a ListBox for more than a couple milliseconds, we start to download. We make the assumption that the user will click and request the file. What can potentially happen is that the user keeps his mouse over the item for one second and then click. In this case two downloads start. I am looking for the best way to handle such scenario.
UPDATE 2:: There is a possibility that the user will move its mouse over multiple items. In consequences, multiple downloads will occur. I've not really tough of this scenario, but right now if we face such scenario we don't abandon the download. The file will be downloaded (files are usually around 50-100kb) and then are going to be cached.
Maintain the state of what's happening in a form variable and have your async method check that state before it does anything. Make sure you synchronize access to it, though! Mutexes and semaphores are good for this kind of thing.
If you can download different files simultaneously, you'll need to keep track of what's being downloaded in a list for reference.
If only one file can be downloaded at a time, and you don't want to queue things up, you could just unhook the event while something is being downloaded, too, and rehook it when the download is complete.
Here is a dummy implementation that supports multiple file downloads:
Dictionary<string, object> downloadLocks = new Dictionary<string, object>();
void DownloadFile(string localFile, string url)
{
object fileLock;
lock (downloadLocks)
{
if (!downloadLocks.TryGetValue(url, out fileLock))
{
fileLock = new object();
downloadLocks[url] = fileLock;
}
}
lock (fileLock)
{
// check if file is already downloaded
// if not then download file
}
}
You can simply wrap your method call within a lock statement like this
private static readonly Object padLock = new Object();
...
lock(padLock)
{
YourMethod();
}
i'm not sure how it would be done in C#, but in java, you would synchonize on an private static final object in the class before downloading the file. This would block any further requests until the current one was completed. You could then check to see if the file was downloaded or not and act appropriately.
private static final Object lock = new Object();
private File theFile;
public method() {
synchronized(lock) {
if(theFile != null) {
//download the file
}
}
}
In general, I agree with Michael, use a lock around the code that actually gets the file. However, if there's a single event that always occurs first and you can always load the file then, consider using Futures. In the initial event, start the future running
Future<String> file = InThe.Future<String>(delegate { return LoadFile(); });
and in every other event, wait on the future's value
DoSomethingWith(file.Value);
If you want one thread to wait for another thread to finish a task, you probably want to use a ManualResetEvent. Maybe something like this:
private ManualResetEvent downloadCompleted = new ManualResetEvent();
private bool downloadStarted = false;
public void Download()
{
bool doTheDownload = false;
lock(downloadCompleted)
{
if (!downloadStarted)
{
downloadCompleted.Reset();
downloadStarted = true;
doTheDownload = true;
}
}
if (doTheDownload)
{
// Code to do the download
lock(downloadCompleted)
{
downloadStarted = false;
}
// When finished notify anyone waiting.
downloadCompleted.Set();
}
else
{
// Wait until it is done...
downloadCompleted.WaitOne();
}
}