I have the following code
AutoResetEvent instanceUnloaded = new AutoResetEvent(false);
WFStepsActivity workflow = WorkflowFactory.Current.BuildWorkflow(workflowinstance, requestinstance.RequestInstanceID);
WorkflowApplication wf = new WorkflowApplication(workflow);
wf.InstanceStore = WFInstanceStore.GetInstanceStore();
WFStepsPersistenceParticipant persist = new WFStepsPersistenceParticipant();
wf.Extensions.Add(persist);
wf.PersistableIdle = (arg) =>
{
return PersistableIdleAction.Unload;
};
wf.Unloaded = (arg) =>
{
log.Info("WFName_" + wfsetting.Name + "_Unloaded and return.");
instanceUnloaded.Set();
};
wf.OnUnhandledException = (arg) =>
{
return UnhandledExceptionAction.Abort;
};
wf.Completed = (arg) =>
{
if (arg.CompletionState == ActivityInstanceState.Closed)
{
persist.WorklflowCompleted = true;
}
};
wf.Run();
instanceUnloaded.WaitOne();
return persist.WorklflowCompleted;
where the WFStepsActivity is just the Acitivity structure that we build from settings.
The workflow currently ends when a bookmark is created.
context.CreateBookmark(context.WorkflowInstanceId.ToString(), new System.Activities.BookmarkCallback(OnResumeBookmark));
The issue is after the bookmark is created the wf.Completed is never triggered.
Would like to check if the Complete is triggered if Bookmark is created.
As I know Complete event will be trigger if u Cancel, Terminate
Isn't it supposed that once the workflow stop of bookmarked, this event will be triggered?
Bookmarks induce the workflow to become Idle, meaning that you must purposefully resume at the bookmark to continue and complete the workflow. First things first: Grok bookmarks.
I'm not sure you have supplied enough code to allow anyone to properly diagnose the precise issue (I can't see your bookmark creation, nor your supporting classes etc).
It is also possible that your workflow is throwing an exception, the details of which are being swallowed. If you want to catch and log any unhandled exceptions I would ammend your code thusly:
Exception error = null;
wf.OnUnhandledException = (arg) =>
{
error = arg.UnhandledException;
return UnhandledExceptionAction.Abort;
};
// and then after you've waited for completion...
if (error != null) {
throw error; // or handle it another way
}
Also useful: Configuring Tracking for a Workflow
Related
I have a data processing job that consists of about 20 sequential steps. The steps all fall under one of three categories:
do some file manipulation
import / export data from a database
make a call to a 3rd party web API
I've refactored the code from one long, awful looking method to a pipeline pattern, using examples here and here. All of the steps are TransformBlock, such as
var stepThirteenPostToWebApi = new TransformBlock<FileInfo, System.Guid>(async csv =>
{
dynamic task = await ApiUtils.SubmitData(csv.FullName);
return task.guid;
});
The code works most of the time, but occasionally a step in the pipeline fails for whatever reason - let's say a corrupt file can't be read in step 6 of 20 (just an example - any step could fail). The pipeline stops running further tasks, as it should.
However, the 3rd party web API introduces a challenge - we are charged for each job we initiate whether we execute all 20 steps or just the first one.
I would like to be able to fix whatever went wrong in the problem step (again, for our example let's say I fix the corrupt file in step 6 of 20), then pick back up at step 6. The 3rd party web API has a GUID for each job, and is asynchronous, so that should be fine - after the problem is fixed, it will happily let a job resume with remaining steps.
My question: Is it possible (and if so advisable?) to design a pipeline that could begin at any step, assuming the pre-requisites for that step were valid?
It would look something like:
job fails on step 6 and logs step 5 as the last successful step
a human comes along and fixes whatever caused step 6 to fail
a new pipeline is started at step 6
I realize a brute-force way would be to have StartAtStep2(), StartAtStep3(), StartAtStep4() methods. That doesn't seem like a good design, but I'm a bit new at this pattern so maybe that's acceptable.
The brute force way is not that bad, for example your above code would just need to be
bool StartAtStepThirteen(FileInfo csv)
{
return stepThirteenPostToWebApi.Post(csv);
}
The setup of the chain should be a separate method than the executing of the chain. You should save stepThirteenPostToWebApi in a class level variable in a class that represent's the entire chain, the setup of the chain could be done in the class's constructor.
Here is a simple 3 step version of the process. When a error happens instead of faulting the task chain I log the error and pass null along the chain for invalid entries. You could make that log method raise a event and then the user can decide what to do with the bad entry.
public class WorkChain
{
private readonly TransformBlock<string, FileInfo> stepOneGetFileInfo;
private readonly TransformBlock<FileInfo, System.Guid?> stepTwoPostToWebApi;
private readonly ActionBlock<System.Guid?> stepThreeDisplayIdToUser;
public WorkChain()
{
stepOneGetFileInfo = new TransformBlock<string, FileInfo>(new Func<string, FileInfo>(GetFileInfo));
stepTwoPostToWebApi = new TransformBlock<FileInfo, System.Guid?>(new Func<FileInfo, Task<Guid?>>(PostToWebApi));
stepThreeDisplayIdToUser = new ActionBlock<System.Guid?>(new Action<Guid?>(DisplayIdToUser));
stepOneGetFileInfo.LinkTo(stepTwoPostToWebApi, new DataflowLinkOptions() {PropagateCompletion = true});
stepTwoPostToWebApi.LinkTo(stepThreeDisplayIdToUser, new DataflowLinkOptions() {PropagateCompletion = true});
}
public void PostToStepOne(string path)
{
bool result = stepOneGetFileInfo.Post(path);
if (!result)
{
throw new InvalidOperationException("Failed to post to stepOneGetFileInfo");
}
}
public void PostToStepTwo(FileInfo csv)
{
bool result = stepTwoPostToWebApi.Post(csv);
if (!result)
{
throw new InvalidOperationException("Failed to post to stepTwoPostToWebApi");
}
}
public void PostToStepThree(Guid id)
{
bool result = stepThreeDisplayIdToUser.Post(id);
if (!result)
{
throw new InvalidOperationException("Failed to post to stepThreeDisplayIdToUser");
}
}
public void CompleteAdding()
{
stepOneGetFileInfo.Complete();
}
public Task Completion { get { return stepThreeDisplayIdToUser.Completion; } }
private FileInfo GetFileInfo(string path)
{
try
{
return new FileInfo(path);
}
catch (Exception ex)
{
LogGetFileInfoError(ex, path);
return null;
}
}
private async Task<Guid?> PostToWebApi(FileInfo csv)
{
if (csv == null)
return null;
try
{
dynamic task = await ApiUtils.SubmitData(csv.FullName);
return task.guid;
}
catch (Exception ex)
{
LogPostToWebApiError(ex, csv);
return null;
}
}
private void DisplayIdToUser(Guid? obj)
{
if(obj == null)
return;
Console.WriteLine(obj.Value);
}
}
I have added an attribute DisableConcurrentExecution(1) on the job, but all that does is delays the execution of second instance of a job until after first one is done. I want to be able to detect when a concurrent job has been run, and then cancel it all together.
I figured, if DisableConcurrentExecution(1) will prevent two instances of same recurrent job from running at the same, it will put the second job on "retry", thus changing it's State. So I added additional custom attribute on the job, which detects failed state, like so :
public class StopConcurrentTask : JobFilterAttribute, IElectStateFilter
{
public void OnStateElection(ElectStateContext context)
{
var failedState = context.CandidateState as FailedState;
if(failedState != null && failedState.Exception != null)
{
if(!string.IsNullOrEmpty(failedState.Exception.Message) && failedState.Exception.Message.Contains("Timeout expired. The timeout elapsed prior to obtaining a distributed lock on"))
{
}
}
}
}
This allows me to detect whether a job failed due to being run concurrently with another instance of same job. The problem is, I can't find a way to Cancel this specific failed job and remove it from being re-run. As it is now, the job will be put on retry schedule and Hangfire will attempt to run it a number of times.
I could of course put an attribute on the Job, ensuring it does not Retry at all. However, this is not a valid solution, because I want jobs to be Retried, except if they fail due to running concurrently.
You can prevent retry to happen if you put validation in OnPerformed method in IServerFilter interface.
Implementation :
public class StopConcurrentTask : JobFilterAttribute, IElectStateFilter, IServerFilter
{
// All failed after retry will be catched here and I don't know if you still need this
// but it is up to you
public void OnStateElection(ElectStateContext context)
{
var failedState = context.CandidateState as FailedState;
if (failedState != null && failedState.Exception != null)
{
if (!string.IsNullOrEmpty(failedState.Exception.Message) && failedState.Exception.Message.Contains("Timeout expired. The timeout elapsed prior to obtaining a distributed lock on"))
{
}
}
}
public void OnPerformed(PerformedContext filterContext)
{
// Do your exception handling or validation here
if (filterContext.Exception == null) return;
using (var connection = _jobStorage.GetConnection())
{
var storageConnection = connection as JobStorageConnection;
if (storageConnection == null)
return;
var jobId = filterContext.BackgroundJob.Id
// var job = storageConnection.GetJobData(jobId); -- If you want job detail
var failedState = new FailedState(filterContext.Exception)
{
Reason = "Your Exception Message or filterContext.Exception.Message"
};
using (var transaction = connection.GetConnection().CreateWriteTransaction())
{
transaction.RemoveFromSet("retries", jobId); // Remove from retry state
transaction.RemoveFromSet("schedule", jobId); // Remove from schedule state
transaction.SetJobState(jobId, failedState); // update status with failed state
transaction.Commit();
}
}
}
public void OnPerforming(PerformingContext filterContext)
{
// Do nothing
}
}
I hope this will help you.
I actually ended up using based on Jr Tabuloc answer - it will delete a job if it has been last executed 15 seconds ago - I noticed that time between server wake up and job execution varies. Usually it is in milliseconds, but since my jobs are executed once a day, I figured 15sec won't hurt.
public class StopWakeUpExecution : JobFilterAttribute, IServerFilter
{
public void OnPerformed(PerformedContext filterContext)
{
}
public void OnPerforming(PerformingContext filterContext)
{
using (var connection = JobStorage.Current.GetConnection())
{
var recurring = connection.GetRecurringJobs().FirstOrDefault(p => p.Job.ToString() == filterContext.BackgroundJob.Job.ToString());
TimeSpan difference = DateTime.UtcNow.Subtract(recurring.LastExecution.Value);
if (recurring != null && difference.Seconds < 15)
{
// Execution was due in the past. We don't want to automaticly execute jobs after server crash though.
var storageConnection = connection as JobStorageConnection;
if (storageConnection == null)
return;
var jobId = filterContext.BackgroundJob.Id;
var deletedState = new DeletedState()
{
Reason = "Task was due in the past. Please Execute manually if required."
};
using (var transaction = connection.CreateWriteTransaction())
{
transaction.RemoveFromSet("retries", jobId); // Remove from retry state
transaction.RemoveFromSet("schedule", jobId); // Remove from schedule state
transaction.SetJobState(jobId, deletedState); // update status with failed state
transaction.Commit();
}
}
}
}
}
First I'm totally new to WorkFlow Foundation 4.5. We use the WF engine for managing States in our Case entities. Instead of building our own state machine we decided to use WF. Mostly due to the fact that our customer have large process flows (not that complicated) that we wanted to draw in xaml. Easy for everybody to actually understand the process and talk about it.
The problem is that our transitions to the end state, Final state, results in that StateMachineStateTracker instance is null when we load it. This code below works perfectly for all transitions and we can load up tracker instance after resuming bookmark to see what the new current state is.
private void ConfigureWorkflowApplication(WorkflowApplication wfApp, SqlWorkflowInstanceStore store)
{
wfApp.InstanceStore = store;
var tracker = new StateMachineStateTracker(wfApp.WorkflowDefinition);
wfApp.Extensions.Add(tracker);
wfApp.Extensions.Add(new StateTrackerPersistenceProvider(tracker));
wfApp.Completed = delegate { Debug.WriteLine("Workflow completed."); };
wfApp.Aborted =
delegate(WorkflowApplicationAbortedEventArgs e)
{
Debug.WriteLine("Workflow Aborted. Exception: {0}\r\n{1}", e.Reason.GetType().FullName,
e.Reason.Message);
};
wfApp.OnUnhandledException = delegate(WorkflowApplicationUnhandledExceptionEventArgs e)
{
Debug.WriteLine("Unhandled Exception: {0}\r\n{1}", e.UnhandledException.GetType().FullName,
e.UnhandledException.Message);
return UnhandledExceptionAction.Terminate;
};
wfApp.PersistableIdle = delegate { return PersistableIdleAction.Unload; };
}
Code above instantiate a WorkFlowApplication instance.
protected bool Execute(Activity process, Case #case, string transition)
{
WorkflowApplicationInstance instance = null;
using (var store = new DisposableStore())
{
instance = WorkflowApplication.GetInstance(#case.InstanceId, store.Store);
var wfApp = new WorkflowApplication(process, WorkflowIdentity);
ConfigureWorkflowApplication(wfApp, store.Store);
var trackerInstance = StateMachineStateTracker.LoadInstance(#case.InstanceId, wfApp.WorkflowDefinition,
_connectionString);
if (!trackerInstance.Transitions.Any(x => x.DisplayName.Equals(transition))) return false;
}
using (var store = new DisposableStore())
{
var wfApp = new WorkflowApplication(process, instance.DefinitionIdentity);
ConfigureWorkflowApplication(wfApp, store.Store);
wfApp.Load(#case.InstanceId);
var sync = new AutoResetEvent(false);
wfApp.ResumeBookmark(transition, null);
wfApp.Unloaded = x => sync.Set();
sync.WaitOne();
// Set case to new state
var trackerInstance = StateMachineStateTracker.LoadInstance(#case.InstanceId, wfApp.WorkflowDefinition,
_connectionString);
#case.ChangeToNewState(trackerInstance.CurrentState);
}
return true;
}
The code above have the intention to Make a transition from one State to next (string transition) and we also want to set the new state to our Case class.
This fails when we want to do this from our State before Final state. No exception. No Logging in output window. Nothing. Just that the row
var trackerInstance = StateMachineStateTracker.LoadInstance(#case.InstanceId, wfApp.WorkflowDefinition,
_connectionString);
Returns null. Is this due to the fact that you cannot Load StateMachineStateTracker with an instance that is on final state (not sure if it actually reaches final state).
Do anybody have any clue about the problem? I have a feeling that this is something basic that we've forgot.
Ok. I found the problem. It was as I though. Not a problem. Just me being rookie on WF.
When you do transitions to final state, WF remove all data about case since its finished. This means SurrogateInstance is removed from DB and state machine state tracker are of course not working since there is no case to load into tracker.
BUT however I did a delegate method on Completed event for WorkflowApplication that can handle closure and finalize our case.
wfApp.Completed = delegate(WorkflowApplicationCompletedEventArgs e)
{
Debug.WriteLine("Workflow completed.");
Debug.WriteLine("State" + e.CompletionState);
if (e.CompletionState == ActivityInstanceState.Closed)
{
_caseIsCompleted = true;
}
};
I'll try to explain the issue with a simplified console application example, however the real project is a ASP.NET MVC3 application.
Having the following tables:
imagine the following scenario:
user creates a report (a line in TestReport, where Text is the report string content, and Ready is a bool flag, saying, if the report is ready to be processed); by default Ready is set to false, i.e. not ready.
user wants the report to be processed, so he submits it; Ready is set to true here.
The system gives an opportunity to recall the report back, if it has not been processed yet. So, when the report is recalled, Ready is set to false back. On the contrary, when the report is processed, a line in TestReportRef, referencing report by its Id, is created.
Now imagine that at one and the same moment
user wants to recall the report;
the report is added to the process list;
As soon as this can happen simultaneously, errors may occur. That is the report will have Ready == false and it'll be referenced in TestReportRef.
Here is a simple console example of how this may happen:
var dc = new TestDataContext('my connection string');
dc.TestReport.InsertOnSubmit(new TestReport
{
Text = "My report content",
Ready = true //ready at once
});
dc.SubmitChanges();
Action recallReport = () =>
{
var _dc = new TestDataContext(cs);
var report = _dc.TestReport.FirstOrDefault(t => t.Ready);
if (report != null && !report.TestReportRef.Any())
{
Thread.Sleep(1000);
report.Ready = false;
_dc.SubmitChanges();
}
};
Action acceptReport = () =>
{
var _dc = new TestDataContext(cs);
var report = _dc.TestReport.FirstOrDefault(t => t.Ready);
if (report != null && !report.TestReportRef.Any())
{
Thread.Sleep(1000);
_dc.TestReportRef.InsertOnSubmit(new TestReportRef
{
FK_ReportId = report.Id
});
_dc.SubmitChanges();
}
};
var task1 = new Task(recallReport);
var task2 = new Task(acceptReport);
task1.Start();
task2.Start();
task1.Wait();
task2.Wait();
foreach (var t in dc.TestReport)
{
Console.WriteLine(string.Format("{0}\t{1}\t{2}", t.Id, t.Text, t.Ready));
}
foreach (var t in dc.TestReportRef)
{
Console.WriteLine("ref id:\t" + t.FK_ReportId);
}
Thread.Sleep(1000); is added to be ensure, that tasks will check one and the same situation.
The given example may sound awkward, however, I hope, it should explain the issue I'm dealing with.
How can I avoid this? Making the repository singleton doesn't seem to be a good idea. Shall I use some shared mutex (one for all web requests) to separate write-operations only?
Or is there a pattern I should use in this kind of scenario?
This is only a simplified example of one of the scenarios I have. However, there are several scenarios in which it may run into a similar discrepancy. The best thing would be to make this kind of intersection impossible, I guess.
Why don't add a version column on the Report table? Task starts by tracking current version,when task end, if the version is the same that the tracked one, operation is ok, otherwise fail. If operation appear ok, update the version to version +1. This is a sort of optimistic lock; that implicitly suppose that conflicts may occur, but they are not so frequent.
UPDATE
If you are using linqto sql maybe you can have a check at the parameter UpdateCheck [Column(UpdateCheck=UpdateCheck.Always)]
This can be useful to handle concurrency in your case.
I have a long-running asynchronous task that is kicked off from an ASP.NET MVC4 web page. The controller method looks like this:
[HttpPost]
public ActionResult Index(IndexModel model)
{
if (ModelState.IsValid)
{
try
{
model.NotificationRecipient = model.NotificationRecipient.Replace(';', ',');
ImportConfiguration config = new ImportConfiguration()
{
BatchId = model.BatchId,
ReportRecipients = model.NotificationRecipient.Split(',').Select(c => c.Trim())
};
System.Threading.ThreadPool.QueueUserWorkItem(foo => LaunchFileImporter(config, this.HttpContext.ApplicationInstance.Context));
if (model.RunExport) ThreadPool.QueueUserWorkItem(foo => LaunchFileExporter());
Log.InfoFormat("Queued the ImportProcessor to process invoices. Send Notification: {0} Email Recipient: {1}",
model.SendNotification, model.NotificationRecipient);
TempData["message"] = "The import processor job has been started.";
//return RedirectToAction("Index", "Home");
}
catch (Exception ex)
{
Log.Error("Failed to properly queue the invoice import job.", ex);
ModelState.AddModelError("", ex.Message);
}
}
var dirInfo = new System.IO.DirectoryInfo(dir);
model.Files = dirInfo.EnumerateFiles("*.xml").OrderBy(x => x.Name.ToLower());
return View(model);
}
My LaunchFileImporter method looks like this:
private void LaunchFileImporter(ImportConfiguration config, System.Web.HttpContext context)
{
//the semaphore prevents concurrent running of this process, which can cause contention.
Log.Trace(t => t("submitter semaphore: {0}", (exporter == null) ? "NULL" : "present."));
submitter.WaitOne();
try
{
Log.Trace(t => t("Context: {0}", context));
using (var processor = new ImportProcessor(context))
{
processor.OnFileProcessed += new InvoiceFileProcessing(InvoiceFileProcessingHandler);
processor.OnInvoiceProcessed += new InvoiceSubmitted(InvoiceSubmittedHandler);
processor.Execute(config);
}
}
catch (Exception ex)
{
Log.Error("Failed in execution of the File Importer.", ex);
}
submitter.Release();
}
My Logger is a Common.Logging private static readonly ILog, and is configured for NLog. It seems properly wired up; at least, I get a fair amount of logs out of it.
Here's the thing: The moment I hit System.Threading.ThreadPool.QueueUserWorkItem, the application pool death spirals into a silent death, resetting the app pool, reloading the membership provider, reprocessing the web.config, the whole shebang... No YSOD, no indication on the web page... everything just quietly blows up. The last log entry I get is Queued the ImportProcessor to process invoices....
I should note the page does the refresh. The TempData["message"] is populated and displayed on the screen, which makes me believe the problem is happening in the asynchronous process... but pretty much immediately. Due to the lack of additional logs I am assuming there is a problem with the logger.
So I'm hoping someone can either tell me what is happening, point to some documented issue with this, tell me how I'm being an idiot, or reproduce something similar as a bug.
Thanks!
UPDATE
#RichardDeeming pointed out that the context information was not getting into the spawned thread, and this seemed to be the cause of the problem. I still haven't wrapped my brain around why this didn't work nor did it write the trace messages, but once I captured the part of the context that I needed, the IPrincipal, and used that instead of the context object, it just worked.
You'll get a NullReferenceException in the line:
ThreadPool.QueueUserWorkItem(foo => LaunchFileImporter(config, HttpContext.ApplicationInstance.Context));
The HttpContext gets cleaned up once the request has completed. Since the exception is thrown on a background thread, it will bring down the whole AppDomain, causing your application to restart.
You need to capture the relevant state from the context in the controller action, and use that state in the WaitCallback delegate:
IPrincipal user = Context.User;
ThreadPool.QueueUserWorkItem(foo => LaunchFileImporter(config, user));
// Or:
// ThreadPool.QueueUserWorkItem(state => LaunchFileImporter(config, (IPrincipal)state);