I have added an attribute DisableConcurrentExecution(1) on the job, but all that does is delays the execution of second instance of a job until after first one is done. I want to be able to detect when a concurrent job has been run, and then cancel it all together.
I figured, if DisableConcurrentExecution(1) will prevent two instances of same recurrent job from running at the same, it will put the second job on "retry", thus changing it's State. So I added additional custom attribute on the job, which detects failed state, like so :
public class StopConcurrentTask : JobFilterAttribute, IElectStateFilter
{
public void OnStateElection(ElectStateContext context)
{
var failedState = context.CandidateState as FailedState;
if(failedState != null && failedState.Exception != null)
{
if(!string.IsNullOrEmpty(failedState.Exception.Message) && failedState.Exception.Message.Contains("Timeout expired. The timeout elapsed prior to obtaining a distributed lock on"))
{
}
}
}
}
This allows me to detect whether a job failed due to being run concurrently with another instance of same job. The problem is, I can't find a way to Cancel this specific failed job and remove it from being re-run. As it is now, the job will be put on retry schedule and Hangfire will attempt to run it a number of times.
I could of course put an attribute on the Job, ensuring it does not Retry at all. However, this is not a valid solution, because I want jobs to be Retried, except if they fail due to running concurrently.
You can prevent retry to happen if you put validation in OnPerformed method in IServerFilter interface.
Implementation :
public class StopConcurrentTask : JobFilterAttribute, IElectStateFilter, IServerFilter
{
// All failed after retry will be catched here and I don't know if you still need this
// but it is up to you
public void OnStateElection(ElectStateContext context)
{
var failedState = context.CandidateState as FailedState;
if (failedState != null && failedState.Exception != null)
{
if (!string.IsNullOrEmpty(failedState.Exception.Message) && failedState.Exception.Message.Contains("Timeout expired. The timeout elapsed prior to obtaining a distributed lock on"))
{
}
}
}
public void OnPerformed(PerformedContext filterContext)
{
// Do your exception handling or validation here
if (filterContext.Exception == null) return;
using (var connection = _jobStorage.GetConnection())
{
var storageConnection = connection as JobStorageConnection;
if (storageConnection == null)
return;
var jobId = filterContext.BackgroundJob.Id
// var job = storageConnection.GetJobData(jobId); -- If you want job detail
var failedState = new FailedState(filterContext.Exception)
{
Reason = "Your Exception Message or filterContext.Exception.Message"
};
using (var transaction = connection.GetConnection().CreateWriteTransaction())
{
transaction.RemoveFromSet("retries", jobId); // Remove from retry state
transaction.RemoveFromSet("schedule", jobId); // Remove from schedule state
transaction.SetJobState(jobId, failedState); // update status with failed state
transaction.Commit();
}
}
}
public void OnPerforming(PerformingContext filterContext)
{
// Do nothing
}
}
I hope this will help you.
I actually ended up using based on Jr Tabuloc answer - it will delete a job if it has been last executed 15 seconds ago - I noticed that time between server wake up and job execution varies. Usually it is in milliseconds, but since my jobs are executed once a day, I figured 15sec won't hurt.
public class StopWakeUpExecution : JobFilterAttribute, IServerFilter
{
public void OnPerformed(PerformedContext filterContext)
{
}
public void OnPerforming(PerformingContext filterContext)
{
using (var connection = JobStorage.Current.GetConnection())
{
var recurring = connection.GetRecurringJobs().FirstOrDefault(p => p.Job.ToString() == filterContext.BackgroundJob.Job.ToString());
TimeSpan difference = DateTime.UtcNow.Subtract(recurring.LastExecution.Value);
if (recurring != null && difference.Seconds < 15)
{
// Execution was due in the past. We don't want to automaticly execute jobs after server crash though.
var storageConnection = connection as JobStorageConnection;
if (storageConnection == null)
return;
var jobId = filterContext.BackgroundJob.Id;
var deletedState = new DeletedState()
{
Reason = "Task was due in the past. Please Execute manually if required."
};
using (var transaction = connection.CreateWriteTransaction())
{
transaction.RemoveFromSet("retries", jobId); // Remove from retry state
transaction.RemoveFromSet("schedule", jobId); // Remove from schedule state
transaction.SetJobState(jobId, deletedState); // update status with failed state
transaction.Commit();
}
}
}
}
}
Related
I've been working on a hobby project being developed in C# + Xamarin Forms + Prism + EF Core + Sqlite, debugging in UWP app.
I've written the following code to store tick data received from broker to Sqlite.
First, the OnTick call back that receives the ticks (approx. 1 tick per sec per instrument):
private void OnTick(Tick tickData)
{
foreach (var instrument in IntradayInstruments.Where(i => i.InstrumentToken == tickData.InstrumentToken))
{
instrument.UpdateIntradayCandle(tickData);
}
}
And the UpdateIntradayCandle method is:
public void UpdateIntradayCandle(Tick tick)
{
if (LastIntradayCandle != null)
{
if (LastIntradayCandle.Open == 0m)
{
LastIntradayCandle.Open = tick.LastPrice;
}
if (LastIntradayCandle.High < tick.LastPrice)
{
LastIntradayCandle.High = tick.LastPrice;
}
if (LastIntradayCandle.Low == 0m)
{
LastIntradayCandle.Low = tick.LastPrice;
}
else if (LastIntradayCandle.Low > tick.LastPrice)
{
LastIntradayCandle.Low = tick.LastPrice;
}
LastIntradayCandle.Close = tick.LastPrice;
}
}
The LastIntradayCandle is a property:
object _sync = new object();
private volatile IntradayCandle _lastIntradayCandle;
public IntradayCandle LastIntradayCandle
{
get
{
lock (_sync)
{
return _lastIntradayCandle;
}
}
set
{
lock (_sync)
{
_lastIntradayCandle = value;
}
}
}
Now, the LastIntradayCandle is changed periodically, say, 5 minutes, and a new candle is put in place for updating, from a different thread coming from a System.Threading.Timer which is scheduled to run every 5m.
public void AddNewIntradayCandle()
{
if (LastIntradayCandle != null)
{
LastIntradayCandle.IsClosed = true;
}
var newIntradayCandle = new IntradayCandle { Open = 0m, High = 0m, Low = 0m, Close = 0m };
LastIntradayCandle = newIntradayCandle;
IntradayCandles.Add(newIntradayCandle);
}
Now, the problem is, I'm getting 0s in those Open, High or Low but not in Close, Open having the most number of zeroes. This is happening very randomly.
I'm thinking that if any of the Open, High, Low or Close values is getting updated, it means the tick is having a value to be grabbed, but somehow one or more assignments in UpdateIntradayCandle method are not running. Having zeroes is a strict NO for the purpose of the app.
I'm neither formally trained as a programmer nor an expert, but a self-learning hobbyist and definitely never attempted at multi-threading before.
So, I request you to please point me what I am doing wrong, or better still, what should I be doing to make it work.
Multithreading and EF Core is not compatible things. EF Core context is not a thread safe. You have to create new context for each thread. Also making your object thread safe is wasting time.
So, schematically you have to do the following and you can remove locks from your object.
private void OnTick(Tick tickData)
{
using var ctx = new MyDbContext(...);
foreach (var instrument in ctx.IntradayInstruments.Where(i => i.InstrumentToken == tickData.InstrumentToken))
{
instrument.UpdateIntradayCandle(tickData);
}
ctx.SaveChanges();
}
I'm working with Quartz scheduler, and attempting to close all jobs on shutdown of the app. I have one specialty job that does a 'Hold' or 'Busy-wait', basically until it gets a condition it sits there waiting patiently.
This job is new, due to a new integration point. The app is run as a service using Topshelf, and whenever we try to shutdown the service to upgrade it, now that this job is running, we have to end up restarting the server to get it to shutdown.
Anyhow, here it gets weird, I have a single jobtype, and when i try to trigger the interrupt in the following section of code using the jobs FireInstanceId or JobKey:
_logger.InfoFormat("{0} scheduler interrupting listener", scheduler.SchedulerName);
scheduler.Interrupt(ListenerKeys.Realtime);
_logger.InfoFormat("{0} scheduler shutting down", scheduler.SchedulerName);
scheduler.Shutdown(true);
_logger.InfoFormat("{0} scheduler shut down", scheduler.SchedulerName);
I get an exception:
Job 'Listeners.Realtime' can not be interrupted, since it does not implement Quartz.IInterruptableJob
One would assume this is straight forward. However Here is the ONLY job that uses this job key:
ListenerJob : BaseJob, IInterruptableJob
{
// some other code referenced in ExecuteJob
public void Interrupt()
{
_dequeuer.StopDequeing();
}
}
I'd go out on a limb and say that's how you implement it, so my question becomes: is there a known bug in Quartz? Is there an issue with group-keys and interrupts maybe? Is there just a way to tell the scheduler to interrupt ALL jobs that are interruptable? Is there an alternative?
UPDATE
I decided to run the following code for more diagnostics from below answers. var interfaces does in fact include IInterruptableJob
var jobs = scheduler.GetCurrentlyExecutingJobs().Where(x => Equals(x.JobDetail.Key, ListenerKeys.Realtime));
var job1 = jobs.First();
var interfaces = job1.JobDetail.JobType.GetInterfaces();
Additionally, I ran ReportInterruptableJob as suggested below, which checked the assembly and confirmed ListenerJob implements the interface.
UPDATE2:
Ok, I went out to git hub, and ran the exact meshos. Job.JobInstance as IInterruptableInterface returns null, which is why I get the error. What I don't understand I guess, I how the JobInstance is formed around the IJo which does implement IInterruptableJob
UPDATE3: Ok.... So I found something in the bootstrap that is using JobWrapper<>. I know nothing about it, but Im sure that is part of it.
So I agree with the other Answer (C Knight) that the JobKey may be off.
If you've implemented the interface...and you have the right JobKey..then you should not get that exception.
Below is the code I have for the interrupting a job. I'm trying to find the "findthekey" logic as well.
private static void InterruptAJob(JobKey foundJobKey, IScheduler sched)
{
if (null != foundJobKey)
{
sched.Interrupt(foundJobKey);
}
}
APPEND
Here is my code for finding a job key.
I would start with it..........put some breakpoints...and see if your JobKey is in the collection. Maybe modify the routine (after you figure out the magic-places) to find a job key by a certain criteria..and throw an exception if you don't find what you're expecting to find. Aka, don't "assume" the JobKey exists.....but query for it and throw the appropriate exception (or write the appropriate == null logic if no match is found).......vs the "assume it has to be there" approach.
Again, the below is "starter" code.......my Proof of Concept only had one job in it, so I didn't have to be specific.
private static JobKey FindaJobKey(IScheduler sched, ILogger logger)
{
JobKey returnJobKey = null;
IList<string> jobGroupNames = sched.GetJobGroupNames();
if (null != jobGroupNames)
{
if (jobGroupNames.Count > 0)
{
GroupMatcher<JobKey> groupMatcher = GroupMatcher<JobKey>.GroupEquals(jobGroupNames.FirstOrDefault());
Quartz.Collection.ISet<JobKey> keys = sched.GetJobKeys(groupMatcher);
returnJobKey = keys.FirstOrDefault();
if (null == returnJobKey)
{
throw new ArgumentOutOfRangeException("No JobKey Found");
}
}
}
Thread.Sleep(TimeSpan.FromSeconds(1));
return returnJobKey;
}
APPEND:
Maybe something like this:
private static JobKey FindJobKey(IScheduler sched, ILogger logger, string jobGroupName)
{
JobKey returnJobKey = null;
IList<string> jobGroupNames = sched.GetJobGroupNames();
if (null != jobGroupNames)
{
string matchingJobGroupName = jobGroupNames.Where(s => s.Equals(jobGroupName, StringComparison.OrdinalIgnoreCase)).FirstOrDefault();
if (null != matchingJobGroupName)
{
GroupMatcher<JobKey> groupMatcher = GroupMatcher<JobKey>.GroupEquals(matchingJobGroupName);
Quartz.Collection.ISet<JobKey> keys = sched.GetJobKeys(groupMatcher);
if (null != keys)
{
if (keys.Count > 0)
{
throw new ArgumentOutOfRangeException(string.Format("More than one JobKey Found. (JobGroupName='{0}')", jobGroupName));
}
returnJobKey = keys.FirstOrDefault();
if (null != returnJobKey)
{
throw new ArgumentOutOfRangeException(string.Format("No JobKey Found. (JobGroupName='{0}')", jobGroupName));
}
}
}
}
Thread.Sleep(TimeSpan.FromSeconds(1));
return returnJobKey;
}
Another quick and dirty "look at what you got going on" method.
private static void ShowJobs(IScheduler sched)
{
Console.WriteLine("");
Console.WriteLine("ShowJobs : Start");
GroupMatcher<JobKey> matcherAll = GroupMatcher<JobKey>.AnyGroup();
Quartz.Collection.ISet<JobKey> jobKeys = sched.GetJobKeys(matcherAll);
foreach (JobKey jk in jobKeys)
{
Console.WriteLine(string.Format("{0} : {1}", jk.Group, jk.Name));
}
Console.WriteLine("ShowJobs : End");
Console.WriteLine("");
}
APPEND:
Maybe come at it another way. I slightly adjusted one method. But added a new one.
ReportIInterruptableJobs
See what ReportIInterruptableJobs reports.
private static void ShowJobs(IScheduler sched, ILogger logger)
{
Console.WriteLine("");
Console.WriteLine("ShowJobs : Start");
GroupMatcher<JobKey> matcherAll = GroupMatcher<JobKey>.AnyGroup();
Quartz.Collection.ISet<JobKey> jobKeys = sched.GetJobKeys(matcherAll);
foreach (JobKey jk in jobKeys)
{
Console.WriteLine(string.Format("{0} : {1}", jk.Group, jk.Name));
IJobDetail jobData = sched.GetJobDetail(jk);
if (null != jobData)
{
Console.WriteLine(string.Format("{0}", jobData.JobType.AssemblyQualifiedName));
}
}
Console.WriteLine("ShowJobs : End");
Console.WriteLine("");
}
private static void ReportIInterruptableJobs()
{
Type typ = typeof(IInterruptableJob);
ICollection<Type> types = AppDomain.CurrentDomain.GetAssemblies()
.SelectMany(s => s.GetTypes())
.Where(p => typ.IsAssignableFrom(p)).ToList();
if (null != types)
{
foreach (Type t in types)
{
Console.WriteLine(string.Format("{0}", t.AssemblyQualifiedName));
}
}
}
Are you 100% positive that the JobKey for your ListenerJob is in fact Listeners.Realtime? Could there be a different job that uses that JobKey? Could Listeners.Realtime be being updated to a different value by the time you call Interrupt()? Quartz is definitely finding a job with that JobKey, because Quartz does not throw an exception if it doesn't find a job at all. The job it IS finding does not implement IInterruptableJob. I would double check the JobKey value for your ListenerJob by setting a breakpoint in the Execute(IJobExecutionContext context) method of ListenerJob and inspecting the context.JobDetail.Key. My money is on the JobKey being different. Also, it might be helpful for us to see what the code for your BaseJob is.
I have the following code:
public class EmailJobQueue
{
private EmailJobQueue()
{
}
private static readonly object JobsLocker = new object();
private static readonly Queue<EmailJob> Jobs = new Queue<EmailJob>();
private static readonly object ErroredIdsLocker = new object();
private static readonly List<long> ErroredIds = new List<long>();
public static EmailJob GetNextJob()
{
lock (JobsLocker)
{
lock (ErroredIdsLocker)
{
// If there are no jobs or they have all errored then get some new ones - if jobs have previously been skipped then this will re get them
if (!Jobs.Any() || Jobs.All(j => ErroredIds.Contains(j.Id)))
{
var db = new DBDataContext();
foreach (var emailJob in db.Emailing_SelectSend(1))
{
// Dont re add jobs that exist
if (Jobs.All(j => j.Id != emailJob.Id) && !ErroredIds.Contains(emailJob.Id))
{
Jobs.Enqueue(new EmailJob(emailJob));
}
}
}
while (Jobs.Any())
{
var curJob = Jobs.Dequeue();
// Check the job has not previously errored - if they all have then eventually we will exit the loop
if (!ErroredIds.Contains(curJob.Id))
return curJob;
}
return null;
}
}
}
public static void ReInsertErrored(long id)
{
lock (ErroredIdsLocker)
{
ErroredIds.Add(id);
}
}
}
I then start 10 threads which do this:
var email = EmailJobQueue.GetNextJob();
if (email != null)
{
// Breakpoint here
}
The thing is that if I put a breakpoint where the comment is and add one item to the queue then the breakpoint gets hit multiple times. Is this an issue with my code or a peculiarity with VS debugger?
Thanks,
Joe
It appears as if you are getting your jobs from the database:
foreach (var emailJob in db.Emailing_SelectSend(1))
Is that database call marking the records as unavailable for section in future queries? If not, I believe that's why you're hitting the break point multiple times.
For example, if I replace that call to the database with the following, I see your behavior.
// MockDB is a static configured as `MockDB.Enqueue(new EmailJob{Id = 1})`
private static IEnumerable<EmailJob> GetJobFromDB()
{
return new List<EmailJob>{MockDB.Peek()};
}
However, if I actually Dequeue from the mock db, it only hits the breakpoint once.
private static IEnumerable<EmailJob> GetJobFromDB()
{
var list = new List<EmailJob>();
if (MockDB.Any())
list.Add(MockDB.Dequeue());
return list;
}
This is a side effect of debugging a multi-threaded piece of your application.
You are seeing the breakpoint being hit on each thread. Debugging a multi-threaded piece of the application is tricky because you're actually debugging all threads at the same time. In fact, at times, it will jump between classes while you're stepping through because it's doing different things on all of those threads, depending on your application.
Now, to address whether or not it's thread-safe. That really depends on how you're using the resources on those threads. If you're just reading, it's likely that it's thread-safe. But if you're writing, you'll need to leverage at least the lock operation on shared objects:
lock (someLockObject)
{
// perform the write operation
}
All users but a specific one should not be allowed to edit or update Quote Details which not satisfy a specific condition, but they have to be capable of revising the Quote if they want to.
The issue is, revising the Quote (i.e. the user clicks the "Revise" button in an Active form record) triggers the Update of the Quote Details and I can't figure out how to recognize what's going on.
My current attempt is based on a plugin which code looks like this:
public class PreQuoteProductUpdate : Plugin
{
// I work with CRM Developer Tools to build plugins
// This goes in Update Message, Pre-Operation, Server Only, pre-image called "preImage"
protected void ExecutePreQuoteProductUpdate(LocalPluginContext localContext)
{
if (localContext == null)
{
throw new ArgumentNullException("localContext");
}
IPluginExecutionContext context = localContext.PluginExecutionContext;
IOrganizationService srv = localContext.OrganizationService;
Entity preImageEntity = (context.PreEntityImages != null && context.PreEntityImages.Contains(this.preImageAlias)) ? context.PreEntityImages[this.preImageAlias] : null;
try
{
PluginBody(context, srv, preImageEntity);
}
catch (Exception ex)
{
throw new InvalidPluginExecutionException("Quote Details Pre-Update", ex);
}
}
protected void PluginBody(IPluginExecutionContext context, IOrganizationService srv, Entity preImage)
{
if(IsRevising()) return;
CheckSomeCondition(context, srv);
if (preImage.Attributes.ContainsKey("ica_daconfigurazione") && preImage.GetAttributeValue<bool>("ica_daconfigurazione"))
{
CheckUser(context, srv);
}
}
protected void IsRevising()
{
// I have no clue about the logic to put here: see below.
}
protected void CheckSomeCondition(IPluginExecutionContext context, IOrganizationService srv)
{
var entity = (Entity)context.InputParameters["Target"];
// if some fields of entity contain some specific data, throw
// this always happens
}
protected void CheckUser(IPluginExecutionContext context, IOrganizationService srv)
{
//allowedUser is read from a configuration entity
var allowedUser = new Guid();
if (context.InitiatingUserId.Equals(serviceUser.Id) == false)
throw new InvalidPluginExecutionException("Can't edit quote details");
}
}
I know that (in a Quote plugin) I can know a revision is ongoing by checking the ParentContext, is anything similar available in a QuoteDetail plugin ? I gave it a try but all I get are NullReferenceExceptions thrown at me.
Should I expect to have State/Status available to check ?
For any more info which I may have overlooked, just ask.
Register on the Pre Create message (stage 20) of QuoteDetail and filter on the parent context not being for Quote. If it is, just return (effectively doing nothing).
The same applies to the Update message of the QuoteDetail.
Both messages run in the context of the ReviseQuote message for Quote.
var parentContext = context.ParentContext;
// While there is a parent context...
while (parentContext != null) {
// When parent context is for "quote", return;
if (parentContext.PrimaryEntityName == "quote")
{
return;
}
// Assign parent's parent context to loop parent context.
parentContext = parentContext.ParentContext;
}
This is driving me nuts. I'm not sure what else to try. This is my latest attempt to update a list of objects within another object using EF 4.3.
The scenario is that a user has added a new Task to an Application that already has one task in its Tasks property. The Application is not attached to the DB context because it was retrieved in a prior logic/DB call. This is the class and property:
public class Application : EntityBase
{
public ObservableCollection<TaskBase> Tasks { // typical get/set code here }
}
This is my attempt to update the list. What happens is that the new Task gets added and the association correctly exists in the DB. However, the first task, that wasn't altered, has its association removed in the DB (its reference to the Application).
This is the Save() method that takes the Application that the user modified:
public void Save(Application newApp)
{
Application appFromContext;
appFromContext = this.Database.Applications
.Include(x => x.Tasks)
.Single(x => x.IdForEf == newApp.IdForEf);
AddTasksToApp(newApp, appFromContext);
this.Database.SaveChanges();
}
And this is the hooey that's apparently necessary to save using EF:
private void AddTasksToApp(Application appNotAssociatedWithContext, Application appFromContext)
{
List<TaskBase> originalTasks = appFromContext.Tasks.ToList();
appFromContext.Tasks.Clear();
foreach (TaskBase taskModified in appNotAssociatedWithContext.Tasks)
{
if (taskModified.IdForEf == 0)
{
appFromContext.Tasks.Add(taskModified);
}
else
{
TaskBase taskBase = originalTasks.Single(x => x.IdForEf == taskModified.IdForEf); // Get original task
this.Database.Entry(taskBase).CurrentValues.SetValues(taskModified); // Update with new
}
}
}
Can anyone see why the first task would be losing its association to the Application in the DB? That first task goes through the else block in the above code.
Next, I'll need to figure out how to delete one or more items, but first things first...
After continual trial and error, this appears to be working, including deleting Tasks. I thought I'd post this in case it helps someone else. I'm also hoping that someone tells me that I'm making this more complicated than it should be. This is tedious and error-prone code to write when saving every object that has a list property.
private void AddTasksToApp(Application appNotAssociatedWithContext, Application appFromContext)
{
foreach (TaskBase taskModified in appNotAssociatedWithContext.Tasks)
{
if (taskModified.IdForEf == 0)
{
appFromContext.Tasks.Add(taskModified);
}
else
{
TaskBase taskBase = appFromContext.Tasks.Single(x => x.IdForEf == taskModified.IdForEf); // Get original task
this.Database.Entry(taskBase).CurrentValues.SetValues(taskModified); // Update with new
}
}
// Delete tasks that no longer exist within the app.
List<TaskBase> tasksToDelete = new List<TaskBase>();
foreach (TaskBase originalTask in appFromContext.Tasks)
{
TaskBase task = appNotAssociatedWithContext.Tasks.Where(x => x.IdForEf == originalTask.IdForEf).FirstOrDefault();
if (task == null)
{
tasksToDelete.Add(originalTask);
}
}
foreach (TaskBase taskToDelete in tasksToDelete)
{
appFromContext.Tasks.Remove(taskToDelete);
this.Database.TaskBases.Remove(taskToDelete);
}
}