I have a data processing job that consists of about 20 sequential steps. The steps all fall under one of three categories:
do some file manipulation
import / export data from a database
make a call to a 3rd party web API
I've refactored the code from one long, awful looking method to a pipeline pattern, using examples here and here. All of the steps are TransformBlock, such as
var stepThirteenPostToWebApi = new TransformBlock<FileInfo, System.Guid>(async csv =>
{
dynamic task = await ApiUtils.SubmitData(csv.FullName);
return task.guid;
});
The code works most of the time, but occasionally a step in the pipeline fails for whatever reason - let's say a corrupt file can't be read in step 6 of 20 (just an example - any step could fail). The pipeline stops running further tasks, as it should.
However, the 3rd party web API introduces a challenge - we are charged for each job we initiate whether we execute all 20 steps or just the first one.
I would like to be able to fix whatever went wrong in the problem step (again, for our example let's say I fix the corrupt file in step 6 of 20), then pick back up at step 6. The 3rd party web API has a GUID for each job, and is asynchronous, so that should be fine - after the problem is fixed, it will happily let a job resume with remaining steps.
My question: Is it possible (and if so advisable?) to design a pipeline that could begin at any step, assuming the pre-requisites for that step were valid?
It would look something like:
job fails on step 6 and logs step 5 as the last successful step
a human comes along and fixes whatever caused step 6 to fail
a new pipeline is started at step 6
I realize a brute-force way would be to have StartAtStep2(), StartAtStep3(), StartAtStep4() methods. That doesn't seem like a good design, but I'm a bit new at this pattern so maybe that's acceptable.
The brute force way is not that bad, for example your above code would just need to be
bool StartAtStepThirteen(FileInfo csv)
{
return stepThirteenPostToWebApi.Post(csv);
}
The setup of the chain should be a separate method than the executing of the chain. You should save stepThirteenPostToWebApi in a class level variable in a class that represent's the entire chain, the setup of the chain could be done in the class's constructor.
Here is a simple 3 step version of the process. When a error happens instead of faulting the task chain I log the error and pass null along the chain for invalid entries. You could make that log method raise a event and then the user can decide what to do with the bad entry.
public class WorkChain
{
private readonly TransformBlock<string, FileInfo> stepOneGetFileInfo;
private readonly TransformBlock<FileInfo, System.Guid?> stepTwoPostToWebApi;
private readonly ActionBlock<System.Guid?> stepThreeDisplayIdToUser;
public WorkChain()
{
stepOneGetFileInfo = new TransformBlock<string, FileInfo>(new Func<string, FileInfo>(GetFileInfo));
stepTwoPostToWebApi = new TransformBlock<FileInfo, System.Guid?>(new Func<FileInfo, Task<Guid?>>(PostToWebApi));
stepThreeDisplayIdToUser = new ActionBlock<System.Guid?>(new Action<Guid?>(DisplayIdToUser));
stepOneGetFileInfo.LinkTo(stepTwoPostToWebApi, new DataflowLinkOptions() {PropagateCompletion = true});
stepTwoPostToWebApi.LinkTo(stepThreeDisplayIdToUser, new DataflowLinkOptions() {PropagateCompletion = true});
}
public void PostToStepOne(string path)
{
bool result = stepOneGetFileInfo.Post(path);
if (!result)
{
throw new InvalidOperationException("Failed to post to stepOneGetFileInfo");
}
}
public void PostToStepTwo(FileInfo csv)
{
bool result = stepTwoPostToWebApi.Post(csv);
if (!result)
{
throw new InvalidOperationException("Failed to post to stepTwoPostToWebApi");
}
}
public void PostToStepThree(Guid id)
{
bool result = stepThreeDisplayIdToUser.Post(id);
if (!result)
{
throw new InvalidOperationException("Failed to post to stepThreeDisplayIdToUser");
}
}
public void CompleteAdding()
{
stepOneGetFileInfo.Complete();
}
public Task Completion { get { return stepThreeDisplayIdToUser.Completion; } }
private FileInfo GetFileInfo(string path)
{
try
{
return new FileInfo(path);
}
catch (Exception ex)
{
LogGetFileInfoError(ex, path);
return null;
}
}
private async Task<Guid?> PostToWebApi(FileInfo csv)
{
if (csv == null)
return null;
try
{
dynamic task = await ApiUtils.SubmitData(csv.FullName);
return task.guid;
}
catch (Exception ex)
{
LogPostToWebApiError(ex, csv);
return null;
}
}
private void DisplayIdToUser(Guid? obj)
{
if(obj == null)
return;
Console.WriteLine(obj.Value);
}
}
Related
I have some code that loads up and AppDomain(call it domain) calling an object function within the domain. The purpose is to get a list of items from a usb device using the device API to retrieve the information. The API requires a callback to return the information.
var AppDomain.CreateDomain(
$"BiometricsDomain{System.IO.Path.GetRandomFileName()}");
var proxy = domain.CreateInstanceAndUnwrap(proxy.Assembly.FullName, proxy.FullName
?? throw new InvalidOperationException()) as Proxy;
var ids = obj.GetIdentifications();
The proxy code loaded into the domain is as follows
public class Proxy : MarshalByRefObject
{
public List<String> GetIdentifications()
{
var control = new R100DeviceControl();
control.OnUserDB += Control_OnUserDB;
control.Open();
int nResult = control.DownloadUserDB(out int count);
// need to be able to return the list here but obviously that is not
// going to work.
}
private void Control_OnUserDB(List<String> result)
{
// Get the list of string from here
}
}
Is there a way to be able to wait on the device and return the information as needed when the callback is called? Since the GetIdentifications() has already returned I don't know how to get the
You can consider wrapping the Event-Based Asynchronous Pattern (EAP) operations as one task by using a TaskCompletionSource<TResult> so that the event can be awaited.
public class Proxy : MarshalByRefObject {
public List<String> GetIdentifications() {
var task = GetIdentificationsAsync();
return task.Result;
}
private Task<List<String>> GetIdentificationsAsync() {
var tcs = new TaskCompletionSource<List<string>>();
try {
var control = new R100DeviceControl();
Action<List<string>> handler = null;
handler = result => {
// Once event raised then set the
// Result property on the underlying Task.
control.OnUserDB -= handler;//optional to unsubscribe from event
tcs.TrySetResult(result);
};
control.OnUserDB += handler;
control.Open();
int count = 0;
//call async event
int nResult = control.DownloadUserDB(out count);
} catch (Exception ex) {
//Bubble the error up to be handled by calling client
tcs.TrySetException(ex);
}
// Return the underlying Task. The client code
// waits on the Result property, and handles exceptions
// in the try-catch block there.
return tcs.Task;
}
}
You can also improve on it by adding the ability to cancel using a CancellationToken for longer than expected callbacks.
With that the proxy can then be awaited
List<string> ids = proxy.GetIdentifications();
Reference How to: Wrap EAP Patterns in a Task
NOTE: Though there may be more elegant solutions to the problem of asynchronous processing, the fact that this occurs in a child AppDomain warrants child AppDomain best practices. (see links below)
i.e.
do not allow code meant for a child AppDomain to be executed in the parent domain
do not allow complex types to bubble to the parent AppDomain
do not allow exceptions to cross AppDomain boundaries in the form of custom exception types
OP:
I am using it for fault tolerance
First I would probably add a Open or similar method to give time for the data to materialise.
var proxy = domain.CreateInstanceAndUnwrap(proxy.Assembly.FullName, proxy.FullName
?? throw new InvalidOperationException()) as Proxy;
proxy.Open(); // <------ new method here
.
. some time later
.
var ids = obj.GetIdentifications();
Then in your proxy make these changes to allow for data processing to occur in the background so that by the time you call GetNotifications data may be ready.
public class Proxy : MarshalByRefObject
{
ConcurrentBag<string> _results = new ConcurrentBag<string>();
public void Open()
{
var control = new R100DeviceControl();
control.OnUserDB += Control_OnUserDB;
control.Open();
// you may need to store nResult and count in a field?
nResult = control.DownloadUserDB(out int count);
}
public List<String> GetIdentifications()
{
var copy = new List<string>();
while (_results.TryTake(out var x))
{
copy.Add(x);
}
return copy;
}
private void Control_OnUserDB(List<String> result)
{
// Get the list of string from here
_results.Add (result);
}
}
Now you could probably improve upon GetNotifications to accept a timeout in the event either GetNotifications is called before data is ready or if you call it multiply but before subsequent data to arrive.
More
How to: Run Partially Trusted Code in a Sandbox
Not sure why you just don't maintain a little state and then wait for the results in the call:
public class Proxy : MarshalByRefObject
{
bool runningCommand;
int lastResult;
R100DeviceControl DeviceControl { get{ if(deviceControl == null){ deviceControl = new R100DeviceControl(); deviceControl.OnUserDB += Control_OnUserDB; } return deviceControl; } }
public List<String> GetIdentifications()
{
if(runningCommand) return null;
DeviceControl.Open();
runningCommand = true;
lastResult = control.DownloadUserDB(out int count);
}
private void Control_OnUserDB(List<String> result)
{
runningCommand = false;
// Get the list of string from here
}
}
Once you have a pattern like this you can easily switch between async and otherwise whereas before it will look a little harder to understand because you integrated the async logic, this way you can implement the sync method and then make an async wrapper if you desire.
I have some issues with GeckoFX file download.
I tried the LauncherDialog method described in a few locations such as How to handle downloading in GeckoFX 29, and a WebRequest / Webclient method.
Both method work, but they do two requests to the server. The first one will trigger the LauncherDialog.Download event, and the LauncherDialog will make a new request to get the actual file.
I use GeckoFX in a custom web client for a customer, and in this particular case, this download request requires several seconds of processing on server side and modifies the data state. The second request is delayed and the returned data isn't the same as for the first request.
Also, this particular application doesn't need a download progress window of any sorts.
Is there any way to get the data stream from the initial request? Modifying GeckoFx-Winforms is not a problem. I would prefer avoiding any modification to GeckoFX-Core, but I'll do it if required.
Well, I revisited my wrong assumptions about XPCOM programming, took a look at selected locations in Firefox/Gecko source code and found one solution. Which could be very obvious for someone with some XPCOM/XUL programming experience, but was not initially for me. So I guess sharing my solution could help a few people.
In my case, the LauncherDialog method is definitely not the way to go.
Instead, I implemented the nsIFactory, nsIExternalHelperAppService and nsIStreamListener interfaces.
nsiStreamListener
internal class MyStreamListener : nsIStreamListener
{
public MyStreamListener(/*...*/) { }
public void OnStartRequest(nsIRequest aRequest, nsISupports aContext)
{
// This will get called once, when the download "begins".
// You can initialize your things here.
}
public void OnStopRequest(nsIRequest aRequest, nsISupports aContext, int aStatusCode)
{
// This will also get called once, when the download is
// complete or interrupted. You can perform the post-download
// actions here.
if (aStatusCode != GeckoError.NS_OK) {
// download interrupted
}
else {
// download completed
}
}
public void OnDataAvailable(nsIRequest aRequest, nsISupports aContext, nsIInputStream aInputStream, ulong aOffset, uint aCount)
{
// This gets called several times with small chunks of data.
// Do what you need with the stream. In my case, I read it
// in a small buffer, which then gets written to an output
// filestream (not shown).
// The aOffset parameter is the sum of all previously received data.
var lInput = InputStream.Create(aInputStream);
byte[] lBuffer = new byte[aCount];
lInput.Read(lBuffer, 0, (int)aCount);
}
}
nsIExternalHelperAppService
public class MyExternalHelperAppService : nsIExternalHelperAppService
{
public MyExternalHelperAppService(/* ... */)
{
/* ... */
}
public nsIStreamListener DoContent(nsACStringBase aMimeContentType, nsIRequest aRequest, nsIInterfaceRequestor aWindowContext, bool aForceSave)
{
var request = Request.CreateRequest(aRequest);
var lChannel = request as HttpChannel;
try {
if (lChannel != null) {
var uri = lChannel.OriginalUri;
var contentType = lChannel.ContentType;
var contentLength = lChannel.ContentLength;
var dispositionFilename = lChannel.ContentDispositionFilename;
// Do your contenttype validation, keeping only what you need.
// Make sure you clean dispositionFilename before using it.
// If you don't want to do anything with that file, you can return null;
return new MyStreamListener(/* ... */);
}
}
catch (COMException) {
/* ... */
}
return null;
}
}
nsIFactory (you can also overload GenericOneClassNsFactory<TFactory,TType>):
public IntPtr CreateInstance(nsISupports aOuter, ref Guid iid)
{
// This is called when the content dispatcher gets a DISPOSITION_ATTACHMENT
// on the channel, or when it doesn't have any builtin handler
// for the content type. It needs an external helper to handle
// the content, so it creates one and calls DoContent on it.
MyExternalHelperAppService _myExternalHelperAppService = new MyExternalHelperAppService(...);
IntPtr result;
IntPtr iUnknownForObject = Marshal.GetIUnknownForObject(_myExternalHelperAppService);
Marshal.QueryInterface(iUnknownForObject, ref iid, out result);
Marshal.Release(iUnknownForObject);
return result;
}
public void LockFactory(bool #lock) {
// do nothing here, it's not used, only kept for backwards compatibility.
}
Then, somewhere in my initialization code, I registered my nsIFactory with the proper contract:
Xpcom.RegisterFactory(typeof(MyExternalHelperAppService).GUID,
"MyExternalHelperAppService",
"#mozilla.org/uriloader/external-helper-app-service;1",
new MyNsFactory());
And that's all.
I'm working with Quartz scheduler, and attempting to close all jobs on shutdown of the app. I have one specialty job that does a 'Hold' or 'Busy-wait', basically until it gets a condition it sits there waiting patiently.
This job is new, due to a new integration point. The app is run as a service using Topshelf, and whenever we try to shutdown the service to upgrade it, now that this job is running, we have to end up restarting the server to get it to shutdown.
Anyhow, here it gets weird, I have a single jobtype, and when i try to trigger the interrupt in the following section of code using the jobs FireInstanceId or JobKey:
_logger.InfoFormat("{0} scheduler interrupting listener", scheduler.SchedulerName);
scheduler.Interrupt(ListenerKeys.Realtime);
_logger.InfoFormat("{0} scheduler shutting down", scheduler.SchedulerName);
scheduler.Shutdown(true);
_logger.InfoFormat("{0} scheduler shut down", scheduler.SchedulerName);
I get an exception:
Job 'Listeners.Realtime' can not be interrupted, since it does not implement Quartz.IInterruptableJob
One would assume this is straight forward. However Here is the ONLY job that uses this job key:
ListenerJob : BaseJob, IInterruptableJob
{
// some other code referenced in ExecuteJob
public void Interrupt()
{
_dequeuer.StopDequeing();
}
}
I'd go out on a limb and say that's how you implement it, so my question becomes: is there a known bug in Quartz? Is there an issue with group-keys and interrupts maybe? Is there just a way to tell the scheduler to interrupt ALL jobs that are interruptable? Is there an alternative?
UPDATE
I decided to run the following code for more diagnostics from below answers. var interfaces does in fact include IInterruptableJob
var jobs = scheduler.GetCurrentlyExecutingJobs().Where(x => Equals(x.JobDetail.Key, ListenerKeys.Realtime));
var job1 = jobs.First();
var interfaces = job1.JobDetail.JobType.GetInterfaces();
Additionally, I ran ReportInterruptableJob as suggested below, which checked the assembly and confirmed ListenerJob implements the interface.
UPDATE2:
Ok, I went out to git hub, and ran the exact meshos. Job.JobInstance as IInterruptableInterface returns null, which is why I get the error. What I don't understand I guess, I how the JobInstance is formed around the IJo which does implement IInterruptableJob
UPDATE3: Ok.... So I found something in the bootstrap that is using JobWrapper<>. I know nothing about it, but Im sure that is part of it.
So I agree with the other Answer (C Knight) that the JobKey may be off.
If you've implemented the interface...and you have the right JobKey..then you should not get that exception.
Below is the code I have for the interrupting a job. I'm trying to find the "findthekey" logic as well.
private static void InterruptAJob(JobKey foundJobKey, IScheduler sched)
{
if (null != foundJobKey)
{
sched.Interrupt(foundJobKey);
}
}
APPEND
Here is my code for finding a job key.
I would start with it..........put some breakpoints...and see if your JobKey is in the collection. Maybe modify the routine (after you figure out the magic-places) to find a job key by a certain criteria..and throw an exception if you don't find what you're expecting to find. Aka, don't "assume" the JobKey exists.....but query for it and throw the appropriate exception (or write the appropriate == null logic if no match is found).......vs the "assume it has to be there" approach.
Again, the below is "starter" code.......my Proof of Concept only had one job in it, so I didn't have to be specific.
private static JobKey FindaJobKey(IScheduler sched, ILogger logger)
{
JobKey returnJobKey = null;
IList<string> jobGroupNames = sched.GetJobGroupNames();
if (null != jobGroupNames)
{
if (jobGroupNames.Count > 0)
{
GroupMatcher<JobKey> groupMatcher = GroupMatcher<JobKey>.GroupEquals(jobGroupNames.FirstOrDefault());
Quartz.Collection.ISet<JobKey> keys = sched.GetJobKeys(groupMatcher);
returnJobKey = keys.FirstOrDefault();
if (null == returnJobKey)
{
throw new ArgumentOutOfRangeException("No JobKey Found");
}
}
}
Thread.Sleep(TimeSpan.FromSeconds(1));
return returnJobKey;
}
APPEND:
Maybe something like this:
private static JobKey FindJobKey(IScheduler sched, ILogger logger, string jobGroupName)
{
JobKey returnJobKey = null;
IList<string> jobGroupNames = sched.GetJobGroupNames();
if (null != jobGroupNames)
{
string matchingJobGroupName = jobGroupNames.Where(s => s.Equals(jobGroupName, StringComparison.OrdinalIgnoreCase)).FirstOrDefault();
if (null != matchingJobGroupName)
{
GroupMatcher<JobKey> groupMatcher = GroupMatcher<JobKey>.GroupEquals(matchingJobGroupName);
Quartz.Collection.ISet<JobKey> keys = sched.GetJobKeys(groupMatcher);
if (null != keys)
{
if (keys.Count > 0)
{
throw new ArgumentOutOfRangeException(string.Format("More than one JobKey Found. (JobGroupName='{0}')", jobGroupName));
}
returnJobKey = keys.FirstOrDefault();
if (null != returnJobKey)
{
throw new ArgumentOutOfRangeException(string.Format("No JobKey Found. (JobGroupName='{0}')", jobGroupName));
}
}
}
}
Thread.Sleep(TimeSpan.FromSeconds(1));
return returnJobKey;
}
Another quick and dirty "look at what you got going on" method.
private static void ShowJobs(IScheduler sched)
{
Console.WriteLine("");
Console.WriteLine("ShowJobs : Start");
GroupMatcher<JobKey> matcherAll = GroupMatcher<JobKey>.AnyGroup();
Quartz.Collection.ISet<JobKey> jobKeys = sched.GetJobKeys(matcherAll);
foreach (JobKey jk in jobKeys)
{
Console.WriteLine(string.Format("{0} : {1}", jk.Group, jk.Name));
}
Console.WriteLine("ShowJobs : End");
Console.WriteLine("");
}
APPEND:
Maybe come at it another way. I slightly adjusted one method. But added a new one.
ReportIInterruptableJobs
See what ReportIInterruptableJobs reports.
private static void ShowJobs(IScheduler sched, ILogger logger)
{
Console.WriteLine("");
Console.WriteLine("ShowJobs : Start");
GroupMatcher<JobKey> matcherAll = GroupMatcher<JobKey>.AnyGroup();
Quartz.Collection.ISet<JobKey> jobKeys = sched.GetJobKeys(matcherAll);
foreach (JobKey jk in jobKeys)
{
Console.WriteLine(string.Format("{0} : {1}", jk.Group, jk.Name));
IJobDetail jobData = sched.GetJobDetail(jk);
if (null != jobData)
{
Console.WriteLine(string.Format("{0}", jobData.JobType.AssemblyQualifiedName));
}
}
Console.WriteLine("ShowJobs : End");
Console.WriteLine("");
}
private static void ReportIInterruptableJobs()
{
Type typ = typeof(IInterruptableJob);
ICollection<Type> types = AppDomain.CurrentDomain.GetAssemblies()
.SelectMany(s => s.GetTypes())
.Where(p => typ.IsAssignableFrom(p)).ToList();
if (null != types)
{
foreach (Type t in types)
{
Console.WriteLine(string.Format("{0}", t.AssemblyQualifiedName));
}
}
}
Are you 100% positive that the JobKey for your ListenerJob is in fact Listeners.Realtime? Could there be a different job that uses that JobKey? Could Listeners.Realtime be being updated to a different value by the time you call Interrupt()? Quartz is definitely finding a job with that JobKey, because Quartz does not throw an exception if it doesn't find a job at all. The job it IS finding does not implement IInterruptableJob. I would double check the JobKey value for your ListenerJob by setting a breakpoint in the Execute(IJobExecutionContext context) method of ListenerJob and inspecting the context.JobDetail.Key. My money is on the JobKey being different. Also, it might be helpful for us to see what the code for your BaseJob is.
I'm looking for some advice on writing some unit tests for the code below. Implementation aside (it's not my code, but I've been tasked to retroactively write some tests for it) could someone suggest how I might test this? I'm not using nUnit or a similar framework; I am using the testing tools built into Visual Studio.
I'm fairly new to writing unit tests, but I imagine I should at least test the following:
Valid response passed into SaveFormBrokerResponse() method
Test for valid exceptions thrown by the catch()
Testing the started Task, but not sure how to do this
I've stripped just a bit out of this function, mostly to do with instantiation and population of some objects:
public void SaveResponse(IForm form, bool isLive, HttpRequestBase request)
{
try
{
var response = new FormBrokerResponses();
// Initialize some vars on response
using (var memory = new MemoryStream())
{
var serializer = new DataContractSerializer(typeof(FormKeyValue[]));
serializer.WriteObject(memory, request.Form.AllKeys.Select(r => new FormKeyValue(r, request.Form[r])).ToArray());
memory.Flush();
memory.Seek(0, SeekOrigin.Begin);
response.Values = Encoding.UTF8.GetString(memory.ToArray());
}
_dataHandler.SaveFormBrokerResponses(response);
}
catch (Exception ex)
{
throw new Exception("boom explosions");
}
Task.Factory.StartNew(() => DispatchFormResponseViaEmail(form, isLive, request.Form.AllKeys.ToDictionary(r => r, r => (object)request.Form[r])));
}
I realize that testing void implementations is tricky and questionable and that there are some integration test concerns here, but that said I can't (currently) change the implementation and need to write tests for what I have.
You can't. You've created a method that fires off an asynchronous operation and then doesn't expose any means of observing the completion/results of that operation to the caller. There are lots of ways of doing this (returning a task, accepting a callback, an event, etc.) but you need to do something for the caller to be able to observe the results of the asynchronous operation. If the method doesn't expose anything, then there is nothing that the caller can reliably do.
If you are allowed to make slight modifications to the code I would do the following which is just a small change anyway :
public void SaveResponse(IForm form, bool isLive, HttpRequestBase request)
{
try
{
var response = new FormBrokerResponses();
// Initialize some vars on response
using (var memory = new MemoryStream())
{
var serializer = new DataContractSerializer(typeof(FormKeyValue[]));
serializer.WriteObject(memory, request.Form.AllKeys.Select(r => new FormKeyValue(r, request.Form[r])).ToArray());
memory.Flush();
memory.Seek(0, SeekOrigin.Begin);
response.Values = Encoding.UTF8.GetString(memory.ToArray());
}
_dataHandler.SaveFormBrokerResponses(response);
}
catch (Exception ex)
{
throw new Exception("boom explosions");
}
Dispatch(form,isLive,request);
}
virtual void Dispatch(IForm form, bool isLive, HttpRequestBase request){
Task.Factory.StartNew(() => DispatchFormResponseViaEmail(form, isLive, request.Form.AllKeys.ToDictionary(r => r, r => (object)request.Form[r])));
}
I don't know what this class is named so suppose the class is named DutClass, you can now derive a different implementation of that class as following:
public class UnitTestClass : DutClass{
override Dispatch(){
//don't do anything or set a state variable that this method was called
}
}
Then instead of testing the DutClass you test the UnitTextClass which has a different implementation of the Dispatch method and does not start a Task at all. You can then test that in fact this method was called, test for the exceptions and so on.
I'm building a T4 template that will help people construct Azure queues in a consistent and simple manner. I'd like to make this self-documenting, and somewhat consistent.
First I made the queue name at the top of the file, the queue names have to be in lowercase so I added ToLower()
The public constructor uses the built-in StorageClient API's to access the connection strings. I've seen many different approaches to this, and would like to get something that works in almost all situations. (ideas? do share)
I dislike the unneeded HTTP requests to check if the queues have been created so I made is a static bool . I didn't implement a Lock(monitorObject) since I don't think one is needed.
Instead of using a string and parsing it with commas (like most MSDN documentation) I'm serializing the object when passing it into the queue.
For further optimization I'm using a JSON serializer extension method to get the most out of the 8k limit. Not sure if an encoding will help optimize this any more
Added retry logic to handle certain scenarios that occur with the queue (see html link)
Q: Is "DataContext" appropriate name for this class?
Q: Is it a poor practice to name the Queue Action Name in the manner I have done?
What additional changes do you think I should make?
public class AgentQueueDataContext
{
// Queue names must always be in lowercase
// Is named like a const, but isn't one because .ToLower won't compile...
static string AGENT_QUEUE_ACTION_NAME = "AgentQueueActions".ToLower();
static bool QueuesWereCreated { get; set; }
DataModel.SecretDataSource secDataSource = null;
CloudStorageAccount cloudStorageAccount = null;
CloudQueueClient cloudQueueClient = null;
CloudQueue queueAgentQueueActions = null;
static AgentQueueDataContext()
{
QueuesWereCreated = false;
}
public AgentQueueDataContext() : this(false)
{
}
public AgentQueueDataContext(bool CreateQueues)
{
// This pattern of setting up queues is from:
// ttp://convective.wordpress.com/2009/11/15/queues-azure-storage-client-v1-0/
//
this.cloudStorageAccount = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
this.cloudQueueClient = cloudStorageAccount.CreateCloudQueueClient();
this.secDataSource = new DataModel.SecretDataSource();
queueAgentQueueActions = cloudQueueClient.GetQueueReference(AGENT_QUEUE_ACTION_NAME);
if (QueuesWereCreated == false || CreateQueues)
{
queueAgentQueueActions.CreateIfNotExist();
QueuesWereCreated = true;
}
}
// This is the method that will be spawned using ThreadStart
public void CheckQueue()
{
while (true)
{
try
{
CloudQueueMessage msg = queueAgentQueueActions.GetMessage();
bool DoRetryDelayLogic = false;
if (msg != null)
{
// Deserialize using JSON (allows more data to be stored)
AgentQueueEntry actionableMessage = msg.AsString.FromJSONString<AgentQueueEntry>();
switch (actionableMessage.ActionType)
{
case AgentQueueActionEnum.EnrollNew:
{
// Add to
break;
}
case AgentQueueActionEnum.LinkToSite:
{
// Link within Agent itself
// Link within Site
break;
}
case AgentQueueActionEnum.DisableKey:
{
// Disable key in site
// Disable key in AgentTable (update modification time)
break;
}
default:
{
break;
}
}
//
// Only delete the message if the requested agent has been missing for
// at least 10 minutes
//
if (DoRetryDelayLogic)
{
if (msg.InsertionTime != null)
if (msg.InsertionTime < DateTime.UtcNow + new TimeSpan(0, 10, 10))
continue;
// ToDo: Log error: AgentID xxx has not been found in table for xxx minutes.
// It is likely the result of a the registratoin host crashing.
// Data is still consistent. Deleting queued message.
}
//
// If execution made it to this point, then we are either fully processed, or
// there is sufficent reason to discard the message.
//
try
{
queueAgentQueueActions.DeleteMessage(msg);
}
catch (StorageClientException ex)
{
// As of July 2010, this is the best way to detect this class of exception
// Description: ttp://blog.smarx.com/posts/deleting-windows-azure-queue-messages-handling-exceptions
if (ex.ExtendedErrorInformation.ErrorCode == "MessageNotFound")
{
// pop receipt must be invalid
// ignore or log (so we can tune the visibility timeout)
}
else
{
// not the error we were expecting
throw;
}
}
}
else
{
// allow control to fall to the bottom, where the sleep timer is...
}
}
catch (Exception e)
{
// Justification: Thread must not fail.
//Todo: Log this exception
// allow control to fall to the bottom, where the sleep timer is...
// Rationale: not doing so may cause queue thrashing on a specific corrupt entry
}
// todo: Thread.Sleep() is bad
// Replace with something better...
Thread.Sleep(9000);
}
Q: Is "DataContext" appropriate name for this class?
In .NET we have a lot of DataContext classes, so in the sense that you want names to appropriately communicate what the class does, I think XyzQueueDataContext properly communicates what the class does - although you can't query from it.
If you want to stay more aligned to accepted pattern languages, Patterns of Enterprise Application Architecture calls any class that encapsulates access to an external system for a Gateway, while more specifically you may want to use the term Channel in the language of Enterprise Integration Patterns - that's what I would do.
Q: Is it a poor practice to name the Queue Action Name in the manner I have done?
Well, it certainly tightly couples the queue name to the class. This means that if you later decide that you want to decouple those, you can't.
As a general comment I think this class might benefit from trying to do less. Using the queue is not the same thing as managing it, so instead of having all of that queue management code there, I'd suggest injecting a CloudQueue into the instance. Here's how I implement my AzureChannel constructor:
private readonly CloudQueue queue;
public AzureChannel(CloudQueue queue)
{
if (queue == null)
{
throw new ArgumentNullException("queue");
}
this.queue = queue;
}
This better fits the Single Responsibility Principle and you can now implement queue management in its own (reusable) class.