In my program I have a set of Links, let's say a million. The requirements are:
1) Process the links in parallel
2) Limit how many of these links will run in parallel.
In the example, we don't care about the execution order of the links but to ensure the process is finished with errors or without.
How can I properly ensure the above conditions?
class Program
{
private static void Main()
{
start_url_list();
}
static void start_url_list()
{
HashSet<string> urls = get_url_list();
Parallel.ForEach(urls, new ParallelOptions { MaxDegreeOfParallelism = 10 },
url =>
{
Console.WriteLine("Trying..:{0}", url);
var my_url = new Url_Class(url);
my_url.Start();
Console.WriteLine("Processing url is over");
//Do stuff with my_url properties
if (my_url.isError)
{
Console.WriteLine("ERROR:{0} msg:{1}",url,my_url.errorMsg);
}
else
{
Console.WriteLine("OK:{0} Size{1}", url, my_url.size);
}
});
}
}
public class Url_Class
{
public string url { get; private set; }
public string errorMsg { get; internal set; }
public bool isError { get; private set; }
public int size { get; private set; }
public Host(string url_str)
{
url = url_str;
isError = false;
}
public void Start()
{
//Long Process
}
}
Related
I am implementing a scheduler. As part of its core logic it processes a custom object Schedule. Basically it iterates over an array of schedules and try to process it. The problem is who ever creates a Schedule needs to register it with the container using an ISchedule interface. My Scheduler then pulls all the ISchedule references from the container. SO far this is working but it does not have the flexibility of loading the schedules runtime. What design and implementation I can adapt to implement a Scheduler that can load those Schedules run time. I am giving some sample code.
Something that is coming to my mind is having the developers writing a json representation of the Schedules and put that inside a config or implementing an endpoint that returns that config to the Scheduler. But can I avoid this? I want the Scheduler to be completely agonistic of developer code.
You can use the factory to register information about schedule classes. And dynamically change the call interval by finding the schedule by Id.
You register in the container: <IScheduleFactory,ScheduleFactory> and <IScheduleManager,ScheduleManager>
public interface ISchedule
{
public string Id { get; set; }
public TimeSpan Interval { get; set; }
public DateTime? LastExecution { get; set; }
public bool CanStart { get; }
void Start();
void Stop();
}
public sealed class Schedule : ISchedule
{
public string Id { get; set; }
public TimeSpan Interval { get; set; }
public DateTime? LastExecution { get; set; }
public bool CanStart {
get
{
lock (_sync)
{
return !LastExecution.HasValue || LastExecution.Value.Add(Interval) >= DateTime.UtcNow;
}
}
}
private readonly object _sync = new object();
public void Start()
{
lock (_sync)
{
if (!LastExecution.HasValue || LastExecution.Value.Add(Interval) >= DateTime.UtcNow)
{
// DO WORK
LastExecution = DateTime.UtcNow;
}
}
}
public void Stop()
{
throw new NotImplementedException();
}
}
public interface IScheduleFactory
{
ISchedule Create();
}
public sealed class ScheduleFactory: IScheduleFactory
{
private readonly IScheduleManager _manager;
public ScheduleFactory(IScheduleManager manager)
{
_manager = manager;
}
public ISchedule Create()
{
var schedule = new Schedule();
_manager.Register(schedule);
return schedule;
}
}
public interface IScheduleManager
{
void Register(ISchedule schedule);
ISchedule Get(string id);
void Start();
void Stop();
}
public sealed class ScheduleManager : IScheduleManager
{
private readonly Dictionary<string,ISchedule> _items = new Dictionary<string, ISchedule>();
private readonly object _sync = new object();
public void Register(ISchedule schedule)
{
lock (_sync)
{
if (_items.ContainsKey(schedule.Id))
_items.Add(schedule.Id, schedule);
}
}
public ISchedule Get(string id)
{
lock (_sync)
{
if (_items.ContainsKey(id))
return _items[id];
}
return null;
}
private bool _isStart;
public void Start()
{
_isStart = true;
while (_isStart)
{
ISchedule[] array = null;
lock (_sync)
{
array = _items.Values.ToArray();
}
foreach (var schedule in array)
{
if (schedule.CanStart)
Task.Factory.StartNew(()=>schedule.Start());
}
}
}
public void Stop()
{
_isStart = false;
}
}
I have 50 IMountCmd objects from one or more threads and drop them in a blocking collection. Each is executed and some get results or errors. They are put into a ConcurrentDictionary where I loop for ContainsKey and return the objects. Does this seems thread safe and correct way to process a blocking queue?
public class CmdGetAxisDegrees : IMountCommand
{
public long Id { get; }
public DateTime CreatedUtc { get; private set; }
public bool Successful { get; private set; }
public Exception Exception { get; private set; }
public dynamic Result { get; private set; }
private readonly AxisId _axis;
public CmdGetAxisDegrees(long id, AxisId axis)
{
Id = id;
_axis = axis;
CreatedUtc = HiResDateTime.UtcNow;
Successful = false;
Queues.AddCommand(this);
}
public void Execute(Actions actions)
{
try
{
Result = actions.GetAxisDegrees(_axis);
Successful = true;
}
catch (Exception e)
{
Successful = false;
Exception = e;
}
}
}
private static void ProcessCommandQueue(IMountCommand command)
{
command.Execute(_actions);
if (command.Id > 0)
{
_resultsDictionary.TryAdd(command.Id, command);
}
}
public static IMountCommand GetCommandResult(long id)
{
IMountCommand result;
while (true)
{
if (!_resultsDictionary.ContainsKey(id)) continue;
var success = _resultsDictionary.TryRemove(id, out result);
if (!success)
{
//log
}
break;
}
return result;
}
static Queues()
{
_actions = new Actions();
_resultsDictionary = new ConcurrentDictionary<long, IMountCommand>();
_commandBlockingCollection = new BlockingCollection<IMountCommand>();
Task.Factory.StartNew(() =>
{
foreach (var command in _commandBlockingCollection.GetConsumingEnumerable())
{
ProcessCommandQueue(command);
}
});
}
public interface IMountCommand
{
long Id { get; }
DateTime CreatedUtc { get; }
bool Successful { get; }
Exception Exception { get; }
dynamic Result { get; }
void Execute(Actions actions);
}
I have a large test set (5k+) using xUnit.net, and I'm having concurrency problems among tests running in parallel.
xUnit randomizes the execution order of tests, which makes it harder for me to detect the issue.
I'd like to know whether is there a way to log, during test execution, the moment a test starts and the moment it ends.
Note: Using constructor and disposer methods does not cut it, because you cannot know which test is being run on the constructor/disposer.
Note 2: In case it is not obvious, I'm looking for a solution that doesn't involve writing log call in each test.
Thanks,
Well, I managed to do it using the BeforeAfterTestAttribute from xUnit. Then I wrote the utility logger below to output the results to a .csv file.
public class LogTestExecutionAttribute: BeforeAfterTestAttribute
{
public override void Before(MethodInfo methodUnderTest)
{
TestExecutionDataLogger.LogBegin(methodUnderTest);
}
public override void After(MethodInfo methodUnderTest)
{
TestExecutionDataLogger.LogEnd(methodUnderTest);
}
}
public static class TestExecutionDataLogger
{
private static readonly string LogFileName = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData), "DbCoud", $"UnitTests_{DateTime.UtcNow:yyyy_MM_dd_HH_mm}_D_{AppDomain.CurrentDomain.Id}.csv");
private static int _startedOrder = 0;
private static int _endedOrder = 0;
private static readonly ConcurrentDictionary<string, testExecutionData> testDataDict = new ConcurrentDictionary<string, testExecutionData>();
private static readonly ConcurrentQueue<string> logQueue = new ConcurrentQueue<string>();
public static void LogBegin(MethodInfo testInfo)
{
var name = $"{testInfo.DeclaringType.FullName}.{testInfo.Name}";
var order = Interlocked.Add(ref _startedOrder, 1);
var startedUtc = DateTime.UtcNow;
var data = testDataDict.GetOrAdd(name, new testExecutionData());
data.StartedUtc = startedUtc;
data.StartedOrder = order;
data.TestName = name;
data.Status = "Started";
data.StartThreadId = Thread.CurrentThread.ManagedThreadId;
writeLog(data);
}
public static void LogEnd(MethodInfo testInfo)
{
var name = $"{testInfo.DeclaringType.FullName}.{testInfo.Name}";
var dataEndedUtc = DateTime.UtcNow;
var order = Interlocked.Add(ref _endedOrder, 1);
var data = testDataDict[name];
data.EndedUtc = dataEndedUtc;
data.EndedOrder = order;
data.Status = "Ended";
data.EndThreadId = Thread.CurrentThread.ManagedThreadId;
writeLog(data);
}
private static void writeLog(testExecutionData data)
{
logQueue.Enqueue(data.ToCsvLine());
if (data.EndedOrder == 1)
{
Directory.CreateDirectory(Path.GetDirectoryName(LogFileName));
Task.Run(logWriter);
}
}
private static Task logWriter()
{
while (true)
{
var logs = new List<string>();
string result;
while (logQueue.TryDequeue(out result))
{
logs.Add(result);
}
if (logs.Any())
{
File.AppendAllLines(LogFileName, logs);
}
}
}
private class testExecutionData
{
public int StartedOrder { get; set; }
public int EndedOrder { get; set; }
public DateTime StartedUtc { get; set; }
public DateTime EndedUtc { get; set; }
public string TestName { get; set; }
public string Status { get; set; }
public int StartThreadId { get; set; }
public int EndThreadId { get; set; }
public string ToCsvLine() { return $"{TestName};{Status};{StartedOrder};{EndedOrder};{StartedUtc:o};{EndedUtc:o};{Math.Max(0, ( EndedUtc - StartedUtc ).TotalMilliseconds)};{StartThreadId};{EndThreadId}"; }
}
}
To use this code, add the LogTestExecutionAttribute to the test classes you want to log (or to the base classes ;p).
I have a saga which checks on the status of an API calls every 30 seconds if the status returned back from the call is successful the saga is ended, if not
the saga waits 30 seconds and attempts again. If the API call has not returned a successful response within 60 minutes, the saga is timed out and ended.
I am having problems getting my 60 minute timeout to fire. The code I have is
public class MonitorSubmissionFeedSagaData: IContainSagaData
{
public Guid Id { get; set; }
public string Originator { get; set; }
public string OriginalMessageId { get; set; }
public bool TimeoutSet { get; set; }
[Unique]
public string JobId { get; set; }
}
public class MonitorSubmissionFeedSaga : Saga<MonitorSubmissionFeedSagaData>,
IAmStartedByMessages<MonitorFeedSubmissonCommand>,
IHandleMessages<StartCheckSubmissionCommand>,
IHandleTimeouts<MonitorSubmissionFeedSagaTimeout>
{
public const int SagaTimeoutInMinutes = 60;
public IEmpathyBrokerClientApi PostFileService { get; set; }
protected override void ConfigureHowToFindSaga(SagaPropertyMapper<MonitorSubmissionFeedSagaData> mapper)
{
mapper.ConfigureMapping<MonitorFeedSubmissonCommand>(x => x.JobId).ToSaga(saga => saga.JobId);
}
public void Handle(MonitorFeedSubmissonCommand message)
{
Data.JobId = message.JobId;
CheckTimeout();
Bus.Send(new StartCheckSubmissionCommand
{
JobId = Data.JobId
});
}
public void Handle(StartCheckSubmissionCommand message)
{
Log.Info("Saga with JobId {0} received", Data.JobId);
bool isCompleted = GetJobStatus(message.JobId);
while (isCompleted)
{
Thread.Sleep(30000);
isCompleted = GetJobStatus(message.JobId);
}
MarkAsComplete();
}
public void CheckTimeout()
{
RequestTimeout<MonitorSubmissionFeedSagaTimeout>(TimeSpan.FromMinutes(SagaTimeoutInMinutes));
}
public void Timeout(MonitorSubmissionFeedSagaTimeout state)
{
MarkAsComplete();
}
bool GetJobStatus(string jobId)
{
return false;
var status = PostFileService.GetJobIdStatus(jobId);
if (status.state == "FAILURE" || status.state == "DISCARDED")
{
return false;
}
return true;
}
}
Can anyone see where I am going wrong?
thanks
Your Saga should go idle. You're keeping it alive with a while loop. The timeout message arrives at some point and then you should check what should happen. Either another checkout or MarkAsComplete.
I wrote this in Notepad, so it might not compile. But it's to get an idea.
public class MonitorSubmissionFeedSagaData: IContainSagaData
{
public Guid Id { get; set; }
public string Originator { get; set; }
public string OriginalMessageId { get; set; }
[Unique]
public string JobId { get; set; }
public DateTime SagaStartTimeUtc { get; set; }
}
public class MonitorSubmissionFeedSaga : Saga<MonitorSubmissionFeedSagaData>,
IAmStartedByMessages<MonitorFeedSubmissonCommand>,
IHandleTimeouts<VerifyApiTimeOut>
{
public IEmpathyBrokerClientApi PostFileService { get; set; }
public void Handle(MonitorFeedSubmissonCommand message)
{
Data.JobId = message.JobId;
Data.SagaStartTimeUtc = DateTime.NowUtc;
CreateTimeoutRequest();
}
public void CreateTimeoutRequest()
{
RequestTimeout<VerifyApiTimeOut>(TimeSpan.FromSeconds(30));
}
public void Timeout(VerifyApiTimeOut state)
{
if (!GetJobStatus(Data.JobId) && DateTime.NowUtc < Data.SagaStartTimeUtc.AddMinutes(60))
{
CreateTimeoutRequest();
}
MarkAsComplete();
}
bool GetJobStatus(string jobId)
{
return false;
var status = PostFileService.GetJobIdStatus(jobId);
if (status.state == "FAILURE" || status.state == "DISCARDED")
{
return false;
}
return true;
}
}
Another comment could be that the Saga itself should not call out to external services. Preferably not even to some database. Delegate this to another service. Every 30 seconds, send out a message to another handler. This handler should call the WebService/WebAPI. When it can confirm that everything is correct, reply to the original Saga. When it's not correct, just let it be. The Saga will send out messages every 30 seconds to retry.
After 60 minutes, the Saga should stop sending messages and MarkAsComplete.
I have the following task i would love to initialize once the app is running, and just populate a value in my App.xaml.cs
public partial class App : Application
{
public bool ProcessedTask = false;
public List<Item> Items = new List<Item>();
public App()
{
...
// Standard XAML initialization
InitializeComponent();
// NOW HERE I WOULD LIKE TO INVOKE MY TASK
}
public async Task<string> RequestDataTask()
{
HttpClient client = new HttpClient();
Task<string> getStringRequests = client.GetStringAsync("http://test.com/get/");
ItemsRootObject responseObject = Newtonsoft.Json.JsonConvert.DeserializeObject<ItemsRootObject>(getStringRequests);
foreach (Item item in responseObject.rows)
{
Items.Add(item);
}
ProcessedTask = true;
string Message = "Processed Task";
return Message;
}
}
public class ItemsRootObject
{
public List<Item> rows { get; set; }
public bool success { get; set; }
public int code { get; set; }
public string msg { get; set; }
}
public class Item
{
public string value { get; set; }
}
I already know the JSON return and objects are there correctly (works in a regular call), but i don't know (understand) how to invoke my task while keeping the phone app going (load MainPage.xaml).
Anyone can tell how i get my task started asynchronously?
You can use async and await where you need to call the task.
private async void yourFunction(object sender, RoutedEventArgs e)
{
string result = await RequestDataTask();
}
http://msdn.microsoft.com/it-it/library/hh191443.aspx
But if you want to call an async method inside a constructor use something like this:
public partial class App
{
private readonly Task _initializingTask;
public App()
{
_initializingTask = Init();
}
private async Task Init()
{
/*
Initialization that you need with await/async stuff allowed
*/
string result = await RequestDataTask();
}
}