We Keep Coding

Executing same function from several async methods simultenously causes errors

Here is a piece of code, where I try to execute different async methods, that need to be executed in specific order (the await, and Task.WhenAll() parts).
//Some other tasks before
Task<bool> taskIfcQuantityArea = Task.Run<bool>(() =>
{
return this.addGroupStringToDictionary("IfcQuantityArea");
});
Task<bool> taskIfcQuantityLength = Task.Run<bool>(() =>
{
return this.addGroupStringToDictionary("IfcQuantityLength");
});
Task<bool> taskIfcSiUnit = Task.Run<bool>(() =>
{
return addGroupStringToDictionary("IfcSiUnit");
});
Task<bool> taskIfcPropertySingleValue = Task.Run<bool>(() =>
{
return addGroupStringToDictionary("IfcPropertySingleValue");
});
//uses IfcPerson, IfcOrganization
Task<bool> taskIfcPersonAndOrganization = Task.Run<bool>(() =>
{
return addGroupStringToDictionary("IfcPersonAndOrganization");
});
//uses IfcOrganization
Task<bool> taskIfcApplication = Task.Run(async () =>
{
await taskIfcSiUnit;
return addGroupStringToDictionary("IfcApplication");
});
//uses IfcSiUnit
Task<bool> taskIfcMeasureWithUnit = Task.Run(async () =>
{
await taskIfcSiUnit;
return addGroupStringToDictionary("IfcMeasureWithUnit");
});
//some other tasks after.
When I do that job synchronously, all works fine, but when I do it in async, I have some random errors. At every test, the errors come randomly.
The only thing I see that could go wrong, is they all execute the same function addGroupStringToDictionary.
Here is the function :
private bool addGroupStringToDictionary(string typeName)
{
//int processCount = await Task.Run<int>(() =>
//{
GroupedListStrings groupElt = this.listGrouppedStrings.FirstOrDefault(x => x.Type == typeName.ToUpper());
if (groupElt != null)
{
List<string> listStringInGroup = groupElt.ListStrings;
foreach (string line in listStringInGroup)
{
try
{
if(typeName== "IfcLocalPlacement($")
{
typeName = "IfcLocalPlacement";
}
var type = Type.GetType("Ifc."+typeName);
if (typeName == "IfcPropertySingleValue" || typeName == "IfcDirection" || typeName == "IfcSiUnit" || typeName == "IfcQuantityLength" || typeName == "IfcQuantityArea" || typeName == "IfcQuantityVolume" || typeName == "IfcQuantityWeight")
{
try
{
object instance = Activator.CreateInstance(type, line);
this.addToListDictionary((IfcElement)instance);
}
catch
{
}
}
else if (typeName == "IfcOpeningElement")
{
try
{
object instance = Activator.CreateInstance(type, line, this.listDictionaries, this.DictionaryBolts);
this.addToListDictionary((IfcElement)instance);
}
catch
{
}
}
else
{
try
{
object instance = Activator.CreateInstance(type, line, this.listDictionaries);
this.addToListDictionary((IfcElement)instance);
}
catch
{
}
}
}
catch
{
this.addError(line);
}
}
this.listGrouppedStrings.Remove(groupElt);
this.reportProgressImport();
}
//return 100;
//});
this.reportProgressImport();
return true;
}
The catch got 1-2 times over a bit more than 1 million lines.
At each test the errors come randomly.
Is it possible that running the function simultaneously from several async methods, this is what causes the problem?
Here is the addToListDictionary function :
private void addToListDictionary(IfcElement elt)
{
if(elt.ErrorFound)
{
this.listReadButError.Add(elt);
return;
}
string type = elt.GetType().ToString();
if (elt is IfcRepere)
{
type = "Ifc.IfcRepere";
}
else if (elt is IfcRepereType)
{
type = "Ifc.IfcRepereType";
}
else if (elt is IfcPhysicalSimpleQuantity)
{
type = "Ifc.IfcPhysicalSimpleQuantity";
}
else if (elt is IfcProfileDef)
{
type = "Ifc.IfcProfileDef";
}
else if (elt is IfcGeometricRepresentationContext)
{
type = "Ifc.IfcGeometricRepresentationContext";
}
GroupDictionary group = this.ListDictionaries.FirstOrDefault(x => x.Name == type);
if(group==null)
{
group = new GroupDictionary { Name = type };
this.ListDictionaries.Add(group);
}
group.ListElements[elt.ID] = elt;
if (elt is IfcMechanicalFastener)
{
IfcMechanicalFastener bolt = (IfcMechanicalFastener)elt;
this.DictionaryBolts[bolt.Tag] = bolt;
}
else if(elt is IfcProject)
{
this.listProjects.Add((IfcProject)elt);
}
else if(elt is IfcElementAssembly ifcAss)
{
this.DictionaryIfcElementAssemblies[ifcAss.Key] = ifcAss;
}
}
Also some additive information about my ListDictionaries :
private List<GroupDictionary> listDictionaries = new List<GroupDictionary>();
public List<GroupDictionary> ListDictionaries { get { return this.listDictionaries; } set { this.listDictionaries = value; } }
And the class GroupDictionary
public class GroupDictionary
{
string name { get; set; }
public string Name { get { return this.name; } set { this.name = value; } }
public ConcurrentDictionary<int, IfcElement> ListElements = new ConcurrentDictionary<int, IfcElement>();
public GroupDictionary()
{
}
}
I made different GroupDictionary because as soon as I don't need one of them, I delete it to free space.
I have one dictionary with IfcPoint, I need it to gt IfcPolyLine (lines), but when I finish to treat all objects using IfcPoint, I clear remove the corresponding GroupDictionary in order to free some memory.

You have a obvious thread-safety issues here (i.e. you are trying to perform some operation which is not thread safe from multiple threads at a time). There are multiple ways you can try tackling it - using locks, or some synchronization primitives.
But in this case it seems that major source of issues is working with standard collections from multiple threads, which is not thread-safe (because thread-safety usually comes with performance price and is not always needed). You can start from switching to appropriate collections from System.Collections.Concurrent namespace.
To go down deeper I recommend free e-book by Joseph Albahari Threading in C#.

Multiple tasks returns incorrect result

What I need to do
I need to start different instances of a class in a synchronous context using an async method.
Application structure
In my console application I've declared a List<Bot> class:
private List<Bot> _bots = new List<Bot>(new Bot[10]);
the class Bot contains some methods that takes data from internet, so these methods need to be waited. The method structure looks like this:
public class Bot
{
Competition Comp { get; set; }
public async Task StartAsync(int instance)
{
string url = "";
//based on the instance I take the data from different source.
switch(instance)
{
case 0:
url = "www.google.com";
break;
case 1:
url = "www.bing.com";
break;
}
//Comp property contains different groups.
Comp.Groups = await GetCompetitionAsync(Comp, url);
if(Comp.Groups.Count > 0)
{
foreach(var gp in group)
{
//add data inside database.
}
}
}
}
the Competition class have the following design:
public class Competition
{
public string Name { get; set; }
public List<string> Groups { get; set; }
}
I start all the instances of Bot class using the following code:
for(int i = 0; i < _bots.Count - 1; i++)
{
_bots[i].StartAsync(i);
}
this code will call different times StartAsync of Bot class, in this way, I can manage each instance of the bot, and I can eventually stop or start a specific instance in a separate method.
The problem
The method GetCompetitionAsync create a List<string>:
public async Task<List<string>> GetCompetitionAsync(Competition comp, string url)
{
if(comp == null)
comp = new Competition();
List<string> groups = new List<string();
using (var httpResonse = await httpClient.GetAsync(url))
{
string content = await httpResponse.Content.ReadAsStringAsync();
//fill list groups
}
return groups;
}
essentially this method will fill the List<string> available in Comp. Now, if I execute a single instance of StartAsync all works well, but when I run multiple instances (as the for above), the Comp object (which contains the Competition) have all the properties NULL.
So seems that when I have multiple Task running the synchronous context doesn't wait the async context, which in this case fill the List<string>.
When the code reach this line: if(Competition.Groups.Count > 0) I get a NULL exception, because Groups is null, and other Comp properties are NULL.
How can I manage this situation?
UPDATE
After other attempts, I though to create a List<Task> instead of a List<Bot>:
List<Task> tasks = new List<Task>(new Task[10]);
then instead of:
for(int i = 0; i < _bots.Count - 1; i++)
{
_bots[i].StartAsync(i);
}
I did:
for (int i = 0; i < tasks.Count - 1; i++)
{
Console.WriteLine("Starting " + i);
if (tasks[i] == null)
tasks[i] = new Task(async () => await new Bot().StartAsync(i));
apparently all is working well, I got no errors. The problem is: why? I though to something like a deadlock, that I can't even solve using ConfigureAwait(false);.
The last solution also doesn't allow me to access to Bot method because is now a Task.
UPDATE 2
Okay maybe I gotcha the issue. Essentially the await inside the asynchronous method StartAsync is trying to comeback on the main thread, meanwhile the main thread is busy waiting the task to complete, and this will create a deadlock.
That's why moving the StartAsync() inside a List<Task> has worked, 'cause now the async call is now running on a thread pool thread, it doesn't try to comeback to the main thread, and everything seems to works. But I can't use this solution for the reasons explained above.

I'm prefer use Threads instead of Tasks. IMHO, Threads more simple for understanding.
Note: seems that property Bot.Comp in your code is NOT initialized! I fix this issue.
My version of your code:
public class Bot
{
Competition Comp { get; set; }
System.Thread _thread;
private int _instance;
public Bot()
{
Comp = new Competition ();
}
public void Start(int instance)
{
_instance = instance;
_thread = new Thread(StartAsync);
_thread.Start();
}
private void StartAsync()
{
string url = "";
//based on the instance I take the data from different source.
switch(_instance)
{
case 0:
url = "www.google.com";
break;
case 1:
url = "www.bing.com";
break;
}
//Comp property contains different groups.
GetCompetitionAsync(Comp, url);
if(Comp.Groups.Count > 0)
{
foreach(var gp in group)
{
//add data inside database.
}
}
}
public List<string> GetCompetitionAsync(Competition comp, string url)
{
if(comp.groups == null) comp.groups = new List<string>();
using (var httpResonse = httpClient.GetAsync(url))
{
string content = await httpResponse.Content.ReadAsStringAsync();
//fill list groups
}
return groups;
}
}
Then we run threads:
for(int i = 0; i < _bots.Count - 1; i++)
{
_bots[i].Start(i);
}
Each instance of Bot starts method private void StartAsync() in it's own thread.
Note a implementation of method Bot.Start():
public void Start(int instance)
{
_instance = instance;
_thread = new Thread(StartAsync); //At this line: set method Bot.StartAsync as entry point for new thread.
_thread.Start();//At this line: call of _thread.Start() starts new thread and returns **immediately**.
}

This sort of thing is far simpler if you think in terms of lists and "pure" functions-- functions that accept input and return output. Don't pass in something for them to fill or mutate.
For example, this function accepts a string and returns the groups:
List<string> ExtractGroups(string content)
{
var list = new List<string>();
//Populate list
return list;
}
This function accepts a URL and returns its groups.
async Task<List<string>> GetCompetitionAsync(string url)
{
using (var httpResponse = await httpClient.GetAsync(url))
{
string content = await httpResponse.Content.ReadAsStringAsync();
return ExtractGroups(content);
}
}
And this function accepts a list of URLs and returns all of the groups as one list.
async Task<List<string>> GetAllGroups(string[] urls)
{
var tasks = urls.Select( u => GetCompetitionAsync(u) );
await Task.WhenAll(tasks);
return tasks.SelectMany( t => t.Result );
}
You can then stuff the data into the database as you had planned.
var groups = GetAllGroups( new string[] { "www.google.com", "www.bing.com" } );
foreach(var gp in groups)
{
//add data inside database.
}
See how much simpler it is when you break it down this way?

How to set a Dynamic Parallel Task Execution instead of manual call dispatch?

What is the best approach in this scenario, where in there are multiple tasks that need to be run, depending on a given parameter. See the code below:
void Mapping()
{
if(param.IsProgram1) {
// spGetProgram1() is a stored procedure call
MapProgram1(context, spGetProgram1().GetIterator());
if(param.IsProgram2) {
MapProgram2(context, spGetProgram2().GetIterator());
}
}
if(param.IsProgram3) {
MapProgram3(context, spGetProgram23().GetIteratior());
}
}
static void MapProgram1(context, IEnumerable<IDataRecord> records) {
// map records to context
}
static void MapProgram2(context, IEnumerable<IDataRecord> records) {
// map records to context
}
static void MapProgram3(context, IEnumerable<IDataRecord> records) {
// map records to context
}
I want to refactor this and run the tasks in parallel. My current approach is that I store the task in a dictionary and invoke it depending on the given parameter
var tasks = new Dictionary<string, Task<IEnumerable<IDataRecord>>>()
{
{ "MapProgram1", null },
{ "MapProgram2", null },
{ "MapProgram3", null },
}
if(param.IsProgram1) tasks["MapProgram1"].Value = Task.Run(() => spGetProgram1().GetIterator())
... ...
tasks.WaitAll(tasks.Select(t => t.Value).ToArray());
And then fetch the result and call the corresponding mapping method like this
foreach(var t in tasks.Where(t => t.Value != null))
{
if(t.Key == "MapProgram1")
{
MapProgram1(context, t.Value.Result);
}
if (t.Key == "MapProgram2")
{
MapProgram2(context, t.Value.Result);
}
.....
}
I'm sure there's a cleaner approach on this, that I don't need to manually call the methods.

I'm not sure I followed your example; however, in general if you want to execute tasks in parallel and conditionally choose which tasks to execute, you might do something like this...
async Task Run() {
List<Task<IResult>> tasks = new List<Task<IResult>>();
if (someCondition) {
tasks.Add(RunSome(someParams));
}
if (othercondition) {
tasks.Add(RunOther(otherParam));
}
IResult[] results = await Task.WhenAll(tasks);
foreach (var result in results) {
if (result is SomeResult someResult) {
// Handle some result
}
else if (result is OtherResult otherResult) {
// Handle other result
}
}
}
static async Task<SomeResult> RunSome(someParams) {
// Run something
}
static async Task<OtherResult> RunOther(otherParams) {
// Run other thing
}
interface IResult {
}
class SomeResult : IResult {
}
class OtherResult: IResult {
}
You add the tasks you want to execute in parallel to a list, have each task return an instance of some result type, and then check the results after all tasks are completed.

You can do a lot with Parallel.ForEach i.e adjust the degree of parallelism, ect
However it does depend on your work load and what you are trying to achieve, however this may give you food for thought
var list = new List<Stuff>();
Parallel.ForEach(list, (item) =>
{
switch(item,ConditionType)
{
case ConditionType.First : DoSomethingWithItem(item); break;
case ConditionType.Second : DoSomethingElseWithItem(item); break;
}
});
More resources
Parallel.ForEach Method
Msdn : How to: Write a Simple Parallel.ForEach Loop

.ToListAsync() vs .ToList() + Task.Run

I have a WPF application.
The data is brought from the repository to the ViewModel. What would be the better way to retrieve the data:
Method 1:
In Repository:
public List<LogDetail> GetLogsOfTypeForCase(int caseId, LoggType logType)
{
using (var ctx = new SidraEntitiesNoChangesDetection())
{
var logs = (from l in ctx.Loggs
where l.log_fk_caseid == caseId && l.log_operation == logType.ToString()
select new LogDetail()
{
ColumnName = l.log_columnname,
DateAndTime = l.log_dateandtime,
IdentificationDetail = l.log_identificationDetail,
NewValue = l.log_new_value,
OldValue = l.log_old_value,
TableName = l.log_tablename,
UserCode = l.User.usr_code
}).ToList();
return logs;
}
}
In ViewModel:
await Task.Run(
() =>
{
if (false == this.CancellationTokenSource.IsCancellationRequested)
{
this.CaseLogs = this.dataAdapter.GetLogsOfTypeForCase(this.CaseId, LoggType.S);
}
},
this.CancellationTokenSource.Token
);
or
Method 2
In Repository:
public async Task<List<LogDetail>> GetLogsOfTypeForCase(int caseId, LoggType logType)
{
using (var ctx = new SidraEntitiesNoChangesDetection())
{
var logs = await (from l in ctx.Loggs
where l.log_fk_caseid == caseId && l.log_operation == logType.ToString()
select new LogDetail()
{
ColumnName = l.log_columnname,
DateAndTime = l.log_dateandtime,
IdentificationDetail = l.log_identificationDetail,
NewValue = l.log_new_value,
OldValue = l.log_old_value,
TableName = l.log_tablename,
UserCode = l.User.usr_code
}).ToListAsync();
return logs;
}
}
and in ViewModel
protected override async void Load()
{
if (false == this.CancellationTokenSource.IsCancellationRequested)
{
this.CaseLogs = await this.dataAdapter.GetLogsOfTypeForCase(this.CaseId, LoggType.S);
}
}
From what I have read, Method 1 would be preferred, but what be the advantages?

Method 2 is preferable, because it uses one less thread.
It also can (with some modifications) properly support cancellation:
public async Task<List<LogDetail>> GetLogsOfTypeForCase(int caseId, LoggType logType, CancellationToken token)
{
...
}).ToListAsync(token);
...
}
protected override async void Load()
{
this.CaseLogs = await this.dataAdapter.GetLogsOfTypeForCase(this.CaseId, LoggType.S, this.CancellationTokenSource.Token);
}

Method 2. The Async versions of these methods do all the "heavy lifting" for you, that was what they were designed for. Task.Run() is a heavy process call, you are required to handle all the potential errors and failures yourself, you don't need a sledge hammer here, just a light weight finishing hammer.
By this I mean that you are trying to create what the framework has already done for you, this is where Async calls were meant to be used so why not just use them?

Saving subscription state for resuming it later

Update - solved
The final solution differs a bit from Brandon's suggestion but his answer brought me on the right track.
class State
{
public int Offset { get; set; }
public HashSet<string> UniqueImageUrls = new HashSet<string>();
}
public IObservable<TPicture> GetPictures(ref object _state)
{
var localState = (State) _state ?? new State();
_state = localState;
return Observable.Defer(()=>
{
return Observable.Defer(() => Observable.Return(GetPage(localState.Offset)))
.SubscribeOn(TaskPoolScheduler.Default)
.Do(x=> localState.Offset += 20)
.Repeat()
.TakeWhile(x=> x.Count > 0)
.SelectMany(x=> x)
.Where(x=> !localState.UniqueImageUrls.Contains(x.ImageUrl))
.Do(x=> localState.UniqueImageUrls.Add(x.ImageUrl));
});
}
IList<TPicture> GetPage(int offset)
{
...
return result;
}
Original Question
I'm currently struggling with the following problem. The PictureProvider implementation shown below is working with an offset variable used for paging results of a backend service providing the actual data. What I would like to implement is an elegant solution making the current offset available to the consumer of the observable to allow for resuming the observable sequence at a later time at the correct offset. Resuming is already accounted for by the intialState argument to GetPictures().
Recommendations for improving the code in a more RX like fashion would be welcome as well. I'm actually not so sure if the Task.Run() stuff is appropriate here.
public class PictureProvider :
IPictureProvider<Picture>
{
#region IPictureProvider implementation
public IObservable<Picture> GetPictures(object initialState)
{
return Observable.Create<Picture>((IObserver<Picture> observer) =>
{
var state = new ProducerState(initialState);
ProducePictures(observer, state);
return state;
});
}
#endregion
void ProducePictures(IObserver<Picture> observer, ProducerState state)
{
Task.Run(() =>
{
try
{
while(!state.Terminate.WaitOne(0))
{
var page = GetPage(state.Offset);
if(page.Count == 0)
{
observer.OnCompleted();
break;
}
else
{
foreach(var picture in page)
observer.OnNext(picture);
state.Offset += page.Count;
}
}
}
catch (Exception ex)
{
observer.OnError(ex);
}
state.TerminateAck.Set();
});
}
IList<Picture> GetPage(int offset)
{
var result = new List<Picture>();
... boring web service call here
return result;
}
public class ProducerState :
IDisposable
{
public ProducerState(object initialState)
{
Terminate = new ManualResetEvent(false);
TerminateAck = new ManualResetEvent(false);
if(initialState != null)
Offset = (int) initialState;
}
public ManualResetEvent Terminate { get; private set; }
public ManualResetEvent TerminateAck { get; private set; }
public int Offset { get; set; }
#region IDisposable implementation
public void Dispose()
{
Terminate.Set();
TerminateAck.WaitOne();
Terminate.Dispose();
TerminateAck.Dispose();
}
#endregion
}
}

I suggest refactoring your interface to yield the state as part of the data. Now the client has what they need to resubscribe where they left off.
Also, once you start using Rx, you should find that using synchronization primitives like ManualResetEvent are rarely necessary. If you refactor your code so that retrieving each page is its own Task, then you can eliminate all of that synchronization code.
Also, if you are calling a "boring web service" in GetPage, then just make it async. This gets rid of the need to call Task.Run among other benefits.
Here is a refactored version, using .NET 4.5 async/await syntax. It could also be done without async/await. I also added a GetPageAsync method that uses Observable.Run just in case you really cannot convert your webservice call to be asynchronous
/// <summary>A set of pictures</summary>
public struct PictureSet
{
public int Offset { get; private set; }
public IList<Picture> Pictures { get; private set; }
/// <summary>Clients will use this property if they want to pick up where they left off</summary>
public int NextOffset { get { return Offset + Pictures.Count; } }
public PictureSet(int offset, IList<Picture> pictures)
:this() { Offset = offset; Pictures = pictures; }
}
public class PictureProvider : IPictureProvider<PictureSet>
{
public IObservable<PictureSet> GetPictures(int offset = 0)
{
// use Defer() so we can capture a copy of offset
// for each observer that subscribes (so multiple
// observers do not update each other's offset
return Observable.Defer<PictureSet>(() =>
{
var localOffset = offset;
// Use Defer so we re-execute GetPageAsync()
// each time through the loop.
// Update localOffset after each GetPageAsync()
// completes so that the next call to GetPageAsync()
// uses the next offset
return Observable.Defer(() => GetPageAsync(localOffset))
.Select(pictures =>
{
var s = new PictureSet(localOffset, pictures);
localOffset += pictures.Count;
})
.Repeat()
.TakeWhile(pictureSet => pictureSet.Pictures.Count > 0);
});
}
private async Task<IList<Picture>> GetPageAsync(int offset)
{
var data = await BoringWebServiceCallAsync(offset);
result = data.Pictures.ToList();
}
// this version uses Observable.Run() (which just uses Task.Run under the hood)
// in case you cannot convert your
// web service call to be asynchronous
private IObservable<IList<Picture>> GetPageAsync(int offset)
{
return Observable.Run(() =>
{
var result = new List<Picture>();
... boring web service call here
return result;
});
}
}
Clients just need to add a SelectMany call to get their IObservable<Picture>. They can choose to store the pictureSet.NextOffset if they wish.
pictureProvider
.GetPictures()
.SelectMany(pictureSet => pictureSet.Pictures)
.Subscribe(picture => whatever);

Instead of thinking about how to save the subscription state, I would think about how to replay the state of the inputs (i.e. I'd try to create a serializable ReplaySubject that, on resume, would just resubscribe and catch back up to the current state).