We Keep Coding

Executing same function from several async methods simultenously causes errors

Here is a piece of code, where I try to execute different async methods, that need to be executed in specific order (the await, and Task.WhenAll() parts).
//Some other tasks before
Task<bool> taskIfcQuantityArea = Task.Run<bool>(() =>
{
return this.addGroupStringToDictionary("IfcQuantityArea");
});
Task<bool> taskIfcQuantityLength = Task.Run<bool>(() =>
{
return this.addGroupStringToDictionary("IfcQuantityLength");
});
Task<bool> taskIfcSiUnit = Task.Run<bool>(() =>
{
return addGroupStringToDictionary("IfcSiUnit");
});
Task<bool> taskIfcPropertySingleValue = Task.Run<bool>(() =>
{
return addGroupStringToDictionary("IfcPropertySingleValue");
});
//uses IfcPerson, IfcOrganization
Task<bool> taskIfcPersonAndOrganization = Task.Run<bool>(() =>
{
return addGroupStringToDictionary("IfcPersonAndOrganization");
});
//uses IfcOrganization
Task<bool> taskIfcApplication = Task.Run(async () =>
{
await taskIfcSiUnit;
return addGroupStringToDictionary("IfcApplication");
});
//uses IfcSiUnit
Task<bool> taskIfcMeasureWithUnit = Task.Run(async () =>
{
await taskIfcSiUnit;
return addGroupStringToDictionary("IfcMeasureWithUnit");
});
//some other tasks after.
When I do that job synchronously, all works fine, but when I do it in async, I have some random errors. At every test, the errors come randomly.
The only thing I see that could go wrong, is they all execute the same function addGroupStringToDictionary.
Here is the function :
private bool addGroupStringToDictionary(string typeName)
{
//int processCount = await Task.Run<int>(() =>
//{
GroupedListStrings groupElt = this.listGrouppedStrings.FirstOrDefault(x => x.Type == typeName.ToUpper());
if (groupElt != null)
{
List<string> listStringInGroup = groupElt.ListStrings;
foreach (string line in listStringInGroup)
{
try
{
if(typeName== "IfcLocalPlacement($")
{
typeName = "IfcLocalPlacement";
}
var type = Type.GetType("Ifc."+typeName);
if (typeName == "IfcPropertySingleValue" || typeName == "IfcDirection" || typeName == "IfcSiUnit" || typeName == "IfcQuantityLength" || typeName == "IfcQuantityArea" || typeName == "IfcQuantityVolume" || typeName == "IfcQuantityWeight")
{
try
{
object instance = Activator.CreateInstance(type, line);
this.addToListDictionary((IfcElement)instance);
}
catch
{
}
}
else if (typeName == "IfcOpeningElement")
{
try
{
object instance = Activator.CreateInstance(type, line, this.listDictionaries, this.DictionaryBolts);
this.addToListDictionary((IfcElement)instance);
}
catch
{
}
}
else
{
try
{
object instance = Activator.CreateInstance(type, line, this.listDictionaries);
this.addToListDictionary((IfcElement)instance);
}
catch
{
}
}
}
catch
{
this.addError(line);
}
}
this.listGrouppedStrings.Remove(groupElt);
this.reportProgressImport();
}
//return 100;
//});
this.reportProgressImport();
return true;
}
The catch got 1-2 times over a bit more than 1 million lines.
At each test the errors come randomly.
Is it possible that running the function simultaneously from several async methods, this is what causes the problem?
Here is the addToListDictionary function :
private void addToListDictionary(IfcElement elt)
{
if(elt.ErrorFound)
{
this.listReadButError.Add(elt);
return;
}
string type = elt.GetType().ToString();
if (elt is IfcRepere)
{
type = "Ifc.IfcRepere";
}
else if (elt is IfcRepereType)
{
type = "Ifc.IfcRepereType";
}
else if (elt is IfcPhysicalSimpleQuantity)
{
type = "Ifc.IfcPhysicalSimpleQuantity";
}
else if (elt is IfcProfileDef)
{
type = "Ifc.IfcProfileDef";
}
else if (elt is IfcGeometricRepresentationContext)
{
type = "Ifc.IfcGeometricRepresentationContext";
}
GroupDictionary group = this.ListDictionaries.FirstOrDefault(x => x.Name == type);
if(group==null)
{
group = new GroupDictionary { Name = type };
this.ListDictionaries.Add(group);
}
group.ListElements[elt.ID] = elt;
if (elt is IfcMechanicalFastener)
{
IfcMechanicalFastener bolt = (IfcMechanicalFastener)elt;
this.DictionaryBolts[bolt.Tag] = bolt;
}
else if(elt is IfcProject)
{
this.listProjects.Add((IfcProject)elt);
}
else if(elt is IfcElementAssembly ifcAss)
{
this.DictionaryIfcElementAssemblies[ifcAss.Key] = ifcAss;
}
}
Also some additive information about my ListDictionaries :
private List<GroupDictionary> listDictionaries = new List<GroupDictionary>();
public List<GroupDictionary> ListDictionaries { get { return this.listDictionaries; } set { this.listDictionaries = value; } }
And the class GroupDictionary
public class GroupDictionary
{
string name { get; set; }
public string Name { get { return this.name; } set { this.name = value; } }
public ConcurrentDictionary<int, IfcElement> ListElements = new ConcurrentDictionary<int, IfcElement>();
public GroupDictionary()
{
}
}
I made different GroupDictionary because as soon as I don't need one of them, I delete it to free space.
I have one dictionary with IfcPoint, I need it to gt IfcPolyLine (lines), but when I finish to treat all objects using IfcPoint, I clear remove the corresponding GroupDictionary in order to free some memory.

You have a obvious thread-safety issues here (i.e. you are trying to perform some operation which is not thread safe from multiple threads at a time). There are multiple ways you can try tackling it - using locks, or some synchronization primitives.
But in this case it seems that major source of issues is working with standard collections from multiple threads, which is not thread-safe (because thread-safety usually comes with performance price and is not always needed). You can start from switching to appropriate collections from System.Collections.Concurrent namespace.
To go down deeper I recommend free e-book by Joseph Albahari Threading in C#.

Rx: Count of Grouped Events in Moving Window

I have started looking at using Reactive Extensions with EventStore. As a proof of concept, I'd like to see if I can get Rx to consume an event stream and output the count of events grouped by type for a window of one second.
So, say that I am consuming a stream with the name "orders", I'd like to see something like the following appear in the console:
OrderCreated 201
OrderUpdated 111
(a second passes..)
OrderCreated 123
OrderUpdated 132
And so on.
So far, I have been able to get an output of the count of all events per second. But can't seem to be able to group them by event type.
The code I am using is based on a gist by James Nugent:
internal class EventStoreRxSubscription
{
public Subject<ResolvedEvent> ResolvedEvents { get; }
public Subject<SubscriptionDropReason> DroppedReasons { get; }
public EventStoreSubscription Subscription { get; }
public EventStoreRxSubscription(EventStoreSubscription subscription, Subject<ResolvedEvent> resolvedEvent, Subject<SubscriptionDropReason> droppedReasons)
{
Subscription = subscription;
ResolvedEvents = resolvedEvent;
DroppedReasons = droppedReasons;
}
}
static class EventStoreConnectionExtensions
{
public static Task<EventStoreRxSubscription> SubscribeTo(this IEventStoreConnection connection, string streamName, bool resolveLinkTos)
{
return Task<EventStoreRxSubscription>.Factory.StartNew(() => {
var resolvedEvents = new Subject<ResolvedEvent>();
var droppedReasons = new Subject<SubscriptionDropReason>();
var subscriptionTask = connection.SubscribeToStreamAsync(streamName, resolveLinkTos,
(subscription, #event) => resolvedEvents.OnNext(#event),
(subscription, dropReason, arg3) => droppedReasons.OnNext(dropReason));
subscriptionTask.Wait();
return new EventStoreRxSubscription(subscriptionTask.Result, resolvedEvents, droppedReasons);
});
}
}
class Program
{
static void Main(string[] args)
{
var connection = EventStoreConnection.Create(new IPEndPoint(IPAddress.Loopback, 1113));
connection.ConnectAsync();
var subscriptionTask = connection.SubscribeTo("orders", true);
subscriptionTask.Wait();
var events = subscriptionTask.Result.ResolvedEvents;
var query = events.Timestamp()
.Buffer(TimeSpan.FromSeconds(1))
.Select(e => e.Count);
query.Subscribe(Console.WriteLine);
Console.ReadLine();
}
}

I have done something similar to this before, I used Throttle to group all events within a set frequency, however you could use Buffer to get the count/collection for every period.
The example below provides an abstract example of how I achieved this, where AggregateType and AggregateFunction would be replaced by your own type and aggregation.
GroupByUntil allows you to group by a type within a set period.
subscription = observable
.GroupByUntil(e => e.Key, e => e.Buffer(TimeSpan.FromSeconds(1)))
.SelectMany(e => e.Aggregate(new AggregateType(), (a, e) => AggregateFunction(a, e))
.Subscribe(onNext, onError, onCompleted);
EDIT
Below is a quick example I've knocked up to show how it can be done
public class EventType
{
public string Type { get; set; }
}
public class AggregatedType
{
public string EventType { get; set; }
public int Count { get; set; }
}
class Program
{
public delegate void ExampleEventHandler(EventType e);
public static event ExampleEventHandler Event;
static void Main(string[] args)
{
var subscription = Observable.FromEvent<ExampleEventHandler, EventType>(e => Event += e, e => Event -= e)
.GroupByUntil(e => e.Type, e => e.Buffer(TimeSpan.FromSeconds(1)))
.SelectMany(e => e
.Select(ev => new AggregatedType { EventType = ev.Type })
.Aggregate(new AggregatedType(), (a, ev) => new AggregatedType { EventType = ev.EventType, Count = a.Count + 1 }))
.Subscribe(OnAggregaredEvent, OnException, OnCompleted);
Event(new EventType { Type = "A" });
Event(new EventType { Type = "A" });
Event(new EventType { Type = "B" });
Event(new EventType { Type = "B" });
SpinWait.SpinUntil(()=> false, TimeSpan.FromSeconds(2));
Event(new EventType { Type = "A" });
Event(new EventType { Type = "A" });
Event(new EventType { Type = "B" });
Event(new EventType { Type = "B" });
Console.ReadLine();
}
static void OnAggregaredEvent(AggregatedType aggregated)
{
Console.WriteLine("Type: {0}, Count: {1}", aggregated.EventType, aggregated.Count);
}
static void OnException(Exception ex)
{
}
static void OnCompleted()
{
}
}

Await on item being present in a list

I'm currently looking for a way to monitor a list for an item to be added (although it may already be in the list) with a certain ID. The below example demonstrates what I want to do, however I was hoping there would be a neater approach to this.
var list = new List<string>();
var task = new Task<bool>(() =>
{
for (int i = 0; i < 10; i++)
{
if (list.Contains("Woft"))
return true;
Thread.Sleep(1000);
}
return false;
});
I would appreciate any suggestions, and can elaborate if required.
EDIT: I'm not sure how I would use a CollectionChanged handler for this, still not sure I'm describing the problem well enough, here is a more complete example.
class Program
{
private static List<string> receivedList = new List<string>();
static void Main(string[] args)
{
var thing = new item() { val = 1234, text = "Test" };
var message = new message() { id = "12", item = thing };
//This would be a message send
//in the receiver we would add the message id to the received list.
var task2 = new Task(() =>
{
Thread.Sleep(2000);
receivedList.Add(message.id);
});
task2.Start();
var result = EnsureReceived(thing, message.id);
if (result == null)
{
Console.WriteLine("Message not received!");
}
else
{
Console.WriteLine(result.text + " " + result.val);
}
Console.ReadLine();
}
//This checks if the message has been received
private static item EnsureReceived(item thing, string id)
{
var task = new Task<bool>(() =>
{
for (int i = 0; i < 10; i++)
{
if (receivedList.Contains(id))
return true;
Thread.Sleep(1000);
}
return false;
});
task.Start();
var result = task.Result;
return result ? thing : null;
}
}
class message
{
public string id { get; set; }
public item item { get; set; }
}
class item
{
public string text { get; set; }
public int val { get; set; }
}

It sounds like what you want to do is use an ObservableCollection and subscribe to the CollectionChanged event. That way, you'll have an event fire whenever the collection is modified. At that point, you can check if the collection has the element you're expecting.
Even better, the CollectionChanged event handler will receive NotifyCollectionChangedEventArgs, which contains a NewItems property.
See MSDN for more information.

You can implement own type, what intercept all operations to add items and exposes ReadOnlyList. Then create an event ItemAdded with added item value (you will have to rise up multiple events when adding a collection obviously). Subscribe to it, check, do whatever.

Prioritized subscriptions invocation

I have a single observable sequence of messages. There is a set of subscribers that can handle these messages. Each subscriber has an execution priority. Each message must be handled once by the highest priority subscriber chosen from the list of currently subscribed subscribers. Subscribers are constantly subscribed/unsubscribed from the sequence, so we don't know the number and priorities of subscribers when constructing the sequence. Is it a possible/viable solution using rx?
To illustrate:
public class Message
{
public string Value { get; set; }
public bool IsConsumed { get; set; }
}
var subject = new Subject<Message>();
var sequence = subject.Publish().RefCount();
Action<Message, int> subscriber = (m, priority) =>
{
if (!m.IsConsumed)
{
m.IsConsumed = true;
Trace.WriteLine(priority);
}
};
var s2 = sequence.Priority(2).Subscribe(m => subscriber(m, 2));
var s1 = sequence.Priority(1).Subscribe(m => subscriber(m, 1));
subject.OnNext(new Message()); // output: 1
s1.Dispose();
subject.OnNext(new Message()); // output: 2
The missing piece to make this solution work is the Priority method which do not exist in Rx library.

This was a very interesting problem...
So, first off: I am not aware of any intrinsic Rx operators that can achieve a "routing" effect similar to what you want in this Priority extension.
That said, I was playing around in LINQPad over lunch today, and came up with a (very) hacky proof of concept that appears to work:
First, your message class
public class Message
{
public string Value { get; set; }
public bool IsConsumed { get; set; }
}
Next, the extension method wrapper-class:
public static class Ext
{
public static PrioritizedObservable<T> Prioritize<T>(this IObservable<T> source)
{
return new PrioritizedObservable<T>(source);
}
}
And what is this PrioritizedObservable<T>?
public class PrioritizedObservable<T>
: IObservable<T>, IObserver<T>, IDisposable
{
private IObservable<T> _source;
private ISubject<T,T> _intermediary;
private IList<Tuple<int, Subject<T>>> _router;
public PrioritizedObservable(IObservable<T> source)
{
// Make sure we don't accidentally duplicate subscriptions
// to the underlying source
_source = source.Publish().RefCount();
// A proxy from the source to our internal router
_intermediary = Subject.Create(this, _source);
_source.Subscribe(_intermediary);
// Holds per-priority subjects
_router = new List<Tuple<int, Subject<T>>>();
}
public void Dispose()
{
_intermediary = null;
foreach(var entry in _router)
{
entry.Item2.Dispose();
}
_router.Clear();
}
private ISubject<T,T> GetFirstListener()
{
// Fetch the first subject in our router
// ordered by priority
return _router.OrderBy(tup => tup.Item1)
.Select(tup => tup.Item2)
.FirstOrDefault();
}
void IObserver<T>.OnNext(T value)
{
// pass along value to first in line
var nextListener = GetFirstListener();
if(nextListener != null)
nextListener.OnNext(value);
}
void IObserver<T>.OnError(Exception error)
{
// pass along error to first in line
var nextListener = GetFirstListener();
if(nextListener != null)
nextListener.OnError(error);
}
void IObserver<T>.OnCompleted()
{
var nextListener = GetFirstListener();
if(nextListener != null)
nextListener.OnCompleted();
}
public IDisposable Subscribe(IObserver<T> obs)
{
return PrioritySubscribe(1, obs);
}
public IDisposable PrioritySubscribe(int priority, IObserver<T> obs)
{
var sub = new Subject<T>();
var subscriber = sub.Subscribe(obs);
var entry = Tuple.Create(priority, sub);
_router.Add(entry);
_intermediary.Subscribe(sub);
return Disposable.Create(() =>
{
subscriber.Dispose();
_router.Remove(entry);
});
}
}
And a test harness:
void Main()
{
var subject = new Subject<Message>();
var sequence = subject.Publish().RefCount().Prioritize();
Action<Message, int> subscriber = (m, priority) =>
{
if (!m.IsConsumed)
{
m.IsConsumed = true;
Console.WriteLine(priority);
}
};
var s3 = sequence.PrioritySubscribe(3, Observer.Create<Message>(m => subscriber(m, 3)));
var s2 = sequence.PrioritySubscribe(2, Observer.Create<Message>(m => subscriber(m, 2)));
var s1 = sequence.PrioritySubscribe(1, Observer.Create<Message>(m => subscriber(m, 1)));
var s11 = sequence.PrioritySubscribe(1, Observer.Create<Message>(m => subscriber(m, 1)));
subject.OnNext(new Message()); // output: 1
s1.Dispose();
subject.OnNext(new Message()); // output: 1
s11.Dispose();
subject.OnNext(new Message()); // output: 2
s2.Dispose();
subject.OnNext(new Message()); // output: 3
sequence.Dispose();
}

ServiceStack Redis how to implement paging

I am trying to find out how to do paging in SS.Redis, I use:
var todos = RedisManager.ExecAs<Todo>(r => r.GetLatestFromRecentsList(skip,take));
it returns 0, but i am sure the database is not empty, because r.GetAll() returns a list of things. What is the correct way to do this?
EDIT: Here is the code:
public class ToDoRepository : IToDoRepository
{
public IRedisClientsManager RedisManager { get; set; } //Injected by IOC
public Todo GetById(long id) {
return RedisManager.ExecAs<Todo>(r => r.GetById(id));
}
public IList<Todo> GetAll() {
return RedisManager.ExecAs<Todo>(r => r.GetAll());
}
public IList<Todo> GetAll(int from, int to) {
var todos = RedisManager.ExecAs<Todo>(r => r.GetLatestFromRecentsList(from,to));
return todos;
}
public Todo NewOrUpdate(Todo todo) {
RedisManager.ExecAs<Todo>(r =>
{
if (todo.Id == default(long)) todo.Id = r.GetNextSequence(); //Get next id for new todos
r.Store(todo); //save new or update
});
return todo;
}
public void DeleteById(long id) {
RedisManager.ExecAs<Todo>(r => r.DeleteById(id));
}
public void DeleteAll() {
RedisManager.ExecAs<Todo>(r => r.DeleteAll());
}
}

As I don't see any code, I'm assuming you're not maintaining the Recents list when you're adding the entites. Here's the test case for GetLatestFromRecentsList:
var redisAnswers = Redis.As<Answer>();
redisAnswers.StoreAll(q1Answers);
q1Answers.ForEach(redisAnswers.AddToRecentsList); //Adds to the Recents List
var latest3Answers = redisAnswers.GetLatestFromRecentsList(0, 3);
var i = q1Answers.Count;
var expectedAnswers = new List<Answer>
{
q1Answers[--i], q1Answers[--i], q1Answers[--i],
};
Assert.That(expectedAnswers.EquivalentTo(latest3Answers));
Redis StackOverflow is another example that uses the Recents list feature to show the latest Questions added. It maintains the recent list of questions by calling AddToRecentsList whenever a new Question is created.