We Keep Coding

How to Update or Add to a Collection properly in Parallel processing (ForEach)

I am trying to process a list of addresses into 2 collections; a list of addresses that meets requirements; a list of scores for each of those addresses that meets requirements.
I have several different criteria to slice and dice the addresses, but I am running into an incorrect implementation of modifying a collection during parallel processing.
For instance, I take the zip code and for each (+388k records) I determine if the Levenshtein Distance is greater or equal to a filtering value. If it does, then add the address to the compiled list of addresses. Then using the SET method, determine if the score for the address already exists. If is does not, then add. Else use the score currently in the collection and update the appropriate property.
I am hitting the "Collection Has Been Modified" error and I can not think of another way to implement the solution. I am looking for alternative ways to accomplish this.
private LevenshteinDistance _ld = new LevenshteinDistance();
private int _filter_zip = int.Parse(ConfigHelper.GetAppSettingByKey("filter_zip"));
public Factory ScoreAddresses(Factory model)
{
Factory result = model;
IList<Address> _compiledList = new List<Address>();
IList<Scores> _compiledScores = new List<Scores>();
ParallelLoopResult MailLoopResult = Parallel.ForEach(
result.ALL_Addresses, factory =>
{
if (factory.MAIL_ZIP_CODE_NBR != null
&& factory.MAIL_ZIP_CODE_NBR.Length >= 1)
{
int _zipDistance = _ld.Compute(
result.Target_Address.MAIL_ZIP_CODE_NBR,
factory.MAIL_ZIP_CODE_NBR);
if (_zipDistance <= _filter_zip)
{
_compiledList.Add(factory);
Scores _score = new Scores();
_compiledScores = _score.Set(_compiledScores,
factory.INDIVIDUAL_ID, "Levenshtein_MAIL_ZIP_CODE_NBR",
_zipDistance, string.Empty, null);
}
}
});
return result;
}
public IList<Scores> Set(IList<Scores> current, int Individual_ID,
string Score_Type, int Levenshtein, string Methaphone, decimal? other)
{
int currentcount = current.Count();
bool updating = false;
IList<Scores> result = current;
try
{
Scores newscore = new Scores();
//Check if scoreing for individual already present
Scores lookingforIndv = current.AsParallel().Where(
p => p.INDIVIDUAL_ID == Individual_ID).FirstOrDefault();
if (lookingforIndv != null)
{
newscore = lookingforIndv;
updating = true;
}
else
{
newscore.INDIVIDUAL_ID = Individual_ID;
updating = false;
}
//Add score based upon indicated score type
switch (Score_Type)
{
case "Levenshtein_MAIL_ZIP_CODE_NBR":
newscore.Levenshtein_MAIL_ZIP_CODE_NBR = Levenshtein;
result.Add(newscore);
break;
//More and More Cases
}
}
catch (Exception e)
{
}
return result;
}

Convert var into List<MapPdf>

I have a program that I used a variable of type List < MapPdf > that I will detach in variables when filling I would like to have the possibility to use it another time but I did not have the right to identify it again as
public static void Create(List<MapPdf> pps, Saison s, Agence agence)
{
foreach (var pelerins in grouped)
{
if (string.IsNullOrEmpty(pelerins.Key) || pelerins.Count() <= 0)
break;
if (writer.PageEvent == null)
{
writer.PageEvent = new Header_List()
{
travel = ctx.Travels.Include("Transport").First(v => v.UniqueId == pelerins.Key),
travelretour = ctx.Travels.Find(pelerins.First().UniqueIdRetour),
Agence = agence,
CountAllPelerin = pelerins.Count().ToString(),
CountFeminin = pelerins.Count(x => x.Sexe == PelerinSexe.Feminin).ToString(),
CountMasculin = pelerins.Count(x => x.Sexe == PelerinSexe.Masculin).ToString(),
NomGroupe = pelerins.First().Groupe,
NumeroDoc = writer.PageNumber
};
}
}
}
And i want to use pelerins as a List when i used in another function when it is of this declaration
I used List < MapPdf > pls = pelerins.ToList(); but it does not work
CreateFr(pls, false, cb, s);

If you are referring to var pelerins within the for-each loop and if I understand the problem, you are unable use it into another method , because var pelerins is a local variable encapsulated within the for-each loop -It does not exist outside it.
You could do the following:
//public property to retrieve pelerins
public List <MapPdf> pls new List<MapPdf>();
...
...
public static void Create(List<MapPdf> pps, Saison s, Agence agence)
{
foreach (var pelerins in grouped)
{
if (string.IsNullOrEmpty(pelerins.Key) || pelerins.Count() <= 0)
break;
if (writer.PageEvent == null)
{
//do logic
...
//store the one you are interested in, so you can use it later on
pls = pelerins.ToList();
}
}
}

Algorithm for detecting cycles finds far to few cycles

I have the following algorithm:
class CycleData : List<IntPoint>{
public IntPoint startPoint;
public Boolean ended=false;
public CycleData(IntPoint startpoint) { startPoint = startpoint; base.Add(startpoint); }
}
class GeoDataGraphPoint
{
private IntPoint point;
private List<GeoDataGraphPoint> connected = new List<GeoDataGraphPoint>();
private int generation=-9999;
public void AddConnection(GeoDataGraphPoint c)
{
connected.Add(c);
c.connected.Add(this);
}
public GeoDataGraphPoint(IntPoint point)
{
this.point = point;
}
public List<CycleData> GetCycles(int gen)
{
if (generation != -9999)
{
var r = new CycleData(point);
return new List<CycleData> { r };
}
generation = gen;
List<CycleData> res = new List<CycleData>();
foreach (GeoDataGraphPoint p in connected)
{
if (p.generation != gen-1)
{
res.AddRange(p.GetCycles(gen + 1));
}
}
foreach (CycleData list in res)
{
if (list.ended == false)
{
list.Add(point);
if (list.startPoint == this.point)
{
list.ended = false;
}
}
}
gen = -9999;
return res;
}
}
Now in principle this should return every cycle in the graph (for polygon detection). However it seems to fail to return anything in some occasions, I suspect that there is some kind of memory problem as removing parts of the graph sometimes causes new cycles to be found.
Here is a part of the input where it fails:
connection:(2282,3) to (2282,-192)
connection:(2282,3) to (2085,3)
connection:(2282,-192) to (2282,3)
connection:(2282,-192) to (2466,-192)
connection:(2466,-192) to (2282,-192)
connection:(2466,-192) to (2466,581)
connection:(2466,581) to (2466,-192)
connection:(2466,581) to (1494,581)
connection:(1494,581) to (2466,581)
connection:(1494,581) to (1494,397)
connection:(1494,397) to (1494,581)
connection:(1494,397) to (2282,397)
connection:(2282,397) to (1494,397)
connection:(2282,397) to (2282,187)
connection:(2282,187) to (2282,397)
connection:(2282,187) to (2085,187)
connection:(2085,187) to (2282,187)
connection:(2085,187) to (2085,3)
connection:(2085,3) to (2085,187)
connection:(2085,3) to (2282,3)
connection:(2085,3) to (2085,187)
connection:(2085,3) to (2282,3)
connection:(2085,187) to (2282,187)
connection:(2085,187) to (2085,3)
connection:(2282,187) to (2282,397)
connection:(2282,187) to (2085,187)
connection:(2282,397) to (1494,397)
The above code is for two triangles arranged to form a square (in coordinates) where both sides touch one another so like this:
Where I can the function as following:
class GeoDataGraph : Dictionary<IntPoint, GeoDataGraphPoint>
{
public void resetGens()
{
foreach(var v in base.Values)
{
v.generation = -9999;
}
}
public static Island GetHolesInIsland(Island input)
{
GeoDataGraph graph = new GeoDataGraph();
for (int i = 0; i < input.area.Count-1; i = i + 2)
{
var p1 = new IntPoint(input.area[i].X, input.area[i].Y);
var p2 = new IntPoint(input.area[i + 1].X, input.area[i + 1].Y);
if (!graph.ContainsKey(p1)) graph.Add(p1, new GeoDataGraphPoint(p1));
if (!graph.ContainsKey(p2)) graph.Add(p2, new GeoDataGraphPoint(p2));
graph[p1].AddConnection(graph[p2]);
}
IntPoint min = new IntPoint(int.MaxValue, int.MaxValue);
List<IntPoint> minCycle = null;
List<List<IntPoint>> cycles = new List<List<IntPoint>>();
while (graph.Count != 0)
{
var first = graph.First();
var NewCycles = first.Value.GetCycles(1);
graph.resetGens();
if (NewCycles.Count == 0)
{
graph.Remove(first.Key);
Console.WriteLine("point" + first.Key + "is uncycled");
}
cycles.AddRange(NewCycles);
foreach (var cycle in NewCycles)
{
foreach (var cycleNode in cycle)
{
graph.Remove(cycleNode);
if (min.X > cycleNode.X || min.Y > cycleNode.Y)
{
minCycle = cycle;
min = cycleNode;
}
}
}
}
cycles.Remove(minCycle);
if (minCycle == null) { minCycle = new List<IntPoint>();
foreach(IntPoint a in input.area) {
Console.Write(a);
} }
input.holes = cycles;
input.area = minCycle;
return input;
}
}
}
where Island contains.area contains a list of points ordered by connected pairs.
The basic algorithm is simple: take a node recursively visit every node connected to it until you detect a cycle, then return that node and append any node on the way out until you find the start of the cycle again. Once you have found every node connected to the start node remove the cycles (they since we checked every node connected to the cycle we shouldn't be deleting the once we already did) and start again on the next node, if a node contains no cycles remove it. I suspect something might be going wrong at this step but am unsure.
Any idea what I'm doing wrong that causes a weird interdependence that causes seemingly unrelated polygons to go wrong?

I think that one problem is the way you ignore connected nodes using p.generation != gen-1. As you use depth first search you will tag all the nodes until the most depth and when backtracking I think it can miss some nodes or explore nodes twice.
As a general advise I can say: Don't reinvent the wheel yourself but use a known algorithm.
First ask yourself what you want to do. The number of cycles can be exponential. So the question is if you really need all the cycles.
If the answer is yes and you want to find all the cycles in a undirected graph you can get more information here.
If you don't really need all the cyles maybe what your are looking for is to find strongly connected components.

Most efficient structure for quick access to a random element + the last element on one criteria

We have a function that is informed that we received an item for a specific timestamp.
The purpose of this is to wait that for one specific timestamp, we wait that we receive every item that we are expecting, then push the notification further once we are "synchronized" with all items.
Currently, we have a Dictionary<DateTime, TimeSlot> to store the non-synchronized TimeSlot(TimeSlot = list of all items we received for a specific timestamp).
//Let's assume that this method is not called concurrently, and only once per "MyItem"
public void HandleItemReceived(DateTime timestamp, MyItem item){
TimeSlot slot;
//_pendingTimeSlot is a Dictionary<DateTime,TimeSlot>
if(!_pendingTimeSlot.TryGetValue(timestamp, out slot)){
slot = new TimeSlot(timestamp);
_pendingTimeSlot.Add(timestamp,slot );
//Sometimes we don't receive all the items for one timestamps, which may leads to some ghost-incomplete TimeSlot
if(_pendingTimeSlot.Count>_capacity){
TimeSlot oldestTimeSlot = _pendingTimeSlot.OrderBy(t=>t.Key).Select(t=>t.Value).First();
_pendingTimeSlot.Remove(oldestTimeSlot.TimeStamp);
//Additional work here to log/handle this case
}
}
slot.HandleItemReceived(item);
if(slot.IsComplete){
PushTimeSlotSyncronized(slot);
_pendingTimeSlot.Remove(slot.TimeStamp);
}
}
We have severals instances of this "Synchronizer" in parallels for differents group of items.
It's working fine, except when the system is under heavy loads, we have more incomplete TimeSlot, and the application uses a lot more CPU. The profiler seems to indicate that the Compare of the LINQ query is taking a lot of time(most of the time). So I'm trying to find some structure to hold those references(replace the dictionary)
Here are some metrics:
We have several(variable, but between 10 to 20) instances of this Synchronizer
The current maximum capacity(_capacity) of the synchronizer is 500 items
The shortest interval that we can have between two different timestamp is 100ms(so 10 new Dictionary entry per seconds for each Synchronizer)(most case are more 1 item/second)
For each timestamp, we expect to receive 300-500 items.
So we will do, for one Synchronizer, per second(worst case):
1 Add
500 Get
3-5 Sorts
What would be my best move? I thought to the SortedDictionary But I didn't find any documentation showing me how to take the first element according to the key.

The first thing you can try is eliminating the OrderBy - all you need is the minimum key, no need to sort for getting that:
if (_pendingTimeSlot.Count > _capacity) {
// No Enumerable.Min(DateTime), so doing it manually
var oldestTimeStamp = DateTime.MaxValue;
foreach (var key in _pendingTimeSlot.Keys)
if (oldestTimeStamp > key) oldestTimestamp = key;
_pendingTimeSlot.Remove(oldestTimeStamp);
//Additional work here to log/handle this case
}
What about SortedDictionary, it is an option for sure, although it will consume much more memory. Since it's sorted, you can use simply sortedDictionary.First() to take the key value pair with the minimum key (hence the oldest element in your case).
UPDATE: Here is a hybrid approach using dictionary for fast lookups and ordered double linked list for the other scenarios.
class MyItem
{
// ...
}
class TimeSlot
{
public readonly DateTime TimeStamp;
public TimeSlot(DateTime timeStamp)
{
TimeStamp = timeStamp;
// ...
}
public bool IsComplete = false;
public void HandleItemReceived(MyItem item)
{
// ...
}
// Dedicated members
public TimeSlot PrevPending, NextPending;
}
class Synhronizer
{
const int _capacity = 500;
Dictionary<DateTime, TimeSlot> pendingSlotMap = new Dictionary<DateTime, TimeSlot>(_capacity + 1);
TimeSlot firstPending, lastPending;
//Let's assume that this method is not called concurrently, and only once per "MyItem"
public void HandleItemReceived(DateTime timeStamp, MyItem item)
{
TimeSlot slot;
if (!pendingSlotMap.TryGetValue(timeStamp, out slot))
{
slot = new TimeSlot(timeStamp);
Add(slot);
//Sometimes we don't receive all the items for one timestamps, which may leads to some ghost-incomplete TimeSlot
if (pendingSlotMap.Count > _capacity)
{
// Remove the oldest, which in this case is the first
var oldestSlot = firstPending;
Remove(oldestSlot);
//Additional work here to log/handle this case
}
}
slot.HandleItemReceived(item);
if (slot.IsComplete)
{
PushTimeSlotSyncronized(slot);
Remove(slot);
}
}
void Add(TimeSlot slot)
{
pendingSlotMap.Add(slot.TimeStamp, slot);
// Starting from the end, search for a first slot having TimeStamp < slot.TimeStamp
// If the TimeStamps almost come in order, this is O(1) op.
var after = lastPending;
while (after != null && after.TimeStamp > slot.TimeStamp)
after = after.PrevPending;
// Insert the new slot after the found one (if any).
if (after != null)
{
slot.PrevPending = after;
slot.NextPending = after.NextPending;
after.NextPending = slot;
if (slot.NextPending == null) lastPending = slot;
}
else
{
if (firstPending == null)
firstPending = lastPending = slot;
else
{
slot.NextPending = firstPending;
firstPending.PrevPending = slot;
firstPending = slot;
}
}
}
void Remove(TimeSlot slot)
{
pendingSlotMap.Remove(slot.TimeStamp);
if (slot.NextPending != null)
slot.NextPending.PrevPending = slot.PrevPending;
else
lastPending = slot.PrevPending;
if (slot.PrevPending != null)
slot.PrevPending.NextPending = slot.NextPending;
else
firstPending = slot;
slot.PrevPending = slot.NextPending = null;
}
void PushTimeSlotSyncronized(TimeSlot slot)
{
// ...
}
}
Some additional usages:
Iterating from oldest to newest:
for (var slot = firstPending; slot != null; slot = slot.NextPending)
{
// do something
}
Iterating from oldest to newest and removing items based on a criteria:
for (TimeSlot slot = firstPending, nextSlot; slot != null; slot = nextSlot)
{
nextSlot = slot.NextPending;
if (ShouldRemove(slot))
Remove(slot);
}
Same for reverse scenarios, but using lastPending and PrevPending members instead.

Here is simple sample. The insert method in a list eliminates swapping elements.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
List<Data> inputs = new List<Data>() {
new Data() { date = DateTime.Parse("10/22/15 6:00AM"), data = "abc"},
new Data() { date = DateTime.Parse("10/22/15 4:00AM"), data = "def"},
new Data() { date = DateTime.Parse("10/22/15 6:30AM"), data = "ghi"},
new Data() { date = DateTime.Parse("10/22/15 12:00AM"), data = "jkl"},
new Data() { date = DateTime.Parse("10/22/15 3:00AM"), data = "mno"},
new Data() { date = DateTime.Parse("10/22/15 2:00AM"), data = "pqr"},
};
Data data = new Data();
foreach (Data input in inputs)
{
data.Add(input);
}
}
}
public class Data
{
public static List<Data> sortedData = new List<Data>();
public DateTime date { get; set; }
public string data { get; set;}
public void Add(Data newData)
{
if(sortedData.Count == 0)
{
sortedData.Add(newData);
}
else
{
Boolean added = false;
for(int index = sortedData.Count - 1; index >= 0; index--)
{
if(newData.date > sortedData[index].date)
{
sortedData.Insert(index + 1, newData);
added = true;
break;
}
}
if (added == false)
{
sortedData.Insert(0, newData);
}
}
}
}
}​

compare List<string> and List<T>

I'm using C# and framework 4.0.
I have a list of type string and another list of type class T;
How can I compare List with a List and save the difference?
private void simpleButton_Compare_Click(object sender, EventArgs e)
{
try
{
bool Is_Egal = true;
int i = 0;
foreach (string Od_Scan in Ordre_Scan)
{
if (!Outils.Get_Ordre_Donne()[i].NoOrdre.Contains(Od_Scan) && !String.IsNullOrWhiteSpace(Od_Scan))
{
Is_Egal = false;
Temp_Od_Scan.Add(Od_Scan);
}
i++;
}
foreach (Pers_Compare Od_Done in Outils.Get_Ordre_Donne())
{
if (!Ordre_Scan.Contains(Od_Done.NoOrdre) && !String.IsNullOrWhiteSpace(Od_Done.NoOrdre))
{
Is_Egal = false;
Temp_Od_Donne.Add(Od_Done);
}
else
{
Temp_Od_Donne_Egal.Add(Od_Done);
}
}
if (Is_Egal)
{
MessageBox.Show("égalité");
}
else
{
MessageBox.Show("PAS égalité");
}
}
catch (Exception excThrown)
{
MessageBox.Show(excThrown.Message);
}
}
and the data :
List<string> Ordre_Scan= new List<string> { "azer","qsdf"};
Pers_Compare obj = new Pers_Compare();
obj.Nolv = 1;
obj.Noordre = "qsdf"
Pers_Compare obj2 = new Pers_Compare();
obj2.Nolv = 1;
obj2.Noordre = "wxcv"
List<Pers_Compare> Ordre_Donne = new List<Pers_Compare>();
Ordre_Donne.add(obj);
Ordre_Donne.add(obj2);
And I want to save the data in Ordre_Donne but not in Od_Scan and vice versa.
foreach (string Od_Scan in Temp_Od_Scan)
{
all item that not found in List A
--> wxcv
}
foreach (var Od_Done in Temp_Od_Donne)
{
all item that not found in List B
--> azer
}

The answer given for a slightly different question (comparing a List with another List) seems to me to be a good solution for your issue, they address multiple issues to do with comparisons of lists.
EDIT: However you should be more specific with your requirements i.e. what exactly is a 'difference', e.g. is {1,1,2} and {1,2} the same?
Here is the answer given the most votes... (included here just encase it gets removed for some reason (as per Bob' suggestion))
"
DESCRIPTION:
I need to check that they both have the same elements, regardless of their position within the list. Each MyType object may appear multiple times on a list. Is there a built-in function that checks this? What if I guarantee that each element appears only once in a list?
EDIT: Guys thanks for the answers but I forgot to add something, the number of occurrences of each element should be the same on both lists.
ANSWER:
If you want them to be really equal (i.e. the same items and the same number of each item), I think that the simplest solution is to sort before comparing:
Enumerable.SequenceEqual(list1.OrderBy(t => t), list2.OrderBy(t => t))
Edit:
Here is a solution that performs a bit better (about ten times faster), and only requires IEquatable, not IComparable:
public static bool ScrambledEquals<T>(IEnumerable<T> list1, IEnumerable<T> list2) {
var cnt = new Dictionary<T, int>();
foreach (T s in list1) {
if (cnt.ContainsKey(s)) {
cnt[s]++;
} else {
cnt.Add(s, 1);
}
}
foreach (T s in list2) {
if (cnt.ContainsKey(s)) {
cnt[s]--;
} else {
return false;
}
}
return cnt.Values.All(c => c == 0);
}
Edit 2:
To handle any data type as key (for example nullable types as Frank Tzanabetis pointed out), you can make a version that takes a comparer for the dictionary:
public static bool ScrambledEquals<T>(IEnumerable<T> list1, IEnumerable<T> list2, IEqualityComparer<T> comparer) {
var cnt = new Dictionary<T, int>(comparer);
...
"

var list1 = Ordre_Donne.Where(o => !Ordre_Scan.Any(s => s == o.Noordre));
var list2 = Ordre_Scan.Where(s => !Ordre_Donne.Any(o => o.Noordre == s));

You can either implement IComparable on your Pers_Compare class, which will look something like:
public int CompareTo(string other)
{
return this.Noordre.CompareTo(other);
}
Or, if you don't have control of the data structure, you could do something like
var Temp_Od_Donne = from od in Ordre_Donne
where !Ordre_Scan.Contains(od.Noordre)
select od;
var Temp_Od_Scan = from os in Ordre_Scan
where !Ordre_Donne.Select(od => od.Noordre).Contains(os)
select os;