I'm getting this exception with this code and can't understand why
private static void LoopBTCtx()
{
Task.Factory.StartNew(async () =>
{
while (true)
{
try
{
Thread.Sleep((int)TimeSpan.FromSeconds(10).TotalMilliseconds);
List<(string, SocketMessage, int)> _btcTX = btcTX;
foreach (var tx in btcTX)
{
int newConfirmations = GetBTCtxConfirmations(tx.Item1);
if (tx.Item3 != newConfirmations)
{
_btcTX.Remove(tx);
if (newConfirmations < 6)
{
_btcTX.Add((tx.Item1, tx.Item2, newConfirmations));
}
await tx.Item2.Channel.SendMessageAsync($"{tx.Item2.Author.Mention}, ``{tx.Item1}`` now has **{newConfirmations}**/6 confirmation{(newConfirmations != 1 ? "s" : null)}.");
}
}
btcTX = _btcTX;
}
catch (Exception e)
{
Console.WriteLine(e);
}
}
});
}
It's thrown after processing the first list element (foreach)
The exception line from the stacktrace is the one containing foreach (var tx in btcTX)
I tried using 2 different lists then updating the main one once the foreach is done, as you can see in my code above, but it didn't fix.
You still have one list.
The following statement just causes _btcTX to point to the same list instance as btcTX:
List<(string, SocketMessage, int)> _btcTX = btcTX;
So actually the main list was modified in the Remove() and/or Add().
One way to remove/add the items is to perform a regular for loop with an index (from last to first), and then you will be able to remove/add items with no problem.
Another way would be to keep the foreach loop, but store the indices to be removed and the items to be added inside the loop and then perform the actual adding/removal after the loop (removing should be done from the last index to the first).
Related
I had some issues when adding to an Entity Framework DbSet from multiple threads from inside a ConcurrentDictionary ValueFactory method. I tried to eliminate that issue by introducing a lock statement. This seems to have some strange side effects though. In some rare and random cases, my code throws a KeyNotFoundException, even though the programming should prevent that from happening. I guess that I oversee something.
using (ESBClient client = new ESBClient()) { // WCF SERVICE
client.Open();
// Limit the maximum number of parallel requests
var esbLimiter = new SemaphoreSlim(4);
ConcurrentDictionary<string, DataEntry> dataEntryDict = new ConcurrentDictionary<string, DataEntry>(
await db.DataEntries
.Where(de => allObjIDs.Contains(de.PAObjID))
.IncludeOptimized(de => de.WorkSchedules)
.ToDictionaryAsync(a => a.PAObjID, a => a)
);
// Get WorkOrderDataSet02 for each data entry number
await Task.WhenAll(allDataEntryNumbers.Batch(20).Select(async workOrderBatch => {
await esbLimiter.WaitAsync();
Debug.WriteLine($"Starting for new batch after {s.ElapsedMilliseconds} with parallel {esbLimiter.CurrentCount}");
try {
int retryCounter = 0;
getWorkOrderDataSet02Response gwoResp;
retryCurrentWorkOrderDataSetResp:
try {
gwoResp = await client.getWorkOrderDataSet02Async(
new getWorkOrderDataSet02Request(
"?",
companyGroup.Key,
string.Join(",", workOrderBatch.Select(wob => wob.DataEntryNumber)),
"WNTREIB",
"?",
"act,sales",
"D"
)
);
} catch (System.ServiceModel.CommunicationException ex) {
// Retry up to 3 times before finally crashing
if (retryCounter++ < 3) {
await HandleServiceRetryError("getWorkOrderDataSet02Async", retryCounter, s.ElapsedMilliseconds, ex);
goto retryCurrentWorkOrderDataSetResp;
} else
throw;
}
// Iterate over all work orders returned by the ESB
foreach (dsyWorkOrder01TtyWorkOrder currDetail in gwoResp.dsyWorkOrder01) { // dsyWorkOrder01 IS AN ARRAY OF OBJECTS. IT COMES FROM A WCF CALL. PAObjID IS UNIQUE.
// Get or create element
DataEntry currentEntry = dataEntryDict.GetOrAdd(
currDetail.Obj,
key => {
DataEntry newDe = new DataEntry();
lock (db.DataEntries) { // I INTRODUCED THOSE LOCK STATEMENTS
db.DataEntries.Add(newDe); // THIS IS THE LINE THAT WAS PROBLEMATIC IN THE FIRST PLACE
}
return newDe;
}
);
// Set regular fields
currentEntry.ApplyTtyWorkOrder(currDetail, resourceDict); // THIS METHOD APPLIES THE PAObjID PROPERTY
}
// Delete all elements, that were not provided by the service anymore
lock(db.DataEntries) {
workOrderBatch
.Where(wob => !gwoResp.dsyWorkOrder01
.Where(wo => wo.DataEntryNumber.HasValue)
.Select(wo => wo.DataEntryNumber.Value)
.Contains(wob.DataEntryNumber)
)
.ToArray()
.ForEach(dataEntry => {
try {
db.DataEntries.Remove(dataEntryDict[dataEntry.ObjID]); // THIS LINE THROWS THE KeyNotFoundException
} catch (Exception ex) {
throw new Exception($"Key {dataEntry.ObjID} not in list.", ex);
}
});
}
// Update progress
progress.Report(.1f + totalSteps * Interlocked.Increment(ref currentStep) * .8f);
} finally {
Debug.WriteLine($"Finished for batch after {s.ElapsedMilliseconds} with parallel {esbLimiter.CurrentCount}");
esbLimiter.Release();
}
}));
}
// HERE'S THE APPLY METHOD
public void ApplyTtyWorkOrder(dsyWorkOrder01TtyWorkOrder src, Dictionary<(string Name, byte ResourceType), int> resourceDict) {
Deleted = false;
DataEntryNumber = src.DataEntryNumber.Value;
PAObjID = src.Obj; // PAObjID IS APPLIED HERE
IsHeader = src.IsHeader;
Pieces = Convert.ToInt16(src.ProductionQty);
PartNo = src.Article;
JobNo = src.WorkOrder;
StartDate = src.StartDate;
FinishDate = src.EndDate;
FinishedPA = src.WorkOrderStatus == "R";
// Update methods
UpdateFromTtyCustomer(src.ttyCustomer?.FirstOrDefault());
UpdateFromPart(src.ttyPart?.FirstOrDefault());
UpdateFromSalesDocHeader(src.ttySalesDocHeader?.FirstOrDefault());
UpdateWorkSchedules(src.ttyWorkOrderActivity, resourceDict);
}
I added an UPPERCASE comment to every line I would consider relevant.
I have no idea why this error happens. From my understanding, I only try to get an entry from the dataEntryDict dictionary dataEntry.ObjID keys that I've added before in the same iteration of the loop.
Before I introduced the two lock statements, the line marked with "THIS IS THE LINE THAT WAS PROBLEMATIC IN THE FIRST PLACE" throw an exception sporadically: "Collection was modified; enumeration operation may not execute." After digging into the code of EF, I realized that this should have something to do with the way how the DbSet.Add method is implemented.
Are there any known side effects when using a lock statement inside the ValueFactory?
lock (db.DataEntries) { // I INTRODUCED THOSE LOCK STATEMENTS
db.DataEntries.Add(newDe); // THIS IS THE LINE THAT WAS PROBLEMATIC IN THE FIRST PLACE
}
The issue is that db.DataEntries is not a thread-safe collection but it is being accessed concurrently by multiple threads. All EF objects are not thread-safe.
Using locking seems like a good solution here. Make sure that you catch all the places.
It is often better to split the concurrent part off from the sequential part. Make only the client.getWorkOrderDataSet02Async call concurrent and collect the results in a collection. Then, process the results sequentially.
So my parallel foreach sometimes gets stuck and when debugging I try break all to see where I'm at and it's always on the beginning of the parallel foreach.
It doesn't do it every time but when it does it , it's after a while.
Can you guys help me out figure it out? I don't really understand why beside the fact that it could be because every threads are accessing the list of checkedPogos
public static Task<List<string>> Checks(List<string> pogos)
{
arret = false;
var pogoTask= new Task<List<string>>(() =>
{
List<string> checkedPogos= new List<string>();
Parallel.ForEach(pogos, (string pogo, ParallelLoopState state) =>
{
if (arret)
state.Stop();
pogo= pogo.Trim();
if (pogo != null && Check(pogo))
{
checkedPogos.Add(pogo);
}
});
return checkedPogos.Distinct().ToList();
});
pogoTask.Start();
return pogoTask;
}
EDIT : I've changed the List to a concurrent collection and it still didn't fix the problem.
I'm trying to take all items in one fell swoop from a ConcurrentBag. Since there's nothing like TryEmpty on the collection, I've resorted to using Interlocked.Exchange in the same fashion as described here: How to remove all Items from ConcurrentBag?
My code looks like this:
private ConcurrentBag<Foo> _allFoos; //Initialized in constructor.
public bool LotsOfThreadsAccessingThisMethod(Foo toInsert)
{
this._allFoos.Add(toInsert);
return true;
}
public void SingleThreadProcessingLoopAsALongRunningTask(object state)
{
var token = (CancellationToken) state;
var workingSet = new List<Foo>();
while (!token.IsCancellationRequested)
{
if (!workingSet.Any())
{
workingSet = Interlocked.Exchange(ref this._allFoos, new ConcurrentBag<Foo>).ToList();
}
var processingCount = (int)Math.Min(workingSet.Count, TRANSACTION_LIMIT);
if (processingCount > 0)
{
using (var ctx = new MyEntityFrameworkContext())
{
ctx.BulkInsert(workingSet.Take(processingCount));
}
workingSet.RemoveRange(0, processingCount);
}
}
}
The problem is that this sometimes misses items that are added to the list. I've written a test application that feeds data to my ConcurrentBag.Add method and verified that it is sending all of the data. When I set a breakpoint on the Add call and check the count of the ConcurrentBag after, it's zero. The item just isn't being added.
I'm fairly positive that it's because the Interlocked.Exchange call doesn't use the internal locking mechanism of the ConcurrentBag so it's losing data somewhere in the swap, but I have no knowledge of what's actually happening.
How can I just grab all the items out of the ConcurrentBag at one time without resorting to my own locking mechanism? And why does Add ignore the item?
I think taking all the items from the ConcurentBag is not needed. You can achieve exactly the same behavior you are trying to implement simply by changing the processing logic as follows (no need for own synchronization or interlocked swaps):
public void SingleThreadProcessingLoopAsALongRunningTask(object state)
{
var token = (CancellationToken)state;
var buffer = new List<Foo>(TRANSACTION_LIMIT);
while (!token.IsCancellationRequested)
{
Foo item;
if (!this._allFoos.TryTake(out item))
{
if (buffer.Count == 0) continue;
}
else
{
buffer.Add(item);
if (buffer.Count < TRANSACTION_LIMIT) continue;
}
using (var ctx = new MyEntityFrameworkContext())
{
ctx.BulkInsert(buffer);
}
buffer.Clear();
}
}
With the new ConcurrentBag<T> in .NET 4, how do you remove a certain, specific object from it when only TryTake() and TryPeek() are available?
I'm thinking of using TryTake() and then just adding the resulting object back into the list if I don't want to remove it, but I feel like I might be missing something. Is this the correct way?
The short answer: you can't do it in an easy way.
The ConcurrentBag keeps a thread local queue for each thread and it only looks at other threads' queues once its own queue becomes empty. If you remove an item and put it back then the next item you remove may be the same item again. There is no guarantee that repeatedly removing items and putting them back will allow you to iterate over the all the items.
Two alternatives for you:
Remove all items and remember them, until you find the one you want to remove, then put the others back afterwards. Note that if two threads try to do this simultaneously you will have problems.
Use a more suitable data structure such as ConcurrentDictionary.
You can't. Its a bag, it isn't ordered. When you put it back, you'll just get stuck in an endless loop.
You want a Set. You can emulate one with ConcurrentDictionary. Or a HashSet that you protect yourself with a lock.
The ConcurrentBag is great to handle a List where you can add items and enumerate from many thread, then eventually throw it away as its name is suggesting :)
As Mark Byers told, you can re-build a new ConcurrentBag that does not contains the item you wish to remove, but you have to protect this against multiple threads hits using a lock. This is a one-liner:
myBag = new ConcurrentBag<Entry>(myBag.Except(new[] { removedEntry }));
This works, and match the spirit in which the ConcurrentBag has been designed for.
As you mention, TryTake() is the only option. This is also the example on MSDN. Reflector shows no other hidden internal methods of interest either.
Mark is correct in that the ConcurrentDictionary is will work in the way you are wanting. If you wish to still use a ConcurrentBag the following, not efficient mind you, will get you there.
var stringToMatch = "test";
var temp = new List<string>();
var x = new ConcurrentBag<string>();
for (int i = 0; i < 10; i++)
{
x.Add(string.Format("adding{0}", i));
}
string y;
while (!x.IsEmpty)
{
x.TryTake(out y);
if(string.Equals(y, stringToMatch, StringComparison.CurrentCultureIgnoreCase))
{
break;
}
temp.Add(y);
}
foreach (var item in temp)
{
x.Add(item);
}
public static void Remove<T>(this ConcurrentBag<T> bag, T item)
{
while (bag.Count > 0)
{
T result;
bag.TryTake(out result);
if (result.Equals(item))
{
break;
}
bag.Add(result);
}
}
This is my extension class which I am using in my projects. It can a remove single item from ConcurrentBag and can also remove list of items from bag
public static class ConcurrentBag
{
static Object locker = new object();
public static void Clear<T>(this ConcurrentBag<T> bag)
{
bag = new ConcurrentBag<T>();
}
public static void Remove<T>(this ConcurrentBag<T> bag, List<T> itemlist)
{
try
{
lock (locker)
{
List<T> removelist = bag.ToList();
Parallel.ForEach(itemlist, currentitem => {
removelist.Remove(currentitem);
});
bag = new ConcurrentBag<T>();
Parallel.ForEach(removelist, currentitem =>
{
bag.Add(currentitem);
});
}
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
public static void Remove<T>(this ConcurrentBag<T> bag, T removeitem)
{
try
{
lock (locker)
{
List<T> removelist = bag.ToList();
removelist.Remove(removeitem);
bag = new ConcurrentBag<T>();
Parallel.ForEach(removelist, currentitem =>
{
bag.Add(currentitem);
});
}
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
}
public static ConcurrentBag<String> RemoveItemFromConcurrentBag(ConcurrentBag<String> Array, String Item)
{
var Temp=new ConcurrentBag<String>();
Parallel.ForEach(Array, Line =>
{
if (Line != Item) Temp.Add(Line);
});
return Temp;
}
how about:
bag.Where(x => x == item).Take(1);
It works, I'm not sure how efficiently...
edit: in case anyone is wondering, the actionhandler invokes code that creates and disposes the same kind of datacontext, in case that might have anything to do with this behaviour. the code doesn't touch the MatchUpdateQueue table, but i figure i should mention it just in case.
double edit: everyone who answered was correct! i gave the answer to the respondent who suffered most of my questioning. fixing the problem allowed another problem (hidden within the handler) to pop up, which happened to throw exactly the same exception. whoops!
I'm having some issues with deleting items in LINQ. The DeleteOnSubmit call in the code below causes a LINQ Exception with the message "Cannot add an entity with a key that is already in use." I'm not sure what I'm doing wrong here, it is starting to drive me up the wall. The primary key is just an integer autoincrement column and I have no other problems until I try to remove an item from the database queue. Hopefully I'm doing something painfully retarded here that is easy to spot for anyone who isn't me!
static void Pacman()
{
Queue<MatchUpdateQueue> waiting = new Queue<MatchUpdateQueue>();
events.WriteEntry("matchqueue worker thread started");
while (!stop)
{
if (waiting.Count == 0)
{
/* grab any new items available */
aDataContext db = new aDataContext();
List<MatchUpdateQueue> freshitems = db.MatchUpdateQueues.OrderBy(item => item.id).ToList();
foreach (MatchUpdateQueue item in freshitems)
waiting.Enqueue(item);
db.Dispose();
}
else
{
/* grab & dispatch waiting item */
MatchUpdateQueue item = waiting.Peek();
try
{
int result = ActionHandler.Handle(item);
if (result == -1)
events.WriteEntry("unknown command consumed : " + item.actiontype.ToString(), EventLogEntryType.Error);
/* remove item from queue */
waiting.Dequeue();
/* remove item from database */
aDataContext db = new aDataContext();
db.MatchUpdateQueues.DeleteOnSubmit(db.MatchUpdateQueues.Single(i => i == item));
db.SubmitChanges();
db.Dispose();
}
catch (Exception ex)
{
events.WriteEntry("exception while handling item : " + ex.Message, EventLogEntryType.Error);
stop = true;
}
}
/* to avoid hammering database when there's nothing to do */
if (waiting.Count == 0)
Thread.Sleep(TimeSpan.FromSeconds(10));
}
events.WriteEntry("matchqueue worker thread halted");
}
You could do something to the effect of
db.MatchUpdateQueues.DeleteOnSubmit(db.MatchUpdateQueues.Single(theItem => theItem == item));
Just a note as other answers hinted towards Attach.. you will not be able to use attach on a context other then the original context the item was received on unless the entity has been serialized.
Try wrapping the entire inside of the while loop in a using statement for a single data context:
Queue<MatchUpdateQueue> waiting = new Queue<MatchUpdateQueue>();
events.WriteEntry("matchqueue worker thread started");
while (!stop)
{
using (var db = new aDataContext())
{
if (waiting.Count == 0)
{
/* grab any new items available */
List<MatchUpdateQueue> freshitems = db.MatchUpdateQueues
.OrderBy(item => item.id)
.ToList();
foreach (MatchUpdateQueue item in freshitems)
waiting.Enqueue(item);
}
...
}
}
Use:
aDataContext db = new aDataContext();
item = new MatchUpdateQueue { id=item.id }; // <- updated
db.MatchUpdateQueues.Attach(item);
db.MatchUpdateQueues.DeleteOnSubmit(item);
db.SubmitChanges();
Since you are using a new datacontext it doesn't know that the object is already in the db.
Remove the first db.Dispose() dispose. It can be the problem code because the entities keep a reference to their data context, so you don't want to dispose it while you are still be working with the instances. This won't affect connections, as they are open/closed only when doing operations that need them.
Also don't dispose the second data context like that, since an exception won't call that dispose code anyway. Use the using keyword, which will make sure to call dispose whether or not an exception occurs. Also grab the item from the db to delete it (to avoid having to serialize/deserialize/attach).
using (aDataContext db = new aDataContext())
{
var dbItem = db.MatchUpdateQueues.Single(i => i.Id == item.Id);
db.MatchUpdateQueues.DeleteOnSubmit(dbItem);
db.SubmitChanges();
}
try this if your TEntity's (here Area) Primary Key is of type Identity column;
Just it, without any change in your SP or Model:
public void InitForm()
{
'bnsEntity is a BindingSource and cachedAreas is a List<Area> created from dataContext.Areas.ToList()
bnsEntity.DataSource = cachedAreas;
'A nominal ID
newID = cachedAreas.LastOrDefault().areaID + 1;
'grdEntity is a GridView
grdEntity.DataSource = bnsEntity;
}
private void tsbNew_Click(object sender, EventArgs e)
{
var newArea = new Area();
newArea.areaID = newID++;
dataContext.GetTable<Area>().InsertOnSubmit(newArea);
bnsEntity.Add(newArea);
grdEntity.MoveToNewRecord();
}