We Keep Coding

Is there a way to avoid using side effects to process this data

I have an application I'm writing that runs script plugins to automate what a user used to have to do manually through a serial terminal. So, I am basically implementing the serial terminal's functionality in code. One of the functions of the terminal was to send a command which kicked off the terminal receiving continuously streamed data from a device until the user pressed space bar, which would then stop the streaming of the data. While the data was streaming, the user would then set some values in another application on some other devices and watch the data streamed in the terminal change.
Now, the streamed data can take different shapes, depending on the particular command that's sent. For instance, one response may look like:
---RESPONSE HEADER---
HERE: 1
ARE: 2 SOME:3
VALUES: 4
---RESPONSE HEADER---
HERE: 5
ARE: 6 SOME:7
VALUES: 8
....
another may look like:
here are some values
in cols and rows
....
So, my idea is to have a different parser based on the command I send. So, I have done the following:
public class Terminal
{
private SerialPort port;
private IResponseHandler pollingResponseHandler;
private object locker = new object();
private List<Response1Clazz> response1;
private List<Response2Clazz> response2;
//setter omited for brevity
//get snapshot of data at any point in time while response is polling.
public List<Response1Clazz> Response1 {get { lock (locker) return new List<Response1Clazz>(response1); }
//setter omited for brevity
public List<Response2Clazz> Response2 {get { lock (locker) return new List<Response1Clazz>(response2); }
public Terminal()
{
port = new SerialPort(){/*initialize data*/}; //open port etc etc
}
void StartResponse1Polling()
{
Response1 = new List<Response1Clazz>();
Parser<List<Response1Clazz>> parser = new KeyValueParser(Response1); //parser is of type T
pollingResponseHandler = new PollingResponseHandler(parser);
//write command to start polling response 1 in a task
}
void StartResponse2Polling()
{
Response2 = new List<Response2Clazz>();
Parser<List<Response2Clazz>> parser = new RowColumnParser(Response2); //parser is of type T
pollingResponseHandler = new PollingResponseHandler(parser); // this accepts a parser of type T
//write command to start polling response 2
}
OnSerialDataReceived(object sender, Args a)
{
lock(locker){
//do some processing yada yada
//we pass in the serial data to the handler, which in turn delegates to the parser.
pollingResponseHandler.Handle(processedSerialData);
}
}
}
the caller of the class would then be something like
public class Plugin : BasePlugin
{
public override void PluginMain()
{
Terminal terminal = new Terminal();
terminal.StartResponse1Polling();
//update some other data;
Response1Clazz response = terminal.Response1;
//process response
//update more data
response = terminal.Response1;
//process response
//terminal1.StopPolling();
}
}
My question is quite general, but I'm wondering if this is the best way to handle the situation. Right now I am required to pass in an object/List that I want modified, and it's modified via a side effect. For some reason this feels a little ugly because there is really no indication in code that this is what is happening. I am purely doing it because the "Start" method is the location that knows which parser to create and which data to update. Maybe this is Kosher, but I figured it is worth asking if there is another/better way. Or at least a better way to indicate that the "Handle" method produces side effects.
Thanks!

I don't see problems in modifying List<>s that are received as a parameter. It isn't the most beautiful thing in the world but it is quite common. Sadly C# doesn't have a const modifier for parameters (compare this with C/C++, where unless you declare a parameter to be const, it is ok for the method to modify it). You only have to give the parameter a self-explaining name (like outputList), and put a comment on the method (you know, an xml-comment block, like /// <param name="outputList">This list will receive...</param>).
To give a more complete response, I would need to see the whole code. You have omitted an example of Parser and an example of Handler.
Instead I see a problem with your lock in { lock (locker) return new List<Response1Clazz>(response1); }. And it seems to be non-sense, considering that you then do Response1 = new List<Response1Clazz>();, but Response1 only has a getter.

block multiple request from same user id to a web method c#

I have a web method upload Transaction (ASMX web service) that take the XML file, validate the file and store the file content in SQL server database. we noticed that a certain users can submit the same file twice at the same time. so we can have the same codes again in our database( we cannot use unique index on the database or do anything on database level, don't ask me why). I thought I can use the lock statement on the user id string but i don't know if this will solve the issue. or if I can use a cashed object for storing all user id requests and check if we have 2 requests from the same user Id we will execute the first one and block the second request with an error message
so if anyone have any idea please help

Blocking on strings is bad. Blocking your webserver is bad.
AsyncLocker is a handy class that I wrote to allow locking on any type that behaves nicely as a key in a dictionary. It also requires asynchronous awaiting before entering the critical section (as opposed to the normal blocking behaviour of locks):
public class AsyncLocker<T>
{
private LazyDictionary<T, SemaphoreSlim> semaphoreDictionary =
new LazyDictionary<T, SemaphoreSlim>();
public async Task<IDisposable> LockAsync(T key)
{
var semaphore = semaphoreDictionary.GetOrAdd(key, () => new SemaphoreSlim(1,1));
await semaphore.WaitAsync();
return new ActionDisposable(() => semaphore.Release());
}
}
It depends on the following two helper classes:
LazyDictionary:
public class LazyDictionary<TKey,TValue>
{
//here we use Lazy<TValue> as the value in the dictionary
//to guard against the fact the the initializer function
//in ConcurrentDictionary.AddOrGet *can*, under some conditions,
//run more than once per key, with the result of all but one of
//the runs being discarded.
//If this happens, only uninitialized
//Lazy values are discarded. Only the Lazy that actually
//made it into the dictionary is materialized by accessing
//its Value property.
private ConcurrentDictionary<TKey, Lazy<TValue>> dictionary =
new ConcurrentDictionary<TKey, Lazy<TValue>>();
public TValue GetOrAdd(TKey key, Func<TValue> valueGenerator)
{
var lazyValue = dictionary.GetOrAdd(key,
k => new Lazy<TValue>(valueGenerator));
return lazyValue.Value;
}
}
ActionDisposable:
public sealed class ActionDisposable:IDisposable
{
//useful for making arbitrary IDisposable instances
//that perform an Action when Dispose is called
//(after a using block, for instance)
private Action action;
public ActionDisposable(Action action)
{
this.action = action;
}
public void Dispose()
{
var action = this.action;
if(action != null)
{
action();
}
}
}
Now, if you keep a static instance of this somewhere:
static AsyncLocker<string> userLock = new AsyncLocker<string>();
you can use it in an async method, leveraging the delights of LockAsync's IDisposable return type to write a using statement that neatly wraps the critical section:
using(await userLock.LockAsync(userId))
{
//user with userId only allowed in this section
//one at a time.
}
If we need to wait before entering, it's done asynchronously, freeing up the thread to service other requests, instead of blocking until the wait is over and potentially messing up your server's performance under load.
Of course, when you need to scale to more than one webserver, this approach will no longer work, and you'll need to synchronize using a different means (probably via the DB).

Executing part of code exactly 1 time inside Parallel.ForEach

I have to query in my company's CRM Solution(Oracle's Right Now) for our 600k users, and update them there if they exist or create them in case they don't. To know if the user already exists in Right Now, I consume a third party WS. And with 600k users this can be a real pain due to the time it takes each time to get a response(around 1 second). So I managed to change my code to use Parallel.ForEach, querying each record in just 0,35 seconds, and adding it to a List<User> of records to be created or to be updated (Right Now is kinda dumb so I need to separate them in 2 lists and call 2 distinct WS methods).
My code used to run perfectly before multithread, but took too long. The problem is that I can't make a batch too large or I get a timeout when I try to update or create via Web Service. So I'm sending them around 500 records at once, and when it runs the critical code part, it executes many times.
Parallel.ForEach(boDS.USERS.AsEnumerable(), new ParallelOptions { MaxDegreeOfParallelism = -1 }, row =>
{
...
user = null;
user = QueryUserById(row["USER_ID"].Trim());
if (user == null)
{
isUpdate = false;
gObject.ID = new ID();
}
else
{
isUpdate = true;
gObject.ID = user.ID;
}
... fill user attributes as generic fields ...
gObject.GenericFields = listGenericFields.ToArray();
if (isUpdate)
listUserUpdate.Add(gObject);
else
listUserCreate.Add(gObject);
if (i == batchSize - 1 || i == (boDS.USERS.Rows.Count - 1))
{
UpdateProcessingOptions upo = new UpdateProcessingOptions();
CreateProcessingOptions cpo = new CreateProcessingOptions();
upo.SuppressExternalEvents = false;
upo.SuppressRules = false;
cpo.SuppressExternalEvents = false;
cpo.SuppressRules = false;
RNObject[] results = null;
// <Critical_code>
if (listUserCreate.Count > 0)
{
results = _service.Create(_clientInfoHeader, listUserCreate.ToArray(), cpo);
}
if (listUserUpdate.Count > 0)
{
_service.Update(_clientInfoHeader, listUserUpdate.ToArray(), upo);
}
// </Critical_code>
listUserUpdate = new List<RNObject>();
listUserCreate = new List<RNObject>();
}
i++;
});
I thought about using lock or mutex, but it isn't gonna help me, since they will just wait to execute afterwards. I need some solution to execute only ONCE in only ONE thread that part of code. Is it possible? Can anyone share some light?
Thanks and kind regards,
Leandro

As you stated in the comments you're declaring the variables outside of the loop body. That's where your race conditions originate from.
Let's take variable listUserUpdate for example. It's accessed randomly by parallel executing threads. While one thread is still adding to it, e.g. in listUserUpdate.Add(gObject); another thread could already be resetting the lists in listUserUpdate = new List<RNObject>(); or enumerating it in listUserUpdate.ToArray().
You really need to refactor that code to
make each loop run as independent from each other as you can by moving variables inside the loop body and
access data in a synchronizing way using locks and/or concurrent collections

You can use the Double-checked locking pattern. This is usually used for singletons, but you're not making a singleton here so generic singletons like Lazy<T> do not apply.
It works like this:
Separate out your shared data into some sort of class:
class QuerySharedData {
// All the write-once-read-many fields that need to be shared between threads
public QuerySharedData() {
// Compute all the write-once-read-many fields. Or use a static Create method if that's handy.
}
}
In your outer class add the following:
object padlock;
volatile QuerySharedData data
In your thread's callback delegate, do this:
if (data == null)
{
lock (padlock)
{
if (data == null)
{
data = new QuerySharedData(); // this does all the work to initialize the shared fields
}
}
}
var localData = data
Then use the shared query data from localData By grouping the shared query data into a subordinate class you avoid the necessity of making its individual fields volatile.
More about volatile here: Part 4: Advanced Threading.
Update my assumption here is that all the classes and fields held by QuerySharedData are read-only once initialized. If this is not true, for instance if you initialize a list once but add to it in many threads, this pattern will not work for you. You will have to consider using things like Thread-Safe Collections.

Parallelize / Multi-Thread a singleton object method call

Code Details:
// Singleton class CollectionObject
public class CollectionObject
{
private static CollectionObject instance = null;
// GetInstance() is not called from multiple threads
public static CollectionObject GetInstance()
{
if (CollectionObject.instance == null)
CollectionObject.instance = new CollectionObject();
return CollectionObject.instance;
}
// Dictionary object contains Service ID (int) as key and Service object as the value
// Dictionary is filled up during initiation, before the method call ReadServiceMatrix detailed underneath
public Dictionary<int, Service> serviceCollectionDictionary = new Dictionary<int,Service>();
public Service GetServiceByIDFromDictionary(int servID)
{
if (this.serviceCollectionDictionary.ContainsKey(servID))
return this.serviceCollectionDictionary[servID];
else
return null;
}
}
DataTable serviceMatrix = new DataTable();
// Fill serviceMatrix data table from the database
private int ReadServiceMatrix()
{
// Access the Singleton class object
CollectionObject collectionObject = CollectionObject.GetInstance();
// Parallel processing of the data table rows
Parallel.ForEach<DataRow>(serviceMatrix.AsEnumerable(), row =>
{
//Access Service ID from the Data table
string servIDStr = row["ServID"].ToString().Trim();
// Access other column details for each row of the data table
string currLocIDStr = row["CurrLocId"].ToString().Trim();
string CurrLocLoadFlagStr = row["CurrLocLoadFlag"].ToString().Trim();
string nextLocIDStr = row["NextLocId"].ToString().Trim();
string nextLocBreakFlagStr = row["NextLocBreakFlag"].ToString().Trim();
string seqStr = row["Seq"].ToString().Trim();
int servID = Int32.Parse(servIDStr);
int currLocID = Int32.Parse(currLocIDStr);
int nextLocID = Int32.Parse(nextLocIDStr);
bool nextLocBreakFlag = Int32.Parse(nextLocBreakFlagStr) > 0 ? true : false;
bool currLocBreakFlag = Int32.Parse(CurrLocLoadFlagStr) > 0 ? true : false;
int seq = Int32.Parse(seqStr);
// Method call leading to the issue (definition in Collection Object class)
// Fetch service object using the Service ID from the DB
Service service = collectionObject.GetServiceByIDFromDictionary(servID);
// Call a Service class method
service.InitLanes.Add(new Service.LaneNode(currLoc.SequentialID, currLocBreakFlag, nextLoc.SequentialID, nextLocBreakFlag, seq));
}
Issue that happens is:
In the code above for all the Service objects in the dictionary, the subsequent method call is not made, leading to issues in further processing. It has to o with fetching the Service object from the dictionary in parallel mode
The db an dictionary contains all the Ids /Service objects, but my understanding is when processing in Parallel mode for the Singleton class, few of the objects are skipped leading to the issue.
In my understanding the service id passed and service object created is local to a thread, so there shouldn't be an issue that I am facing. This kind of issue is only possible, when for a given method call one thread replace service id value of another thread by its, thus both end up with Service object and few are thus skipped, which is strange in my view until and unless I do not understand the Multi threading in this case correctly
Currently I am able to run the same code in non threaded mode by using the foreach loop instead of Parallel.ForEach / Parallel.Invoke
Please review and let me know your view or any pointer that can help me resolve the issue

In my understanding the service id passed and service object created
is local to a thread
Your understanding is incorrect, if two threads request the same service id the two threads will be both working on the same singular object. If you wanted separate objects you would need to put some kind of new Service() call in GetServiceByIDFromDictionary instead of a dictionary of existing values.
Because multiple threads could be using the same service objects I think your problem lies from the fact that service.InitLanes.Add is likely not thread safe.
The easiest fix is to just lock on that single step
//...SNIP...
Service service = collectionObject.GetServiceByIDFromDictionary(servID);
// Call a Service class method, only let one thread do it for this specific service instance,
// other threads locking on other instances will not block, only other threads using the same instance will block
lock(service)
{
service.InitLanes.Add(new Service.LaneNode(currLoc.SequentialID, currLocBreakFlag, nextLoc.SequentialID, nextLocBreakFlag, seq));
}
}
This assumes that this Parallel.Foreach is the only location collectionObject.GetServiceByIDFromDictionary is used concurrently. If it is not, any other locations that could potentially be calling any methods on returned services must also lock on service.
However if Service is under your control and you can somehow modify service.InitLanes.Add to be thread safe (perhaps change InitLanes out with a thread safe collection from the System.Collections.Concurrent namespace) that would be a better solution than locking.

1.Implementing singleton always think about using of it in mulithreaded way. Always use multithreaded singleton pattern variant, one of them - lazy singleton. Use Lazy singleton using System.Lazy with appropriate LazyThreadSafeMode consturctor argument:
public class LazySingleton3
{
// static holder for instance, need to use lambda to enter code here
//construct since constructor private
private static readonly Lazy<LazySingleton3> _instance
= new Lazy<LazySingleton3>(() => new LazySingleton3(),
LazyThreadSafeMode.PublicationOnly);
// private to prevent direct instantiation.
private LazySingleton3()
{
}
// accessor for instance
public static LazySingleton3 Instance
{
get
{
return _instance.Value;
}
}
}
Read about it here
2.Use lock-ing of your service variable in parallel loop body
// Method call leading to the issue (definition in Collection Object class)
// Fetch service object using the Service ID from the DB
Service service = collectionObject.GetServiceByIDFromDictionary(servID);
lock (service)
{
// Call a Service class method
service.InitLanes.Add(new Service.LaneNode(currLoc.SequentialID,
currLocBreakFlag, nextLoc.SequentialID,
nextLocBreakFlag, seq));
}
3.Consider to use multithreading here. Using lock-ing code make your code not so perfomant as synchronous. So make sure you multithreaded/paralelised code gives you advantages
4.Use appropriate concurrent collections instead of reinventing wheel - System.Collections.Concurrent Namespace

Ideas on logic/algorithm and how to prevent race in threaded writes to SqlServer

I have the following logic:
public void InQueueTable(DataTable Table)
{
int incomingRows = Table.Rows.Count;
if (incomingRows >= RowsThreshold)
{
// asyncWriteRows(Table)
return;
}
if ((RowsInMemory + incomingRows) >= RowsThreshold)
{
// copy and clear internal table
// asyncWriteRows(copyTable)
}
internalTable.Merge(Table);
}
There is one problem with this lagorithm:
Given RowsThreshold = 10000
If incomingRows puts RowsInMemory
over RowsThreshold: (1)
asynchronously write out data, (2)
merge incoming data
If incomingRows is over
RowsThreshold, asynchronously write
incoming data
But what if??? Assume a second thread spins up and calls asyncWriteRows(xxxTable); also, that each thread owning the asynchronous method will be writing to the same table in SqlServer: Does SqlServer handle this sort of multi-threaded write functionality to the same table?
Follow up
Based on Greg D's suggestion:
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connectionString,
sqlBulkCopyOptions.KeepIdentity | SqlBulkCopyOptions.UseInternalTransaction))
{
// perform bulkcopy
}
Regardless, I still have the issue of signaling the asyncWriteRows(copyTable). The algorithm needs to determine the need to go ahead and copy internalTable, clear internalTable, and asyncWriteRows(copyTable). I think that what I need to do is move the internalTable.Copy() call to it's own method:
private DataTable CopyTable (DataTable srcTable)
{
lock (key)
{
return srcTable.Copy();
}
}
...and then the following changes to the InQueue method:
public void InQueueTable(DataTable Table)
{
int incomingRows = Table.Rows.Count;
if (incomingRows >= RowsThreshold)
{
// asyncWriteRows(Table)
return;
}
if ((RowsInMemory + incomingRows) >= RowsThreshold)
{
// copy and clear internal table
// asyncWriteRows(CopyTable(Table))
}
internalTable.Merge(Table);
}
...finally, add a callback method:
private void WriteCallback(Object iaSyncResult)
{
int rowCount = (int)iaSyncResult.AsyncState;
if (RowsInMemory >= rowCount)
{
asyncWriteRows(CopyTable(internalTable));
}
}
This is what I have determined as a solution. Any feedback?

Is there some reason you can't use transactions?

I'll admit now that I'm not an expert in this field.
With transactions and cursors you will get lock escalation if your operation is large. E.g. your operation will start locking a row, then a page then a table if it needs to, preventing other operations from functioning.
The idiot that I was assumed that SQL Server would just queue these blocked operations up and wait for locks to be released, but it just returns errors and it's up to the API programmer to keep retrying (someone correct me if I'm wrong, or if it's fixed in a later version).
If you are happy to be reading possibly old data that you then copy over, like we were, we changed our isolation mode to stop the server blocking operations unnecessarily.
ALTER DATABASE [dbname] SET READ_COMMITTED_SNAPSHOT ON;
You may also alter your insert statments to use NOLOCK. But please read up on this.