I need to parse large text that is similar to XML. Because the text it is not in memory ( I have a StreamReader object) placing that stream on memory is where I take the most time. So on one thread I place that stream into an array (memory). And I have another thread that process that array. But I am having wierd behavieours. For example take a look at this image:
Note that listToProcess[counter] = buffer and right now that should be listToProcess[10] = buffer Note that the debugger says that listToProcess[10]=null why!? . the other thread just reads the items it does not modify them. At first I thought that maybe the other thread was making that item = null but that is not the case. why am I experiencing this behavior?
In case you want to see my code here it is:
Semaphore sem = new Semaphore(0, 1000000);
bool w;
bool done = false;
// this task is responsible for parsing text created by main thread. Main thread
// reads text from the stream and places chunks in listToProces[]
var task1 = Task.Factory.StartNew(() =>
{
sem.WaitOne(); // wait so there are items on list (listToProcess) to work with
// counter to identify which chunk of char[] in listToProcess we are ading to the dictionary
int indexOnList = 0;
while (true)
{
if (listToProcess[indexOnList] == null)
{
if (done)
break;
w = true;
sem.WaitOne();
w = false;
if (done)
break;
if (listToProcess[indexOnList] == null)
{
throw new NotFiniteNumberException();
}
}
// add chunk to dictionary
ProcessChunk(listToProcess[indexOnList]);
indexOnList++;
}
}); // close task1
bool releaseSem = false;
// this main thread is responsible for placing the streamreader into chunks of char[] so that
// task1 can start processing those chunks
int counter = 0;
while (true)
{
char[] buffer = new char[2048];
// unparsedDebugInfo is a streamReader object
var charsRead = unparsedDebugInfo.Read(buffer, 0, buffer.Length);
if (charsRead < 1)
{
listToProcess[counter] = pattern;
break;
}
listToProcess[counter] = buffer;
counter++;
if (releaseSem)
{
sem.Release();
releaseSem = false;
}
if (counter == 10 || w)
{
releaseSem = true;
}
}
done = true;
sem.Release();
task1.Wait();
Edit
Sorry in other words why do I hit this break point:
I thought that counter was the problem but maybe I am doing something wrong with the semaphore...
You have a counter++ so the one you updated before that was at index 9, not index 10.
Meaning : your claim that it set
listToProcess[10] = buffer:
Is incorrect: it set
listToProcess[9] = buffer:
Related
I read data from the serial port and parse it in a separate class. However data is incorrectly parsed and some samples are repeated while others are missing.
Here is an example of the parsed packet. It starts with the packetIndex (shoudl start from 1 and incrementing). You can see how the packetIdx repeats and some of the other values repeat as well. I think that's due to multithreading but I'm not sure how to fix it.
2 -124558.985180734 -67934.4168823262 -164223.049786454 -163322.386243628
2 -124619.580759952 -67962.535376851 -164191.757344217 -163305.68949052
3 -124685.719571795 -67995.8394760894 -164191.042088394 -163303.119039907
5 -124801.747477263 -68045.7062179692 -164195.288919841 -163299.140429394
6 -124801.747477263 -68045.7062179692 -164221.105184687 -163297.46404856
6 -124832.8387538 -68041.9287731563 -164214.936103217 -163294.983004926
This is what I should receive:
1 -124558.985180734 -67934.4168823262 -164223.049786454 -163322.386243628
2 -124619.580759952 -67962.535376851 -164191.757344217 -163305.68949052
3 -124685.719571795 -67995.8394760894 -164191.042088394 -163303.119039907
4 -124801.747477263 -68045.7062179692 -164195.288919841 -163299.140429394
...
This is the SerialPort_DataReceived
public void serialPort1_DataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
lock (_lock)
{
byte[] buffer = new byte[_serialPort1.BytesToRead];
_serialPort1.Read(buffer, 0, buffer.Length);
for (int i = 0; i < buffer.Length; i++)
{
//Parse data
double[] samplesAtTimeT = DataParserObj.interpretBinaryStream(buffer[i]);
//Add data to BlockingCollection when parsed
if (samplesAtTimeT != null)
_bqBufferTimerSeriesData.Add(samplesAtTimeT);
}
}
}
And the class that parses the data:
public class DataParser
{
private int packetSampleCounter = 0;
private int localByteCounter = 0;
private int packetState = 0;
private byte[] tmpBuffer = new byte[3];
private double[] ParsedData = new double[5]; //[0] packetIdx (0-255), [1-4] signal
public double[] interpretBinaryStream(byte actbyte)
{
bool returnDataFlag = false;
switch (packetState)
{
case 0: // end packet indicator
if (actbyte == 0xC0)
packetState++;
break;
case 1: // start packet indicator
if (actbyte == 0xA0)
packetState++;
else
packetState = 0;
break;
case 2: // packet Index
packetSampleCounter = 0;
ParsedData[packetSampleCounter] = actbyte;
packetSampleCounter++;
localByteCounter = 0;
packetState++;
break;
case 3: //channel data (4 channels x 3byte/channel)
// 3 bytes
tmpBuffer[localByteCounter] = actbyte;
localByteCounter++;
if (localByteCounter == 3)
{
ParsedData[packetSampleCounter] = Bit24ToInt32(tmpBuffer);
if (packetSampleCounter == 5)
packetState++; //move to next state, end of packet
else
localByteCounter = 0;
}
break;
case 4: // end packet
if (actbyte == 0xC0)
{
returnDataFlag = true;
packetState = 1;
}
else
packetState = 0;
break;
default:
packetState = 0;
break;
}
if (returnDataFlag)
return ParsedData;
else
return null;
}
}
Get rid of the DataReceived event and instead use await serialPort.BaseStream.ReadAsync(....) to get notified when data comes in. async/await is much cleaner and doesn't force you into multithreaded data processing. For high speed networking, parallel processing is great. But serial ports are slow, so extra threads have no benefit.
Also, BytesToRead is buggy (it does return the number of queued bytes, but it destroys other state) and you should never call it.
Finally, do NOT ignore the return value from Read (or BaseStream.ReadAsync). You need to know how bytes were actually placed into your buffer, because it is not guaranteed to be the same number you asked for.
private async void ReadTheSerialData()
{
var buffer = new byte[200];
while (serialPort.IsOpen) {
var valid = await serialPort.BaseStream.ReadAsync(buffer, 0, buffer.Length);
for (int i = 0; i < valid; ++i)
{
//Parse data
double[] samplesAtTimeT = DataParserObj.interpretBinaryStream(buffer[i]);
//Add data to BlockingCollection when parsed
if (samplesAtTimeT != null)
_bqBufferTimerSeriesData.Add(samplesAtTimeT);
}
}
}
Just call this function after opening the port and setting your flow control, timeouts, etc. You may find that you no longer need the blocking queue, but can just handle the contents of samplesAtTimeT directly.
I am calling a VB 6.0 dll in Parallel.ForEach and expecting all calls to be started simultaneously or at least 2 of them based on my PC's cores or threads availability in thread pool
VB6 dll
Public Function DoJunk(ByVal counter As Long, ByVal data As String) As Integer
Dim i As Long
Dim j As Long
Dim s As String
Dim fno As Integer
fno = FreeFile
Open "E:\JunkVB6Dll\" & data & ".txt" For Output Access Write As #fno
Print #fno, "Starting loop with counter = " & counter
For i = 0 To counter
Print #fno, "counting " & i
Next
Close #fno
DoJunk = 1
End Function
counter is being passed from the caller to control execution time of the call and file is being written to make it an IO based process.
C# caller
private void ReportProgress(int value)
{
progressBar.Value = value;
//progressBar.Value++;
}
private void button1_Click(object sender, EventArgs e)
{
progressBar.Value = 0;
counter = 0;
Stopwatch watch = new Stopwatch();
watch.Start();
//var range = Enumerable.Range(0, 100);
var range = Enumerable.Range(0, 20);
bool finished = false;
Task.Factory.StartNew(() =>
{
Parallel.ForEach(range, i =>
{
#region COM CALL
JunkProject.JunkClass junk = new JunkProject.JunkClass();
try
{
Random rnd = new Random();
int dice = rnd.Next(10, 40);
int val = 0;
if (i == 2)
val = junk.DoJunk(9000000, i.ToString());
else
val = junk.DoJunk(dice * 10000, i.ToString());
System.Diagnostics.Debug.Print(junk.GetHashCode().ToString());
if (val == 1)
{
Interlocked.Increment(ref counter);
progressBar.Invoke((Action)delegate { ReportProgress(counter); });
}
junk = null;
}
catch (Exception excep)
{
i = i;
}
finally { junk = null; }
#endregion
});
}).ContinueWith(t =>
{
watch.Stop();
MessageBox.Show(watch.ElapsedMilliseconds.ToString());
});
}
This line is making a specific call longer than the others.
val = junk.DoJunk(9000000, i.ToString());
Here this second process is causing all calls inside the Parallel.ForEach to stop i.e. no other file is created unless this 2nd call gets completed.
Is it an expected behavior or i am doing something wrong?
As #John Wu suggested that you can create AppDomain to allow COM to run on different App Domain, I believe you could run your parallel like this.
Parallel.ForEach(range, i =>
{
AppDomain otherDomain = AppDomain.CreateDomain(i.ToString());
otherDomain.DoCallBack(delegate
{
//Your COM call
});
});
EDIT
Right.. I am not sure how can you set serializable on VB6.0 class. You can try the other way (Marshaling objects by reference). Noted: I haven't actually tested this, but I would like to know if that will work.
Parallel.ForEach(range, i =>
{
AppDomain otherDomain = AppDomain.CreateDomain(i.ToString());
var comCall = (ComCall) otherDomain.CreateInstanceFromAndUnwrap(Assembly.GetExecutingAssembly().Location, typeof(ComCall).ToString());
comCall.Run();
AppDomain.Unload(otherDomain);
});
and the class
public class ComCall : MarshalByRefObject
{
public void Run()
{
//Your COM Call
}
}
Here is also additional reference regarding the topic.
https://www.codeproject.com/Articles/14791/NET-Remoting-with-an-easy-example
I have a bunch of incoming web requests which I need to serialize and process one after each other. I cannot process more than one at a time. My current solution is to run a tool which can be accessed by a Web API using a Memory Mapped File acting as shared memory. I also need a Mutex to allow exclusive access to the shared memory. An event will be used to signal that a task was added.
So, basically we have there a multiple producers and one consumer. What follows is my first solution. Could someone tell whether or not there are some sort of race conditions or any other problems:
MemoryMappedFile MMF = MemoryMappedFile.CreateNew("Task_Queue", 5000);
MemoryMappedViewAccessor MMF_Accessor = MMF.CreateViewAccessor();
bool Mutex_Created = false;
Mutex Mutex = new System.Threading.Mutex(false, "Mutex", out Mutex_Created);
if(Mutex_Created==false)
{
// bad error
return;
}
EventWaitHandle EWH = new EventWaitHandle(false, EventResetMode.ManualReset);
Random Rand = new Random();
// Consumer
// work on the Tasks
Task.Factory.StartNew(() =>
{
while (true)
{
// wait until a task has been added
EWH.WaitOne();
// get exclusive access to read in task
Mutex.WaitOne();
byte[] Buffer = ASCIIEncoding.ASCII.GetBytes(new string(' ', 5000));
MMF_Accessor.ReadArray(0, Buffer, 0, Buffer.Length);
int Position = 0;
foreach (var b in Buffer) { if (b == 0) break; Position++; }
string Message = string.Empty;
for (int i = 0; i < Position; ++i)
{
Message += (char)Buffer[i];
}
if(string.IsNullOrEmpty(Message) == false)
{
Console.WriteLine(Message);
}
Mutex.ReleaseMutex();
EWH.Reset();
}
});
// Producer
// via a web request
Task.Factory.StartNew(() =>
{
while (true)
{
if(EWH.WaitOne(0))
{
// consumer must take in the task first
// wait a bit and then try again
Thread.Sleep(100);
break;
}
// wait until we access Shared Memory and claim it when we can
Mutex.WaitOne();
string Request = "Task 1 ";
byte[] Buffer = ASCIIEncoding.ASCII.GetBytes(Request);
Buffer[Buffer.Length - 1] = 0;
MMF_Accessor.WriteArray(0, Buffer, 0, Buffer.Length);
// Signal that a tasks has been added to shared memory
EWH.Set();
// release the mutex so that others can use the shared memory
Mutex.ReleaseMutex();
Thread.Sleep(Rand.Next(10, 1000));
}
});
// Producer
// via a web request
Task.Factory.StartNew(() =>
{
while (true)
{
// wait until we access Shared Memory and claim it when we can
Mutex.WaitOne();
string Request = "Task 2 ";
byte[] Buffer = ASCIIEncoding.ASCII.GetBytes(Request);
Buffer[Buffer.Length - 1] = 0;
MMF_Accessor.WriteArray(0, Buffer, 0, Buffer.Length);
// Signal that a tasks has been added to shared memory
EWH.Set();
// release the mutex so that others can use the shared memory
Mutex.ReleaseMutex();
Random r = new Random();
Thread.Sleep(Rand.Next(10, 1000));
}
});
while (Console.Read() != 'q') ;
Please note the posted code is just for demonstration purposes.
I am trying to dynamically create X amount of threads(specified by user) then basically have all of them execute some code at the exact same time in intervals of 1 second.
The issue I am having is that the task I am trying to complete relies on a loop to determine if the current IP is equal to the last. (It scans hosts) So since I have this loop inside, it is going off and then the other threads are not getting created, and not executing the code. I would like them to all go off at the same time, wait 1 second(using a timer or something else that doesnt lock the thread since the code it is executing has a timeout it waits for.) Can anyone help me out? Here is my current code:
int threads = Convert.ToInt32(txtThreads.Text);
List<Thread> workerThreads = new List<Thread>();
string from = txtStart.Text, to = txtEnd.Text;
uint current = from.ToUInt(), last = to.ToUInt();
ulong total = last - current;
for (int i = 0; i < threads; i++)
{
Thread thread = new Thread(() =>
{
for (int t = 0; t < Convert.ToInt32(total); t += i)
{
while (current <= last)
{
current = Convert.ToUInt32(current + t);
var ip = current.ToIPAddress();
doSomething(ip);
}
}
});
workerThreads.Add(thread);
thread.Start();
}
Don't use a lambda as the body of your thread, otherwise the i value isn't doing what you think it's doing. Instead pass the value into a method.
As for starting all of the threads at the same time do something like the following:
private object syncObj = new object();
void ThreadBody(object boxed)
{
Params params = (Params)boxed;
lock (syncObj)
{
Monitor.Wait(syncObj);
}
// do work here
}
struct Params
{
// passed values here
}
void InitializeThreads()
{
int threads = Convert.ToInt32(txtThreads.Text);
List<Thread> workerThreads = new List<Thread>();
string from = txtStart.Text, to = txtEnd.Text;
uint current = from.ToUInt(), last = to.ToUInt();
ulong total = last - current;
for (int i = 0; i < threads; i++)
{
Thread thread = new Thread(new ParameterizedThreadStart(this.ThreadBody, new Params { /* initialize values here */ }));
workerThreads.Add(thread);
thread.Start();
}
lock(syncObj)
{
Monitor.PulseAll(syncObj);
}
}
You're running into closure problems. There's another question that somewhat addresses this, here.
Basically you need to capture the value of i as you create each task. What's happening is by the time the task gets around to actually running, the value of i across all your tasks is the same -- the value at the end of the loop.
I used 4 threads to write data to a file. Each thread uses MMF to write a part of a file.
When I open the file with Notpad++ which always displays null byte between part of the file.
Thread 1 writes 1
Thread 2 writes 12
Thread 3 writes 123
Thread 4 writes 1234
The result will be like this: 1Nul12Nul123Nul1234Nul
MMF requires 2 bytes to write "1" character, but actually "1" character is only one byte.
If I set 1 byte to MMF it will throw: There are not enough bytes remaining in the accessor to write at this position.
How to remove these null bytes ?
Update code:
I use only 2 threads for easier look.
static void Main(string[] args)
{
File.Delete(#"C:\abc.txt");
CancellationTokenSource tokenSource = new CancellationTokenSource();
CancellationToken cancellationToken = tokenSource.Token;
// Write data to file using MMF.
Task<int>[] tasks = new Task<int>[2];
tasks[0] = new Task<int>(() => F1(0, 2, "1", cancellationToken), cancellationToken);
tasks[1] = new Task<int>(() => F2(2,4, "12", cancellationToken), cancellationToken);
// Create MMF File.
MemoryMappedFileSecurity CustomSecurity = new MemoryMappedFileSecurity();
MemoryMappedFile MemoryMapped = MemoryMappedFile.CreateFromFile(
new FileStream(#"C:\abc.txt", FileMode.OpenOrCreate),
"MapAbcFile",// Name
20, // Size
MemoryMappedFileAccess.ReadWrite, // Access type
null, // You can customize the security
HandleInheritability.None
, false); // Inherit to child process
try
{
ParallelOptions op = new ParallelOptions();
op.CancellationToken = tokenSource.Token;
Parallel.ForEach(tasks, op, t =>
{
try
{
t.Start();
}
catch (Exception)
{
tokenSource.Cancel();
}
});
// Wait for all the tasks to finish.
Task.WaitAll(tasks, cancellationToken);
Console.WriteLine("Done!");
}
catch (OperationCanceledException)
{
Console.WriteLine("Cancelled!");
}
catch (AggregateException ae)
{
//tokenSource.Cancel();
Console.WriteLine("AggregateException!");
// Assume we know what's going on with this particular exception.
// Rethrow anything else. AggregateException.Handle provides
// another way to express this. See later example.
foreach (var ie in ae.InnerExceptions)
{
Console.WriteLine(ie.Message);
}
}
MemoryMapped.Dispose();
}
private static int F1(long offset, long size, string firstName, CancellationToken cancellationToken)
{
Console.WriteLine(string.Format("ManagedThreadId(F1): {0} - {1}", System.Threading.Thread.CurrentThread.ManagedThreadId, DateTime.Now.ToLongTimeString()));
MemoryMappedFile MemoryMapped = MemoryMappedFile.OpenExisting("MapAbcFile");
MemoryMappedViewAccessor WriteFirstPart = MemoryMapped.CreateViewAccessor(offset, size, MemoryMappedFileAccess.ReadWrite);
int i = 0;
foreach (char item in firstName.ToCharArray())
{
WriteFirstPart.Write(i, item);
i++;
}
WriteFirstPart.Dispose();
return 1;
}
private static int F2(long offset, long size, string lastName, CancellationToken cancellationToken)
{
Console.WriteLine(string.Format("ManagedThreadId(F2): {0} - {1}", System.Threading.Thread.CurrentThread.ManagedThreadId, DateTime.Now.ToLongTimeString()));
MemoryMappedFile MemoryMapped = MemoryMappedFile.OpenExisting("MapAbcFile");
MemoryMappedViewAccessor WriteFirstPart = MemoryMapped.CreateViewAccessor(offset, size, MemoryMappedFileAccess.ReadWrite);
int i = 0;
foreach (char item in lastName.ToCharArray())
{
WriteFirstPart.Write(i, item);
i++;
}
WriteFirstPart.Dispose();
return 1;
}
Your mistake is here:
but actually "1" character is only one byte.
Whereas, a Char is actually defined as:
Represents a character as a UTF-16 code unit.
UTF-16 code units take up two bytes.
If you want a 1 to only take up one byte, then you need to use something from the System.Text.Encoding class to use say ASCII or UTF-8, or some other encoding where the character 1 takes up one byte. E.g.
foreach (byte item in System.Text.Encoding.UTF8.GetBytes(firstName))
{
WriteFirstPart.Write(i, item);
i++;
}