We Keep Coding

Serializing tasks using a Memory Mapped File

I have a bunch of incoming web requests which I need to serialize and process one after each other. I cannot process more than one at a time. My current solution is to run a tool which can be accessed by a Web API using a Memory Mapped File acting as shared memory. I also need a Mutex to allow exclusive access to the shared memory. An event will be used to signal that a task was added.
So, basically we have there a multiple producers and one consumer. What follows is my first solution. Could someone tell whether or not there are some sort of race conditions or any other problems:
MemoryMappedFile MMF = MemoryMappedFile.CreateNew("Task_Queue", 5000);
MemoryMappedViewAccessor MMF_Accessor = MMF.CreateViewAccessor();
bool Mutex_Created = false;
Mutex Mutex = new System.Threading.Mutex(false, "Mutex", out Mutex_Created);
if(Mutex_Created==false)
{
// bad error
return;
}
EventWaitHandle EWH = new EventWaitHandle(false, EventResetMode.ManualReset);
Random Rand = new Random();
// Consumer
// work on the Tasks
Task.Factory.StartNew(() =>
{
while (true)
{
// wait until a task has been added
EWH.WaitOne();
// get exclusive access to read in task
Mutex.WaitOne();
byte[] Buffer = ASCIIEncoding.ASCII.GetBytes(new string(' ', 5000));
MMF_Accessor.ReadArray(0, Buffer, 0, Buffer.Length);
int Position = 0;
foreach (var b in Buffer) { if (b == 0) break; Position++; }
string Message = string.Empty;
for (int i = 0; i < Position; ++i)
{
Message += (char)Buffer[i];
}
if(string.IsNullOrEmpty(Message) == false)
{
Console.WriteLine(Message);
}
Mutex.ReleaseMutex();
EWH.Reset();
}
});
// Producer
// via a web request
Task.Factory.StartNew(() =>
{
while (true)
{
if(EWH.WaitOne(0))
{
// consumer must take in the task first
// wait a bit and then try again
Thread.Sleep(100);
break;
}
// wait until we access Shared Memory and claim it when we can
Mutex.WaitOne();
string Request = "Task 1 ";
byte[] Buffer = ASCIIEncoding.ASCII.GetBytes(Request);
Buffer[Buffer.Length - 1] = 0;
MMF_Accessor.WriteArray(0, Buffer, 0, Buffer.Length);
// Signal that a tasks has been added to shared memory
EWH.Set();
// release the mutex so that others can use the shared memory
Mutex.ReleaseMutex();
Thread.Sleep(Rand.Next(10, 1000));
}
});
// Producer
// via a web request
Task.Factory.StartNew(() =>
{
while (true)
{
// wait until we access Shared Memory and claim it when we can
Mutex.WaitOne();
string Request = "Task 2 ";
byte[] Buffer = ASCIIEncoding.ASCII.GetBytes(Request);
Buffer[Buffer.Length - 1] = 0;
MMF_Accessor.WriteArray(0, Buffer, 0, Buffer.Length);
// Signal that a tasks has been added to shared memory
EWH.Set();
// release the mutex so that others can use the shared memory
Mutex.ReleaseMutex();
Random r = new Random();
Thread.Sleep(Rand.Next(10, 1000));
}
});
while (Console.Read() != 'q') ;
Please note the posted code is just for demonstration purposes.

Calculate pending time to finish processing threaded code C#

I have a code with threads and I want to show the pending time to finish processing. The button1 calls the
function "Function1()" that reads a file in chunks of 1024 bytes controlled in a while loop until get end
of file. Within the "While loop" there is a "foreach loop" where is called the "Function2()". I'm starting
the timer at the beginning of "while loop" and stopping it at the end of "while loop". After that I'm trying
to calculate aprox the Pending time knowing first the number of iterations that will be processed by "while loop".
Then I save the "elapsed time for the first iteration" (lets say T1) and then I multiply it by number of iterations.
This would be
PendingTime = T1*Iterations.
Then I do
PendingTime = PendingTime - Ti, where Ti is the ElapsedTime of the ith iteration.
The issue is when I try with the real code, the multiplation of T1*Iterations gives me 402s and actually
the processing takes 12s.
Maybe some expert could see what I'm doing wrong. Thanks in advance.
The code looks like this:
async void button1_Click(object sender, EventArgs e)
{
//Some code
await Task.Run(() => Function1(inputfile, cts.Token), cts.Token);
//Some code
}
public void Function2()
{
//Some code
}
public void Function1(string inputfile, CancellationToken token)
{
int buffer = 1024;
int IterationCounter = 0;
decimal Iterations = 1;
int PendingTime = 0;
using (BinaryReader reader = new BinaryReader(File.Open(inputfile, FileMode.Open)))
{
FileLength = (int)reader.BaseStream.Length;
Iterations = (int)FileLength/buffer;
while (chunk.Length > 0)
{
Stopwatch sw1 = Stopwatch.StartNew(); //Start time counter
//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
//some code
chunk = reader.ReadBytes(buffer);
foreach (byte data in chunk)
{
//Some code
Function2(); //Call to Function2
}
//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
//Checking if it is the first iteration to save the pending time
//Pending time would be the elapsed time for the first iteration
//multiplied by the number of iterations (FileLength/1024).
sw1.Stop(); //Stop time counter
if (IterationCounter == 1)
{
PendingTime = (int)((decimal)Math.Round(sw1.Elapsed.TotalMilliseconds / 1000, 4)*Iterations);
}
//Show in TexBox1 the pending time
TextBox1.Invoke((MethodInvoker)delegate
{
PendingTime = PendingTime - (int)Math.Round(sw1.Elapsed.TotalMilliseconds / 1000, 4);
TextBox1.Text = PendingTime + " s";
});
}
}
}
Update:
I'm testing with the following code based on the example of Peter Duniho.
It can be tested with any file(i.e. txt file). I've tested with a txt file of 5MB and execution time was 3 seconds, but the pending time appear always as zero in TextBox1. Where I'm wrong?
Note: I changed this:
double timePerIteration = sw1.Elapsed / ++IterationCounter;
to this
double timePerIteration = sw1.ElapsedMilliseconds/1000/ ++IterationCounter;
Since I was getting the error:
Operator '/' cannot be applied to operands of type 'System.TimeSpan' and 'int' (CS0019)
The code so far is. Thanks for help.
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Drawing;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using System.Windows.Forms;
namespace TestTimer
{
public partial class MainForm : Form
{
CancellationTokenSource cts = new CancellationTokenSource();
string filename = "";
long FileLength;
FileInfo fInfo;
Stopwatch sw1 = new Stopwatch();
public MainForm()
{
InitializeComponent();
}
void BtnSelectFileClick(object sender, EventArgs e)
{
OpenFileDialog ofd = new OpenFileDialog();
ofd.Title = "Select file";
DialogResult dr = ofd.ShowDialog();
if (dr == DialogResult.OK)
{
filename = ofd.FileName;
fInfo = new FileInfo(filename);
}
else
{
MessageBox.Show("File not found");
return;
}
}
async void BtnRunProcessClick(object sender, System.EventArgs e)
{
cts = new CancellationTokenSource();
await Task.Run(() => Function1(filename, cts.Token), cts.Token);
}
public void Function1(string inputfile, CancellationToken token)
{
int buffer = 1024;
int IterationCounter = 0;
int Iterations = 0;
double pendingTime = 0;
using (BinaryReader reader = new BinaryReader(File.Open(inputfile, FileMode.Open)))
{
FileLength = (int)reader.BaseStream.Length;
Iterations = (int)FileLength/buffer;
byte[] chunk;
sw1 = Stopwatch.StartNew(); //Start time counter
while (true)
{
chunk = reader.ReadBytes(buffer);
if (chunk.Length == 0) {break;}
foreach (byte data in chunk)
{
Thread.Sleep(90/100);
}
// pendingTime is the current average time-per-iteration,
// times the number of iterations left
double timePerIteration = sw1.ElapsedMilliseconds/1000/ ++IterationCounter;
pendingTime = timePerIteration * (Iterations - IterationCounter);
TextBox1.Invoke((MethodInvoker)delegate
{
// Let string.Format() take care of rounding for you
TextBox1.Text = string.Format("{0:0} s", pendingTime / 1000);
});
}
MessageBox.Show("Execution time: " + string.Format("{0:0} s", sw1.ElapsedMilliseconds / 1000) );
}
}
}
}

I don't see how the code you posted ever actually compiled, never mind worked. The FileLength variable does not appear to be declared, and you never increment the IterationCounter variable, giving you a negative PendingTime value with each iteration. Even if you had incremented the counter, your PendingTime variable's actual meaning changes from the block that executes when the counter is 1 and a little later when you subtract your elapsed time from the current PendingTime variable.
That suggests the code you posted isn't really the code you're using, since the displayed time remaining would always have been negative (even assuming the declaration of FileLength just got accidently dropped from your post for some reason). For the sake of argument, I'll add a statement that does the increment…
As commenter Chris says, when each iteration's actual duration can vary, as it seems to be the case here, the best you're going to do is average all of the iterations up to the current one. Even that may lead to an erroneous time-remaining display, with a fair amount of variation from iteration to iteration (especially if the number of iterations is small), but at least it's more likely to be close.
Something like this would likely work better for you:
public void Function1(string inputfile, CancellationToken token)
{
int buffer = 1024;
int IterationCounter = 0;
int Iterations;
using (BinaryReader reader = new BinaryReader(File.Open(inputfile, FileMode.Open)))
{
if (reader.BaseStream.Length == 0)
{
// nothing to do
return;
}
// NOTE: this won't work for files with length > int.MaxValue!
// Your original code has the same limitation, and I have not
// bothered to change that.
// Now that we know for sure the length is > 0, we can
// do the following to ensure a correct iteration count
Iterations = ((int)reader.BaseStream.Length - 1) / buffer + 1;
Stopwatch sw1 = Stopwatch.StartNew();
while (chunk.Length > 0)
{
//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
//some code
chunk = reader.ReadBytes(buffer);
foreach (byte data in chunk)
{
//Some code
Function2(); //Call to Function2
}
//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
// pendingTime is the current average time-per-iteration,
// times the number of iterations left
double timePerIteration = sw1.ElapsedMilliseconds / ++IterationCounter,
pendingTime = timePerIteration *
(Iterations - IterationCounter);
//Show in TexBox1 the pending time
TextBox1.Invoke((MethodInvoker)delegate
{
// Let string.Format() take care of rounding for you
TextBox1.Text = string.Format("{0:0} s", pendingTime / 1000);
});
}
}
}
Unless you are guaranteed that your input file is always exactly a multiple of 1024 bytes in length, you also had a bug in your calculation of the total iteration count. I fixed that in the above as well.

ASP.net C# : How to read 20 to 200 GB file line by line using File.ReadLines(fileName).GetEnumerator()?

We are trying with below code.
public static int SplitFile(string fileName, string tmpFolder, List<string> queue, int splitSize = 100000)
{
int chunk = 0;
if (!Directory.Exists(tmpFolder))
Directory.CreateDirectory(tmpFolder);
using (var lineIterator = File.ReadLines(fileName).GetEnumerator())
{
bool stillGoing = true;
for (chunk = 0; stillGoing; chunk++)
{
stillGoing = WriteChunk(lineIterator, splitSize, chunk, tmpFolder, queue);
}
}
return chunk;
}
private static bool WriteChunk(IEnumerator<string> lineIterator,
int splitSize, int chunk, string tmpFolder, List<string> queue)
{
try
{
//int tmpChunkSize = 1000;
//int tmpChunkInc = 0;
string splitFile = Path.Combine(tmpFolder, "file" + chunk + ".txt");
using (var writer = File.CreateText(splitFile))
{
queue.Add(splitFile);
for (int i = 0; i < splitSize; i++)
{
if (!lineIterator.MoveNext())
{
return false;
}
writer.WriteLine(lineIterator.Current);
}
}
return true;
}
catch (Exception)
{
throw;
}
}
It creates around 36 text files (around 800 MB), but starting throwing "Out of memory exception" in creation of 37th File at lineIterator.MoveNext().
While lineIterator.Current shows the value in debugger.

As It s a huge file you should read it Seek and ReadBytes methods of BinaryReader.
You can see a simple example here. After you use the ReadBytes check for the last new lines and write the process file in certain amount of lines you read. Don t write every line you read and also don t keep all the data in the memory.
The rest is in your hands.
Maybe it is realted to that one When does File.ReadLines free resources
IEnumerable doesn't inherit from IDisposable because typically, the class that implements it only gives you the promise of being enumerable, it hasn't actually done anything yet that warrants disposal.

Why array behaves different when different thread access it

I need to parse large text that is similar to XML. Because the text it is not in memory ( I have a StreamReader object) placing that stream on memory is where I take the most time. So on one thread I place that stream into an array (memory). And I have another thread that process that array. But I am having wierd behavieours. For example take a look at this image:
Note that listToProcess[counter] = buffer and right now that should be listToProcess[10] = buffer Note that the debugger says that listToProcess[10]=null why!? . the other thread just reads the items it does not modify them. At first I thought that maybe the other thread was making that item = null but that is not the case. why am I experiencing this behavior?
In case you want to see my code here it is:
Semaphore sem = new Semaphore(0, 1000000);
bool w;
bool done = false;
// this task is responsible for parsing text created by main thread. Main thread
// reads text from the stream and places chunks in listToProces[]
var task1 = Task.Factory.StartNew(() =>
{
sem.WaitOne(); // wait so there are items on list (listToProcess) to work with
// counter to identify which chunk of char[] in listToProcess we are ading to the dictionary
int indexOnList = 0;
while (true)
{
if (listToProcess[indexOnList] == null)
{
if (done)
break;
w = true;
sem.WaitOne();
w = false;
if (done)
break;
if (listToProcess[indexOnList] == null)
{
throw new NotFiniteNumberException();
}
}
// add chunk to dictionary
ProcessChunk(listToProcess[indexOnList]);
indexOnList++;
}
}); // close task1
bool releaseSem = false;
// this main thread is responsible for placing the streamreader into chunks of char[] so that
// task1 can start processing those chunks
int counter = 0;
while (true)
{
char[] buffer = new char[2048];
// unparsedDebugInfo is a streamReader object
var charsRead = unparsedDebugInfo.Read(buffer, 0, buffer.Length);
if (charsRead < 1)
{
listToProcess[counter] = pattern;
break;
}
listToProcess[counter] = buffer;
counter++;
if (releaseSem)
{
sem.Release();
releaseSem = false;
}
if (counter == 10 || w)
{
releaseSem = true;
}
}
done = true;
sem.Release();
task1.Wait();
Edit
Sorry in other words why do I hit this break point:
I thought that counter was the problem but maybe I am doing something wrong with the semaphore...

You have a counter++ so the one you updated before that was at index 9, not index 10.
Meaning : your claim that it set
listToProcess[10] = buffer:
Is incorrect: it set
listToProcess[9] = buffer:

C# Async Ping: How to avoid an out of memory exception?

Question: I want to search the subnet for all computers in it.
So I send a ping to all IP addresses in the subnet.
The problem is it works fine if I only scan 192.168.0.".
But if I scan 192.168..*", then I get an "Out of memory" exception.
Why ? Do I have to limit the threads, or is the problem the memory consumed by new ping which doesn't get destructed once finished, or do I need to call gc.collect() ?
static void Main(string[] args)
{
string strFromIP = "192.168.0.1";
string strToIP = "192.168.255.255";
Oyster.Math.IntX omiFromIP = 0;
Oyster.Math.IntX omiToIP = 0;
IsValidIP(strFromIP, ref omiFromIP);
IsValidIP(strToIP, ref omiToIP);
for (Oyster.Math.IntX omiThisIP = omiFromIP; omiThisIP <= omiToIP; ++omiThisIP)
{
Console.WriteLine(IPn2IPv4(omiThisIP));
System.Net.IPAddress sniIPaddress = System.Net.IPAddress.Parse(IPn2IPv4(omiThisIP));
SendPingAsync(sniIPaddress);
}
Console.WriteLine(" --- Press any key to continue --- ");
Console.ReadKey();
} // Main
// http://pberblog.com/post/2009/07/21/Multithreaded-ping-sweeping-in-VBnet.aspx
// http://www.cyberciti.biz/faq/how-can-ipv6-address-used-with-webbrowser/#comments
// http://www.kloth.net/services/iplocate.php
// http://bytes.com/topic/php/answers/829679-convert-ipv4-ipv6
// http://stackoverflow.com/questions/1434342/ping-class-sendasync-help
public static void SendPingAsync(System.Net.IPAddress sniIPaddress)
{
int iTimeout = 5000;
System.Net.NetworkInformation.Ping myPing = new System.Net.NetworkInformation.Ping();
System.Net.NetworkInformation.PingOptions parmPing = new System.Net.NetworkInformation.PingOptions();
System.Threading.AutoResetEvent waiter = new System.Threading.AutoResetEvent(false);
myPing.PingCompleted += new System.Net.NetworkInformation.PingCompletedEventHandler(AsyncPingCompleted);
string data = "ABC";
byte[] dataBuffer = Encoding.ASCII.GetBytes(data);
parmPing.DontFragment = true;
parmPing.Ttl = 32;
myPing.SendAsync(sniIPaddress, iTimeout, dataBuffer, parmPing, waiter);
//waiter.WaitOne();
}
private static void AsyncPingCompleted(Object sender, System.Net.NetworkInformation.PingCompletedEventArgs e)
{
System.Net.NetworkInformation.PingReply reply = e.Reply;
((System.Threading.AutoResetEvent)e.UserState).Set();
if (reply.Status == System.Net.NetworkInformation.IPStatus.Success)
{
Console.WriteLine("Address: {0}", reply.Address.ToString());
Console.WriteLine("Roundtrip time: {0}", reply.RoundtripTime);
}
}

According to this thread, System.Net.NetworkInformation.Ping seems to allocate one thread per async request, and "ping-sweeping a class-B network creates 100's of threads and eventually results in an out-of-memory error."
The workaround that person used was to write their own implementation using raw sockets. You don't have to do that in F#, of course, but there are a number of advantages in doing so.

First: Only start like 1000 pings the first time (in the loop in Main)
Second: Move the following parameters to Program class (member variables)
Oyster.Math.IntX omiFromIP = 0;
Oyster.Math.IntX omiToIP = 0;
Oyster.Math.IntX omiCurrentIp = 0;
object syncLock = new object();
Third: In AsyncPingCompleted do something like this in the bottom:
public void AsyncPingCompleted (bla bla bla)
{
//[..other code..]
lock (syncLock)
{
if (omiToIP < omiCurrentIp)
{
++omiCurrentIp;
System.Net.IPAddress sniIPaddress = System.Net.IPAddress.Parse(IPn2IPv4(omiCurrentIp));
SendPingAsync(sniIPaddress);
}
}
}
Update with complete code example
public class Example
{
// Number of pings that can be pending at the same time
private const int InitalRequests = 10000;
// variables from your Main method
private Oyster.Math.IntX _omiFromIP = 0;
private Oyster.Math.IntX _omiToIP = 0;
private Oyster.Math.IntX _omiCurrentIp = 0;
// synchronoize so that two threads
// cannot ping the same IP.
private object _syncLock = new object();
static void Main(string[] args)
{
string strFromIP = "192.168.0.1";
string strToIP = "192.168.255.255";
IsValidIP(strFromIP, ref _omiFromIP);
IsValidIP(strToIP, ref _omiToIP);
for (_omiCurrentIp = _omiFromIP; _omiCurrentIp <= _omiFromIP + InitalRequests; ++_omiCurrentIp)
{
Console.WriteLine(IPn2IPv4(_omiCurrentIp));
System.Net.IPAddress sniIPaddress = System.Net.IPAddress.Parse(IPn2IPv4(_omiCurrentIp));
SendPingAsync(sniIPaddress);
}
Console.WriteLine(" --- Press any key to continue --- ");
Console.ReadKey();
} // Main
// http://pberblog.com/post/2009/07/21/Multithreaded-ping-sweeping-in-VBnet.aspx
// http://www.cyberciti.biz/faq/how-can-ipv6-address-used-with-webbrowser/#comments
// http://www.kloth.net/services/iplocate.php
// http://bytes.com/topic/php/answers/829679-convert-ipv4-ipv6
// http://stackoverflow.com/questions/1434342/ping-class-sendasync-help
public void SendPingAsync(System.Net.IPAddress sniIPaddress)
{
int iTimeout = 5000;
System.Net.NetworkInformation.Ping myPing = new System.Net.NetworkInformation.Ping();
System.Net.NetworkInformation.PingOptions parmPing = new System.Net.NetworkInformation.PingOptions();
System.Threading.AutoResetEvent waiter = new System.Threading.AutoResetEvent(false);
myPing.PingCompleted += new System.Net.NetworkInformation.PingCompletedEventHandler(AsyncPingCompleted);
string data = "ABC";
byte[] dataBuffer = Encoding.ASCII.GetBytes(data);
parmPing.DontFragment = true;
parmPing.Ttl = 32;
myPing.SendAsync(sniIPaddress, iTimeout, dataBuffer, parmPing, waiter);
//waiter.WaitOne();
}
private void AsyncPingCompleted(Object sender, System.Net.NetworkInformation.PingCompletedEventArgs e)
{
System.Net.NetworkInformation.PingReply reply = e.Reply;
((System.Threading.AutoResetEvent)e.UserState).Set();
if (reply.Status == System.Net.NetworkInformation.IPStatus.Success)
{
Console.WriteLine("Address: {0}", reply.Address.ToString());
Console.WriteLine("Roundtrip time: {0}", reply.RoundtripTime);
}
// Keep starting those async pings until all ips have been invoked.
lock (_syncLock)
{
if (_omiToIP < _omiCurrentIp)
{
++_omiCurrentIp;
System.Net.IPAddress sniIPaddress = System.Net.IPAddress.Parse(IPn2IPv4(_omiCurrentIp));
SendPingAsync(sniIPaddress);
}
}
}
}

I guess the problem is that you are spawning roughly 63K ping requests near-simultaneously. Without further memory profiling it is hard to say which parts consume the memory. You are working with network resources, which probably are limited. Throttling the number of active pings will ease the use of local resources, and also network traffic.
Again I would look into the Task Parallel Library, the Parallel.For construct combined with the Task<T> should make it easy for you.
Note: for .Net 3.5 users, there is hope.

I did something similar to this. The way I solved the problem on my project was to cast the ping instance to IDisposable:
(myPing as IDisposable).Dispose()
So get a list of say 254 ping instances running asynchronously (X.X.X.1/254) and keep track of when all of them have reported in. When they have, iterate through your list of ping instances, run the above code on each instance, and then dump the list.
Works like a charm.

pseudo-code
do
if pings_running > 100 then
sleep 100ms.
else
start ping
endif
loop while morepings

Finally... No ping requried at all...
http://www.codeproject.com/KB/cs/c__ip_scanner.aspx
All I needed to do is to make it thread-safe for debugging.
Changing Add to:
void Add( string m )
{
Invoke(new MethodInvoker(
delegate
{
add.Items.Add(m);
}));
//add.Items.Add( m );
}
Private Sub Add(m As String)
Invoke(New MethodInvoker(Function() Do
add.Items.Add(m)
End Function))
'add.Items.Add(m);'
End Sub