We Keep Coding

outofmemoryexception: threads using too much virtual memory, about 12GB

actually i have to create lots of threads to send pcap file using UDP protocol, when thread completely sends the pcap file it then sleep for some time. when i sleep thread to 420 seconds virtual memory gets full after creating more than 3100 threads and program throws a OutOfMemoryException.
i searched internet about this problem but found that a thread takes only 1MB to create and pcap file is just 60KB, and my 3100 threads are consuming more than 12GB(1.06*3100<12GB). on the other hand physical memory is not used more than 200MB. i have to create more than 5000 threads at the same time
what am i doing wrong? can anyone help me?
thanks
my code:
public static void send_pcap_file_with_single_port()
{
string callID = Call_ID;
try
{
//CREATING CONNECTION HERE
using (FileStream stream = new FileStream("g711a.pcap", FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite))
{
for (Pos = 0; Pos < (StreamBytes - ChunkSize); Pos += ChunkSize)
{
//creating RTP_header here
stream.Read(RTP_payload, 0, ChunkSize);
//combining both the byte arrays
System.Buffer.BlockCopy(RTP_header, 0, Bytes_to_send, 0, RTP_header.Length);
System.Buffer.BlockCopy(RTP_payload, 0, Bytes_to_send, 16, RTP_payload.Length);
RTPpacket_queue.Enqueue(Bytes_to_send);
//RTP_handler.Send(Bytes_to_send, Bytes_to_send.Length, remote_EP);
}
//done processing here
stream.Close();
stream.Dispose();
RTP_count++;
GC.Collect();
}
System.Threading.Thread.Sleep(420000);
}
catch (Exception e)
{
//using (StreamWriter sw = new StreamWriter(stream_logFile))
//{
// sw.WriteLine(e.ToString());
//}
//send_BYE_message_toSIPp(client, "BYE", 5060, 2, callID);
Console.WriteLine(e.ToString());
}
}
creating threads here:
Thread RTP_sender = new Thread(new ThreadStart(send_pcap_file_with_single_port));
RTP_sender.Start();

In simple terms you exhaust you garbage collector by creating objects in long term pile (objects that survive longer then few seconds). Fix would be to free and recreate thread when it is needed.
In any case by default i5 has 2 cores, if you have 3 or more threads than they are running them on same cpu. Running 3000+ of them means 1500 each, it is not a problem unless they try to write in same place (in case they start blocking like hell).

To demonstrate you do not need 5000 permanent threads to accomplish something like this, I have created a sample program.
The program does not do much really but what it does do is that it creates 5000 objects, each of which creates a thread when it needs to do its work. There isn't any actual work being done other than simply sleeping for a random interval but still.
Just run the program, leave it running for a while and keep an eye on its memory use. You will see it is very much manageable while still actually doing work on 5000 objects.
You will probably need to actually be creative in applying this approach to your situation but you could do something along the lines of what I am doing.
namespace Test
{
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Diagnostics;
using System.Threading;
public static class MainClass
{
public static Random sleeper = new Random ();
public static void Main (string[] args)
{
Stopwatch timer = new Stopwatch ();
List<WorkerClass> workload = new List<WorkerClass> ();
// Create a workload of 5000 objects
for (int i = 0; i < 5000; i++) {
workload.Add (new WorkerClass ());
}
int fires = 0;
// Start processing the workload
while (true) {
// We'll measure the time it took to go through the entire workload
// to illustrate that it does not take all that long.
timer.Restart ();
foreach (WorkerClass w in workload) {
// for each of the worker objects in the entire workload
// we decrease its internal counter by 1.
// Because after the loop is done, we sleep for 1 secondd
// that amounts to reducing the counter by 1 every second.
w.counter--;
if (w.counter == 0) {
fires++;
// Once the counter hits 0, do the work.
w.DoWork ();
}
}
timer.Stop ();
Console.WriteLine ("Processing the entire workload of {0} objects took {1} milliseconds, {2} workers actually fired.", workload.Count, timer.ElapsedMilliseconds, fires);
fires = 0;
Thread.Sleep (1000);
}
}
}
public class WorkerClass
{
public int counter = 0;
public WorkerClass ()
{
// When the worker is created, set its internal counter
// to a random value between 5 and 10.
// This is to mimic sleeping it for a random interval.
// Also see the primary loop in MainClass.Main
this.counter = MainClass.sleeper.Next (5, 10);
}
public void DoWork ()
{
// Whenever we do the work, we'll create a background worker thread
// that actually does the work.
BackgroundWorker work = new BackgroundWorker ();
work.RunWorkerCompleted += (object sender, RunWorkerCompletedEventArgs e) => {
// This simulates going back to sleep for a random interval, see
// the main loop in MainClass.Main
this.counter = MainClass.sleeper.Next (5, 10);
};
work.DoWork += (object sender, DoWorkEventArgs e) => {
// Simulate working by sleeping a random interval
Thread.Sleep (MainClass.sleeper.Next (2000, 5000));
};
// And now we actually do the work.
work.RunWorkerAsync ();
}
}
}

Prevent my console application from closing after a .Net.Timer starts

So I have this simple code:
static void Main(string[] args)
{
var timer = new Timer(0.5 * 60 * 1000); // 0.5 minutes times 60 seconds times 1000 miliseconds
timer.Elapsed += new System.Timers.ElapsedEventHandler(Start);
timer.AutoReset = true;
timer.Start();
}
Which starts a timer that starts the whole program logic. Problem is, the console application closes and nothing is done. How do I prevent this? I know that if I do something like while(true) it works, but that does not seem like an elegant/proper solution.

Maybe try
Console.Read();
At the end of the code
Thanks to this your console window will not be closed as long as a user will not press a key.

The last answer is 7 years old. In the meantime, methods from System.Threading like Timer, Sleep, ...
should not be used anymore. They carry the risk of deadlocks, and I have to warn against their use especially in ASP.NET Core.
There are several things to consider:
No program runs forever. What happens when the program should be closed (by a shutdown or a signal)?
The termination must be implemented in your program logic. Many async .NET Core methods supports a CancellationToken as a parameter.
There are 2 approaches to create an "endless" program:
Worker Services in .NET Core are the common approach today to write programs that should run permanently as a service. They can be registered as Windows Service.
More Infos on learn.microsoft.com and devblogs.microsoft.com if you want to register your service as a "real" service.
If you need a "real" console application that performs tasks in the foreground, this can be done with the following sample code:
using System;
using System.Threading.Tasks;
using System.Threading;
namespace DemoApp
{
internal class Program
{
private static int _isRunning = 0;
private static async Task Main(string[] args)
{
// No program can run infinitely. We always have to provide a mechanism for termination.
var tcs = new CancellationTokenSource();
var periodTimeSpan = TimeSpan.FromSeconds(10);
// This mechanism is CTRL+C, so we catch this.
Console.CancelKeyPress += (sender, e) =>
{
tcs.Cancel();
e.Cancel = true;
};
try
{
// No endless loop. We are checking for a cancellation request!
while (!tcs.IsCancellationRequested)
{
// Perform your work.
var task1 = Run(tcs.Token);
var task2 = Task.Delay(periodTimeSpan, tcs.Token);
// If the execution time of Run is greater than periodTimeSpan, we will wait. Depends on your needs.
await Task.WhenAll(task1, task2);
}
}
catch (TaskCanceledException)
{
Console.WriteLine("User canceled your app.");
}
}
private static async Task Run(CancellationToken token)
{
// Should never occur if you use WhenAll()
if (Interlocked.Exchange(ref _isRunning, 1) == 0)
{
// Perform your work.
_isRunning = 0;
}
}
}
}

Add a Console.ReadKey(); this will allow you to close the console window by pressing any key.
static void Main(string[] args)
{
var timer = new Timer(0.5 * 60 * 1000); // 0.5 minutes times 60 seconds times 1000 miliseconds
timer.Elapsed += new System.Timers.ElapsedEventHandler(Start);
timer.AutoReset = true;
timer.Start();
Console.ReadKey();
}

If you want to just run a timer and wait, then Console.Read() is your friend.
The reason why your code is terminating is because the function initializes the timer, starts it and then ... hits the end of the Main function. As a result, the function exits.
The current code doesn't do anything useful (besides starting a timer which is ignored).
If you want to see your timer in action, do some other work after you start the timer. Then, stop the timer after an interval and print it/evaluate it how you deem fit. (The timer doesn't run on the main thread, whereas something like Console.Read() will be on the main thread and thus, block the thread until you have input)

Your console application does not end because the timer starts. It ends because the program reaches the end of the Main() method.
Whatever action you take to prevent the program from exiting Main() will help. While many proposed solutions actually do this. I wanted to highlight the fact that it's not the timer causing the issue.

stopwatch c# behave differently in different threads?

I'm currently using a stopwatch as a global timer. I have main thread running, another thread, and an event method.
The main thread launches the other thread and the event method is triggered by events. Both methods will call the stopwatch and get its time. The thing is, the times are not consistent:
from main thread:
START REC AT 9282
STOp REC AT 19290
from another thread:
audio 1
audio 304
audio 354
audio 404
audio 444
audio 494
audio 544
audio 594
from event method:
video 4
video 5
video 29
video 61
video 97
video 129
video 161
I don't get why if i start my rec at 9282, the other two functions that call the stopwatch will have timers that start at zero? Is this a thread related issue? How can i fix this? Thanks
UPDATE:*********
when i save my frames i changed to:
long a = relogio.EllapseMilliseconds
i print out this value and its ok, as expected. but when i print the value stored in the lists, they come as starting from the beggining. strange huh?
SORRY FOR ALL THE TROUBLE, I PRINTED IT WITHOUT THE STARTING TIME,THATS WHY THEY ALL SEEMED TO START FROM ZERO! MANY THANKS AND SORRY!
main thread
private void Start_Recording_Click(object sender, RoutedEventArgs e)
{
rec_starting_time = relogio.ElapsedMilliseconds;
Console.WriteLine("START REC AT " + rec_starting_time);
write_stream.enableRecording();
Thread a = new Thread(scheduleAudioVideoFramePicks);
a.Start();
scheduleAudioVideoFramePicks - this thread just counts the time, so i know when to stop
//while....
if (rec_starting_time + time_Actual > rec_starting_time+recording_time * 1000)//1000 - 1s = 1000ms
{
totalRecordingTimeElapsed = true;
write_stream.disableRecording();
Console.WriteLine("STOp REC AT " + relogio.ElapsedMilliseconds);
}
//end while
lock (list_audio)
{
int b = 0;
//print time of frames gathered
foreach(AudioFrame item in list_audio){
Console.WriteLine("audio " + (item.getTime() - rec_starting_time));
}
lock (list_video)
{
}
foreach (VideoFrame item in list_video)
{
Console.WriteLine("video " + (item.getTime() - rec_starting_time));
}
}
the another thread, where i get the time
if (write_stream.isRecording())
{
list_audio.Enqueue(new AudioFrame(relogio.ElapsedMilliseconds, audioBuffer));
}
event method
if (write_stream.isRecording())
{
list_video.Add(new VideoFrame(relogio.ElapsedMilliseconds, this.colorPixels));
}~
i dont know if this is relevant, but i start my stopwatch like this
public MainWindow()
{
InitializeComponent();
//some code
this.relogio = new Stopwatch();
relogio.Start();
}

Stopwatch is not threadsafe, particularly for 32-bit programs.
It uses the Windows API call QueryPerformanceCounter() to update a private long field. On 32-bit systems you could get a "torn read" when one thread reads the long value while another thread is updating it.
To fix that, you'd have to put a lock around access to the Stopwatch.
Also note that one some older systems there were bugs where inconsistent values could be returned from different threads calling QueryPerformanceCounter(). From the documentation:
On a multiprocessor computer, it should not matter which processor is called. However, you can get different results on different processors due to bugs in the basic input/output system (BIOS) or the hardware abstraction layer (HAL). To specify processor affinity for a thread, use the SetThreadAffinityMask function.
I have never encountered this bug myself, and I don't think it's very common.
What results do you get with the following test program? The times should be mostly increasing in value, but you are liable to get one or two out of order just because their threads get rescheduled just after they've read a value and before they add it to the queue.
namespace Demo
{
class Program
{
Stopwatch sw = Stopwatch.StartNew();
object locker = new object();
ConcurrentQueue<long> queue = new ConcurrentQueue<long>();
Barrier barrier = new Barrier(9);
void run()
{
Console.WriteLine("Starting");
for (int i = 0; i < 8; ++i)
Task.Run(()=>test());
barrier.SignalAndWait(); // Make sure all threads start "simultaneously"
Thread.Sleep(2000); // Plenty of time for all the threads to finish.
Console.WriteLine("Stopped");
foreach (var elapsed in queue)
Console.WriteLine(elapsed);
Console.ReadLine();
}
void test()
{
barrier.SignalAndWait(); // Make sure all threads start "simultaneously".
for (int i = 0; i < 10; ++i)
queue.Enqueue(elapsed());
}
long elapsed()
{
lock (locker)
{
return sw.ElapsedTicks;
}
}
static void Main()
{
new Program().run();
}
}
}
Having said all that, the most obvious answer is that in fact you aren't sharing a single Stopwatch between the threads, but instead you have accidentally started a new one for each thread...

How to correctly implement a custom timer in .Net

I have a requirement for a timer that has the following behaviour:
Millisecond accuracy
I want the tick event handler to only be called once the current tick handler has completed (much like the winforms timer)
I want exceptions on the main UI thread not to be swallowed up by the thread timer so this requires Invoke/Send instead of BeginInvoke/Post
I've played around with CreateTimerQueueTimer and had some success but at the same time had problems with code reentrance and/or locks when deleting the timer.
I decided to create my own timer so that I could get a better idea of what is going on under the hood so that I can fix the locking and reentrance problems. My code seems to work fine leading me to believe that I may as well use it. Does it look sound?
I've put in a check if the timer is deleted to make sure that the deletion is complete before the timer can be created again. Does that look ok?
Note: I should say that I call timeBeginPeriod(1) and timeEndPeriod(1) inorder to achieve the millisecond accuracy.
(The following code is converted from vb.net to c#, so apologies for any missed mess-ups}
ETA: I've found a problem with it. If the timer is running at an interval of 1 millisecond, and I call, say, Change(300), it locks up # while (this.DeleteRequest). This
must be because the TimerLoop is in the this.CallbackDelegate.Invoke(null) call.
public class MyTimer : IDisposable
{
private System.Threading.TimerCallback CallbackDelegate;
private bool DeleteRequest;
private System.Threading.Thread MainThread;
public MyTimer(System.Threading.TimerCallback callBack)
{
this.CallbackDelegate = callBack;
}
public void Create(int interval)
{
while (this.DeleteRequest) {
System.Threading.Thread.Sleep(0);
}
if (this.MainThread != null) {
throw new Exception("");
}
this.MainThread = new System.Threading.Thread(TimerLoop);
// Make sure the thread is automatically killed when the app is closed.
this.MainThread.IsBackground = true;
this.MainThread.Start(interval);
}
public void Change(int interval)
{
// A lock required here?
if (!this.IsRunning()) {
throw new Exception("");
}
this.Delete();
this.Create(interval);
}
public void Delete()
{
this.DeleteRequest = true;
}
public bool IsRunning()
{
return (this.MainThread != null) && this.MainThread.IsAlive;
}
private void TimerLoop(object args)
{
int interval = (int)args;
Stopwatch sw = new Stopwatch();
sw.Start();
do {
if (this.DeleteRequest) {
this.MainThread = null;
this.DeleteRequest = false;
return;
}
long t1 = sw.ElapsedMilliseconds;
// I want to wait until the operation completes, so I use Invoke.
this.CallbackDelegate.Invoke(null);
if (this.DeleteRequest) {
this.MainThread = null;
this.DeleteRequest = false;
return;
}
long t2 = sw.ElapsedMilliseconds;
int temp = Convert.ToInt32(Math.Max(interval - (t2 - t1), 0));
sw.Reset();
if (temp > 0) {
System.Threading.Thread.Sleep(temp);
}
sw.Start();
} while (true);
}
// The dispose method calls this.Delete()
}

I would recommend using p/Invoke and using the timers from Win32's Timer Queues:
http://msdn.microsoft.com/en-us/library/ms686796(v=vs.85).aspx
One should be mindful that the managed CLR environment has a lot of non-determinism built into it, garbage collection, for instance. Just because your timer is has a period of 1 millisecond doesn't mean that that is necessarily what happens.
Also, the documentation doesn't mention it, but the callback invoked by the timer must be pinned in memory and not garbage collectable, via a GCHandle or other construct. When a timer (or timers, if you kill off a timer queue), the callback will be executed one last time. Not sure whether that happens by the internal wait expiring, or by signalling the internal event handle.
Execution of DeleteTimerQueueTimer() and DeleteTimerQueueEx() can be made synchronous, so they won't return until all timers have signalled and invoked their last callback, but doing that would be suboptimal.
If you don't pin the callbacks and prevent them from being garbage-collected, things will go swimmingly...most of the time. You'll encounter random exceptions.
Also, the callback should be smart enough to bail out if the timer is being deleted, lest it make reference to something that's already GC'd.

μTimer would be a better example!
You can find it here # https://stackoverflow.com/questions/15725711/obtaining-microsecond-precision-using-net-without-platform-invoke?noredirect=1#comment22341931_15725711
It provides accurate wait times down to 1µs and possibly lower depending on your NIC!
Let me know if you need anything else!

Locking main() thread

Not exactly sure on the terminology here but here I go basically I have the main() thread of my application that starts and calls two threads, one sets up an event handler to wait for specific registry keys to change, while the other starts a timer to write any changes made to an xml file every 5 mins or so and runs continuously. The issue I have is that once the two methods called are initialized it goes back to main and ends the program. My relevant code sections can be found below, so any help would be appreciated:
static void Main(string[] args)
{
runner one = new runner();
runner two = new runner();
Thread thread1 = new Thread(new ThreadStart(one.TimerMeth));
Thread thread2 = new Thread(new ThreadStart(two.start));
thread1.Start();
thread2.Start();
thread1.Join();
thread2.Join();
}
public void TimerMeth()
{
System.Timers.Timer timer = new System.Timers.Timer();
timer.Elapsed += new ElapsedEventHandler(OnElapsedTime);
timer.Interval = 300000;
timer.Enabled = true;
}
private void OnElapsedTime(object source, ElapsedEventArgs e)
{
file write = new file();
write.write(RegKeys);
}
public void start()
{
if (File.Exists("C:\\test.xml"))
{
file load = new file();
RegKeys = load.read(RegKeys);
}
string hiveid = "HKEY_USERS";
WindowsIdentity identity = WindowsIdentity.GetCurrent();
string id = identity.User.ToString();
string key1 = id + "\\\\Software\\\\Microsoft\\\\Windows NT\\\\CurrentVersion\\\\Windows Messaging Subsystem\\\\Profiles\\\\Outlook\\\\0a0d020000000000c000000000000046";
List<string> value1 = new List<String> { "01020402", "test" };
valuechange = new RegistryValueChange(hiveid, key1, value1);
valuechange.RegistryValueChanged += new EventHandler<RegistryValueChangedEventArgs>(valuechange_RegistryValueChanged);
try
{
valuechange.Start();
}
catch
{
StreamWriter ut;
ut = File.AppendText("C:\\test.txt");
ut.WriteLine("error occured in starting management");
ut.Close();
}
file test = new file();
test.checkfile("C:\\test.xml");
}
void valuechange_RegistryValueChanged(object sender, RegistryValueChangedEventArgs e)
{
// deals with the returned values
}
Basically all the code works fine I've been testing it in a windows form application but now I need to run it in a standalone app with no interface in the background and need it to keep writing to the xml file and the change event to stay alive.

As you can expect, the Main() method is terminating because execution flows out of the Join() methods back to the main thread, and then terminates.
Either place loops in the methods TimerMeth() and start(), or more appropriately redesign the application into a Windows Service (as zac says).

You have a couple of issues.
Your first thread is simply creating a timer (which launches another thread). This thread is terminating very quickly, making your call to Join rather meaningless. What this thread should be doing is actually doing the waiting and the checking. You can easily adapt your code like this:
public void TimerMeth()
{
System.Timers.Timer timer = new System.Timers.Timer();
timer.Elapsed += new ElapsedEventHandler(OnElapsedTime);
timer.Interval = 300000;
timer.Enabled = true;
try
{
while(true)
{
OnElapsedTime(null, null); // you should change the signature
Thread.Sleep(30000);
}
}
catch(ThreadAbortException)
{
OnElapsedTime(null, null);
throw;
}
}
Obviously you should change the signature of OnElapsedTime to eliminate the parameters, since they aren't used here.
I have a feeling that something is amiss in the way that the file handling is being done, but given that I don't understand exactly what that code does, I'm going to refrain from commenting. What, exactly, is the purpose of the file? Where is RegKeys defined?

Try building this into a windows service.
This thread contains two suggestions for finding the logged on user from a windows service, but I am not sure if they work.

Your methods will run once, then the thread will exit. There is nothing to keep them running.
Try this:
thread1.IsBackground = true;
thread2.IsBackground = true;
public void start()
{
while(true)
{
// ... do stuff
Thread.Sleep(1000*60*5) // sleep for 5 minutes
}
}
public void TimerMeth()
{
while(true)
{
file write = new file();
write.write(RegKeys);
Thread.Sleep(30000);
}
}
As other posters have noted, you will also then need to ensure your main method doesn't exit. Making the application a windows service seems like a good way to solve this in your case.
You might also want to handle ThreadInterruptedException and ThreadAbortException on your threads.
And if you really want to get into the nitty gritty of threading, check out this Free C# Threading E-Book by Joe Albahari.

To keep the main thread alive, one of the simplest ways is to add the following line to the end of your main function:
Thread.Sleep(Timeout.Infinite);

The Thread will terminate when your ThreadStart function returns, which allows the main thread to continue after Join(). Since you are just setting up a timer to fire off, the method will return very quickly. You need to provide a lock of some sort to keep your application from exiting.
Also, it doesn't look like you need to use threads at all to do what you are trying. Instead, just use the Timer and provide the lock to keep your Main() from terminating.

It looks to me like all of your functions are completing? ie, they all "fall out the bottom". Once all the functions have run through there is nothing left to do and your app will close. You want to run a loop of some sort in main.
You will also need to take a look at your timer. I suspect it is currently being garbage collected. You create it in the scope of your function but that function is being left so there is no longer a reference to your timer and it will be collected. Your timer needs to be a root.