I'm debugging a C# application that becomes almost unresponsive after a few days. The application calculates memory/CPU usage every second and displays it in the footer of the main UI.
The cause for the unresponsiveness is because of the time it takes to fetch the RawValue of a PerformanceCounter ("Working Set - Private"). After a couple of days, it takes almost a second to fetch the RawValue, freezing the main UI thread. If I restart my computer, everything is fast again for a few days until it slowly becomes less responsive. If I recompile this application without the PerformanceCounter code (it's open source), it runs normally immediately.
To rule out that it's the application, here's some sample code that does the exact same thing:
static void Main(string[] args)
{
using (var memoryWorkingSetCounter = new PerformanceCounter("Process", "Working Set - Private", Process.GetCurrentProcess().ProcessName, true))
{
while (!Console.KeyAvailable)
{
var memoryWorkingSetSw = new Stopwatch();
memoryWorkingSetSw.Start();
var memoryWorkingSetValue = memoryWorkingSetCounter.RawValue;
memoryWorkingSetSw.Stop();
Console.WriteLine(memoryWorkingSetValue.ToString());
Console.WriteLine(memoryWorkingSetSw.Elapsed.ToString());
Thread.Sleep(TimeSpan.FromSeconds(1));
}
}
Console.Read();
}
I left this running for ~2.5 days and graphed the Elapsed time in milliseconds:
What could cause a performance counter in Windows to become slow over time? Could another app be not cleaning it up? Is there a way to debug which apps are also looking at this performance counter? I'm on Windows 10.
why you declare Stopwatch inside a loop?
remove any new declaration possible inside loops.
when you do so the memory is increasing overtime and you count on garbage collector to do the work
I'm planning to write a nes emulator. But first, to understand how emulation works, I'll write a Chip-8 emulator.
The emulator is nearly finished. I've some bugs in games, but this will be fixed soon.
My problem number 1 is to synchronize the emulator with the clock speed of the Chip-8.
In the internet I've often read, that the general clock speed should be ~ 540Hz. The timers of the chip should be ticked at a frequenz of 60Hz.
To synchronize my emulator with the Chip-8 I've written follow logic:
private void GameTick()
{
Stopwatch watch = new Stopwatch();
var instructionCount = 0;
_gameIsRunning = true;
while (_gameIsRunning)
{
watch.Restart();
EmulateCycle();
//Updates the internal timer at a 60hz frequenz
//540hz (game tick) divided by 9 equals 60hz (timer tick)
instructionCount++;
if(instructionCount == 9)
{
UpdateSoundAndDelay();
instructionCount = 0;
}
if (_readyToDraw)
{
DrawGraphics();
_readyToDraw = false;
}
SetKeys();
//Pause the game to get a virtual clock speed of ca. 540mhz
var elapsedMicroseconds = watch.ElapsedTicks / (Stopwatch.Frequency / (1000L * 1000L));
while(elapsedMicroseconds < 1852)
{
elapsedMicroseconds = watch.ElapsedTicks / (Stopwatch.Frequency / (1000L * 1000L));
}
}
}
For more detailed information look at my repo: https://github.com/Marcel-Hoffmann/Chip-8-Emulator
As you can see, for each cpu cycle, I'll wait for 1852 microseconds. The result will be ~ 540 cycles in a second equals to 540Hz.
But I'm not very happy with this logic.
Has someone a better Idea, how to synchronize the clock speed?
This is the typical approach, and has many drawbacks - most notably, unnecessary CPU usage and potentially scheduling issues (your application will be seen as 100% CPU beast, so other applications might get their thread quanta before you under load).
A better approach would use a sleep instead - however, by default, the system timer has nowhere near the frequency to accommodate a wait that's less than 2ms. So if you want to use a sleep, you'll need to change the system timer. This is a bit tricky on older Windows (it's a system-wide setting and has noticeable impact on other applications and general CPU usage), but even in that case, it's better than a "busy loop" - as long as you restore the system settings afterwards. On Windows 8 (and to some extent, 7 and Vista), the timer is asynchronous and no longer requires a busy loop, so it's a lot easier to have higher timer resolution.
The system timer APIs are not exposed by .NET, so you'll need to use P/Invokes (timeBeginPeriod and timeEndPeriod for the old-style API). If this isn't available, you can always fall back to your busy loop :)
I'm working on one program that takes information from files and then stores them in MySQL database. This MySQL database is located in another dedicated server which is much more powerful than this server here. Data is being sent over LAN using 1gbps connection.
It is using 8 threads because my server has 8 cores, but somehow it runs so slowly.
CPU is: Intel Xeon E3-1270 v 3 # 3.50Ghz
RAM: 16 GB ECC
HDD: SATA 3 1TB
My program's CPU usage is only 0-5%
CPU affinity is all 8 cores
So, do you have any ideas what's wrong or how can I increase the speed of my program?
UPDATE:
I updated my code and it appears to be faster:
Parallel.For(0, this.data_files.Count, new ParallelOptions { MaxDegreeOfParallelism = this.MaxThreads }, i =>
{
this.ThreadCount++;
this.ParseFile(this.GetSource());
});
Here's a code snippet that deploys threads:
while (true)
{
if (this.ThreadCount < this.MaxThreads)
{
Task.Factory.StartNew(() =>
this.ParseFile(this.GetFile())
);
this.ThreadCount++;
}
else
{
Thread.Sleep(1);
}
this.UpdateConsole();
}
GetFile function:
private string GetFile()
{
string file = "";
string source = "";
while (true)
{
if (this.data_files.Count() != 0)
{
file = this.data_files[0];
this.data_files.RemoveAt(0);
if (File.Exists(file) == true)
{
source = File.ReadAllText(file);
File.Delete(file);
break;
}
}
}
return source;
}
I'm working on one program that takes information from files and then stores them in MySQL database.
Clearly your program is not CPU bound, it's IO bound. The bottlenecks are going to be based on your hard disk(s) and your network connection. Odds are even a single thread is going to be able to ensure proper utilization of these resources (in a well designed application). Adding extra threads generally won't help, it'll just create a bunch of threads that will spend their time waiting on various IO operations.
To use all the hardware resources is not the right goal for a program to have.
Instead, a better goal is to be as fast as possible. This is significantly different. While using more hardware resources can help, it is not always sufficient.
Sometimes, adding more resources to a problem doesn't help. In those cases, don't. Adding threads makes your program more complex, but not necessarily faster as you've seen.
C# already has good Asynchronous programming features with the TPL (which you are already using), so why not take advantage of that?
This will mean that the .NET framework will automatically manage the threads for you in an efficient way.
Here's what I propose:
foreach (var file in GetFilesToRead()) {
var task = PerformOperation(file);
// Keep a list of tasks, if you wish.
}
...
Task PerformOperation (string filename) {
var file = await ReadFile(file);
await ParseFile(file);
DoSomething();
}
Note that even in CPU-bound programs, threads (and tasks) may not help you if you're using locks.
Although locks help keep programs well-behaved, they come at a significant performance cost.
Within a lock, only one thread may be executing at a time.
This means that the first thread is locking your _lock instance, and then the other threads are waiting for that lock to be released.
In your program, only one thread is active at a time.
To solve this, don't use locks. Instead, write programs that do not need locks at all. Copy variables instead of sharing them. Use immutable collections instead of mutable collections and so on.
My program above uses exactly zero locks and, as such, will better utilize your threads.
I've got the following code that is supposed to measure the current download and upload speed. The issue I'm facing is that there are often usages recorded that my network and/or internet connection can't even handle (above my bandwidth).
public static IStatistics GetNetworkStatistics(string interfaceName) {
var networkStats = _interfaces[interfaceName];
var dataSentCounter = new PerformanceCounter("Network Interface", "Bytes Sent/sec", interfaceName);
var dataReceivedCounter = new PerformanceCounter("Network Interface", "Bytes Received/sec", interfaceName);
float sentSum = 0;
float receiveSum = 0;
var sw = new Stopwatch();
sw.Start();
while (sw.ElapsedMilliseconds < 1000) {
sentSum += dataSentCounter.NextValue();
receiveSum += dataReceivedCounter.NextValue();
}
sw.Stop();
Console.WriteLine("Download:\t{0} KBytes/s", receiveSum / 1024);
Console.WriteLine("Upload:\t\t{0} KBytes/s\n", sentSum / 1024);
networkStats.AddSentData(sentSum);
networkStats.AddReceivedData(receiveSum);
return networkStats;
}
Sample output:
As you can see most of these entries indicate a pretty heavily used network, up to an excessive amount of almost 160MB/s. I realize that you can't measure transfer speed with just one record (this is test data, in the actual application I use the mean of the latest 3), but even so: how can I ever receive 160MB in one second. I believe it's safe to say that I must have made an error somewhere, but I can't find where.
One thought I had was that I should keep a counter in the loop to show me how many times the PerformanceCounter.NextValue() were accessed (generally between 46 and 48), but in the end I believed this shouldn't matter: in the grand picture I'm still having a too large number for just one second of bandwidth usage. I can't shake the feeling that the performance counter might be the cause though.
Sidenote: the 160MB/s number was recorded the moment I loaded a new youtube video and other (+1000 KB/s) recordings are usually done when I refresh a tab, so it should be a (relative) display of my network usage.
Have I overlooked something in my approach?
Edit:
Upon following #Sam's advice and checking my results against the built-in perfmon.exe I noticed that my bursts in bandwith usage generally occur at the same time as those shown in Perfmon, but mine are way larger. I have tried to link the simultaneous bursts and find something in common, but it seems rather random (possibly because Perfmon might combine several results to get their current speed, whereas I'm only using the latest second).
Same goes for the lower numbers: Perfmon usually shows < 10kbps whereas I'm constantly around 50kbps.
Edit2:
This is the code I used in reference to #Hans' comment
var initSent = dataSentCounter.NextValue();
var initReceived = dataReceivedCounter.NextValue();
Thread.Sleep(1000);
sentSum = dataSentCounter.NextValue() - initSent;
receiveSum = dataReceivedCounter.NextValue() - initReceived;
Of late, I'm becoming more health oriented when constructing my program, I have observed that most of programs take 2 or 3 minutes to execute and when I check on the task scheduler, I see that they consume 100% of CPU usage, can I limit this usage programatically in code? This will certainly enable me to run multiple programs at a given time.
Thanks,
Nidhi
This thread is over four years old, and it still annoys me that the accepted answer criticizes the question rather than answering it. There are many valid reasons you would want to limit the CPU time taken by your program, I can list a few off the top of my head.
It might seem like a waste not to use all free CPU cycles available, but this mentality is flawed. Unlike older CPUs, most modern CPUs do not run at a fixed clock speed - many have power saving modes where they drop the clock speed and cpu voltage when load is low. CPUs also consume more power when performing calculations than they do running NOOPs. This is especially relevant to laptops that require fans to cool the CPU when it is under high load. Running a task at 100% for a short time can use far more energy than running a task at 25% for four times as long.
Imagine you are writing a background task that is designed to index files periodically in the background. Should the indexing task use as much of the CPU as it can at a lower priority, or throttle itself to 25% and take as long as it needs? Well, if it were to consume 100% of the CPU on a laptop, the CPU would heat up, the fans would kick in, and the battery would drain fairly quickly, and the user would get annoyed. If the indexing service throttled itself, the laptop may be able to run with completely passive cooling at a very low cpu clock speed and voltage.
Incidentally, the Windows Indexing Service now throttles itself in newer versions of Windows, which it never did in older versions. For an example of a service that still doesn't throttle itself and frequently annoys people, see Windows Installer Module.
An example of how to throttle part of your application internally in C#:
public void ThrottledLoop(Action action, int cpuPercentageLimit) {
Stopwatch stopwatch = new Stopwatch();
while(true) {
stopwatch.Reset();
stopwatch.Start();
long actionStart = stopwatch.ElapsedTicks;
action.Invoke();
long actionEnd = stopwatch.ElapsedTicks;
long actionDuration = actionEnd - actionStart;
long relativeWaitTime = (int)(
(1/(double)cpuPercentageLimit) * actionDuration);
Thread.Sleep((int)((relativeWaitTime / (double)Stopwatch.Frequency) * 1000));
}
}
First of all, I agree with Ryan that the question is perfectly valid and there are cases where thread priorities are not at all sufficient. The other answers appear highly theoretical and of no practical use in situations where the application is properly designed but still needs to be throttled. Ryan offers a simple solution for cases in which a relatively short task is performed in high frequency. There are cases, however, when the task takes a very long time (say a minute or so) and you cannot or don't want to break it into smaller chunks between which you can do the throttling. For these cases the following solution might be helpful:
Rather that implementing throttling into the business code, you can design the algorithm itself to work at full steam and simply throttle the thread which runs the operation "from the outside". The general approach is the same as in Ryan's answer: Calculate a suspension time based on the current usage and suspend the thread for this timespan before resuming it again. Given a process which you want to throttle, this is the logic:
public static class ProcessManager
{
[Flags]
public enum ThreadAccess : int
{
TERMINATE = (0x0001),
SUSPEND_RESUME = (0x0002),
GET_CONTEXT = (0x0008),
SET_CONTEXT = (0x0010),
SET_INFORMATION = (0x0020),
QUERY_INFORMATION = (0x0040),
SET_THREAD_TOKEN = (0x0080),
IMPERSONATE = (0x0100),
DIRECT_IMPERSONATION = (0x0200)
}
[DllImport("kernel32.dll")]
static extern IntPtr OpenThread(ThreadAccess dwDesiredAccess, bool bInheritHandle, uint dwThreadId);
[DllImport("kernel32.dll")]
static extern uint SuspendThread(IntPtr hThread);
[DllImport("kernel32.dll")]
static extern int ResumeThread(IntPtr hThread);
[DllImport("kernel32.dll")]
static extern int CloseHandle(IntPtr hThread);
public static void ThrottleProcess(int processId, double limit)
{
var process = Process.GetProcessById(processId);
var processName = process.ProcessName;
var p = new PerformanceCounter("Process", "% Processor Time", processName);
while (true)
{
var interval = 100;
Thread.Sleep(interval);
var currentUsage = p.NextValue() / Environment.ProcessorCount;
if (currentUsage < limit) continue;
var suspensionTime = (currentUsage-limit) / currentUsage * interval;
SuspendProcess(processId);
Thread.Sleep((int)suspensionTime);
ResumeProcess(processId);
}
}
private static void SuspendProcess(int pid)
{
var process = Process.GetProcessById(pid);
if (process.ProcessName == string.Empty)
return;
foreach (ProcessThread pT in process.Threads)
{
IntPtr pOpenThread = OpenThread(ThreadAccess.SUSPEND_RESUME, false, (uint)pT.Id);
if (pOpenThread == IntPtr.Zero)
{
continue;
}
SuspendThread(pOpenThread);
CloseHandle(pOpenThread);
}
}
private static void ResumeProcess(int pid)
{
var process = Process.GetProcessById(pid);
if (process.ProcessName == string.Empty)
return;
foreach (ProcessThread pT in process.Threads)
{
IntPtr pOpenThread = OpenThread(ThreadAccess.SUSPEND_RESUME, false, (uint)pT.Id);
if (pOpenThread == IntPtr.Zero)
{
continue;
}
var suspendCount = 0;
do
{
suspendCount = ResumeThread(pOpenThread);
} while (suspendCount > 0);
CloseHandle(pOpenThread);
}
}
}
The benefit of this solution is that the checking interval becomes independent of the duration of your "long running task". Furthermore, business logic and throttling logic are separated. The suspense/resume code is inspired by this thread. Please note that disposal and ending the throttling needs to be implemented in the solution above, it is not production code.
That's not your concern... It's the job of the operating system to distribute processor time between running processes. If you'd like to give other processes first crack at getting their stuff done, then simply reduce the priority of your own process by modifying the Process.PriorityClass value for it.
See also: Windows Equivalent of ‘nice’
You could write a Governor class that throttles the CPU usage. This class would contain a utility method that should be called on a regular basis (e.g. calling this utility function within a while loop of your function) by your CPU bound function. The governor would check if the amount of time elapsed exceeded a particular threshold, and then sleep for a period of time so as to not consume all the CPU.
Here's a simple Java implementation off the top of my head (just so you get the idea) that will throttle the CPU usage to 50% if you have a single threaded CPU bound function.
public class Governor
{
long start_time;
public Governor()
{
this.start_time = System.currentTimeMillis();
}
public void throttle()
{
long time_elapsed = System.currentTimeMillis() - this.start_time;
if (time_elapsed > 100) //throttle whenever at least a 100 millis of work has been done
{
try { Thread.sleep(time_elapsed); } catch (InterruptedExceptione ie) {} //sleep the same amount of time
this.start_time = System.currentTimeMillis(); //reset after sleeping.
}
}
}
Your CPU bound function would instantiate a Governor, and then just call throttle on a regular basis within the function.
Thank all of you for answering. I've been working on this and the exe it runs for a few hours and want to share to help others. I wrote a class I'm going to set and forget in a WPF app that'll encrypt and push data to the cloud, but I couldn't ever have it ever interfere with the timing of the WPF app and what the WPF app needs in the way of resources, which I am also going to add a flag to disable when the WPF app is in it's highest resource consumption state. I've already highly threaded this WPF with the TPL. This solution has both the priority set of the process
myProcess.PriorityClass = ProcessPriorityClass.Idle;
and the CPU percentage limited.
then in my mainDisplay.xaml.cs I'll use
ProcessManagement.StartProcess(5);
in MainWindow()
And there is no window popping up when that exe is run
RedirectStandardOutput = true,
UseShellExecute = false,
CreateNoWindow = true
in the object initalizer
internal class ProcessManagement
{
private static int CpuPercentageLimit { get; set; }
public static void StartProcess(int cpuPercent)
{
CpuPercentageLimit = cpuPercent;
var stopwatch = new Stopwatch();
while (true)
{
stopwatch.Reset();
stopwatch.Start();
var actionStart = stopwatch.ElapsedTicks;
try
{
var myProcess = new Process
{
StartInfo =
{
FileName = #"D:\\Source\\ExeProgram\\ExeProgram\\bin\\Debug\\ExeProgram.exe",
RedirectStandardOutput = true,
UseShellExecute = false,
CreateNoWindow = true
}
};
myProcess.Start();
myProcess.PriorityClass = ProcessPriorityClass.Idle;
myProcess.Refresh();
myProcess.WaitForExit();
var actionEnd = stopwatch.ElapsedTicks;
var actionDuration = actionEnd - actionStart;
long relativeWaitTime = (int)((1 / (double)CpuPercentageLimit) * actionDuration);
var sleepTime = (int)((relativeWaitTime / (double)Stopwatch.Frequency) * 1000);
Thread.Sleep(sleepTime);
myProcess.Close();
}
catch (Exception e)
{
// ignored
}
}
}
}
In my application, there is ample time, like 24/7/365, to upload lots of data, including thousands of images, but the UI also needs to stay active when used and when the system runs, nothing else can be running.
If you have a multi core processor, you can set the Affinity on each process to only use which cores you want it to use. This is the closest method I know of. But it will only allow you to assign percentages that are a factor of 50% on a dual core, and 25% on a quad core.
You can run your program in a thread with a lower threadpriority, the rest is up to your operating system. Having a process eat up 100% of your CPU is not bad. My SETI is usually taking up all my remaining CPU time without bothering my other programs. It only gets a problem when your thread gets priority over more important programs.
According to MSDN, you can only set a thread priority, i.e.
var t1 = new Thread(() => doSomething());
t1.Priority = ThreadPriority.BelowNormal;
t1.Start();
where doSomething is the function you want to create a thead for. The priority can be one of the ThreadPriority enumeration members Lowest, BelowNormal, Normal, AboveNormal, Highest - for a description see the MSDN link above. Priority Normal is the default.
Note that CPU usage also depends on how many cores and logical processors your physical CPU has *) - and how the threads and processes are assigned to those cores (the assignment to a dedicated processor is called "processor affinity" - if you want to know more about that, see this StackOverflow question).
*) To find that out, open the task manager (via Ctrl+Alt+Delete - select "task manager"), go to Performance and select CPU there: Below the utilization graph you can see "Cores" and "Logical processors". A core is a physical unit built into the CPU, while a logical processor is just an abstraction, which means the more cores your CPU consists of, the faster it can process parallel tasks.
I honestly think rather than worry about trying to limit CPU utilization by your app, you should focus more of your energies on profiling the application to uncover and correct bottlenecks and inefficiencies that may exist.
If you code is running at all, it is at 100%
I suppose slipping in some sleeps might have an effect.
I have to wonder about that 2-3 minute figure. I've seen it too, and I suppose it's loading and initializing lots of stuff I probably don't really need.
This is something I have come across a lot of times with complex integrations (for example a daily update of products, stock and pricing on an ecomm system).
Writing the integrations as efficiently as possible is always good, using DB server power where you can instead for iterating objects in the code but at the end of the day these things WILL take up processor time and you may want to run them on a server that is doing other things too.
Sharing is caring ;-)
A good approach to avoid coding a Governor or Throttling mechanism is to use the power of a web server. Expose the integration as a "local API call" and run it in IIS (for instance). There you have multiple throttling options and affinity masks you can apply to the application pool. This can then be easily adjusted "on the fly" to give a good balance and monitor closely.
If there is no other task running, is it wrong for your app to use all the cpu capacity that is available? It is available, as in it's there and it is free to use. So use it!
If you somehow limit the cpu usage of your task, it will take longer to complete. But it will still take the same number of cpu cycles, so you gain nothing. You just slow down your application.
Don't do it. Don't even try it. There's no reason why you should.
I think what you need to do is to understand the performance problem in your application instead of trying to put a cap on the CPU usage.
You Can use Visual Studio Profiler to see why you application is taking 100% CPU for 2-3 minutes in the first place. This should reveal the hot spot in your app, and then you can be able to address this issue.
If you are asking in general regarding how to do resource throttling in windows, then you can look at the "Task" objects, Job objects allows you to set limits such as Working set, process priority...etc.
You can check out the Job objects documentation here
http://msdn.microsoft.com/en-ca/library/ms684161(VS.85).aspx
Hope this helps.
Thanks