I have two functions using a lock statement, where after some operations finished, I redirect the output in a .txt file. I have realised that their executions take a lot of time, resulting in blocking their operations and degrading the app's performance in general.
I was thinking that the high execution time could be due to the write operations into a file. What would be the most efficient way to reduce the execution time? Should I use another thread for the write operations, is it possible inside a lock without holding the lock?
A simplified version of my code is illustrated below:
StatsInformation statsInfo = new StatsInformation ();
List<int> lInt = new List<int>();
public void FunctionEnq(List<byte> lByte, int _int)
{
lock (OperationLock)
{
//Do some work here
lInt.Add(_int);
string result = "New Int " + _int + " size " + lInt.Count + " time " + DateTime.Now.ToString("hh:mm:ss.fff");
statsInfo.WriteStatsInFile(result);
}
}
public (List<byte> _outByte, int _time) FunctionDeq()
{
List<byte> _outByte = new List<byte> ();
int _time = -1;
lock (OperationLock)
{
//Do some work here
_outByte.Add(...);
int _int = lInt[0];
//do operations
_time = _int;
lInt.RemoveAt(0);
string result = "Get Int " + _int + " new size " + lInt.Count + " time " + DateTime.Now.ToString("hh:mm:ss.fff");
statsInfo.WriteStatsInFile(result);
}
return (_outByte, _time);
}
I need to get the 'Commit size' (Windows Task Manager > Details) of a process in C#.
At first sight the Process class does not provide a relevant property.
Can somebody help me?
Edited
private static void ShowCommitSize(string processName)
{
Process process = Process.GetProcessesByName(processName).FirstOrDefault();
if (process != null)
{
var pagedMemMb = ConvertBytesToMegabytes(process.PagedMemorySize64);
Console.WriteLine(process.ProcessName + "\t" + process.Id + "\t" + Math.Round(pagedMemMb, 3) + " MB");
}
Console.ReadLine();
}
static double ConvertBytesToMegabytes(long bytes)
{
return (bytes / 1024f) / 1024f;
}
Output
There is a difference between my calculated Commit Size and the 'Commit Size' in Task Manager. Any ideas?
Solution
private static void ShowCommitSize(string processName)
{
var process = Process.GetProcessesByName(processName).FirstOrDefault();
if (process != null)
{
var memKb = ConvertBytesToKilobytes(process.PagedMemorySize64);
Console.WriteLine(process.ProcessName + "\t" + process.Id + "\t" + memKb.ToString("N") + " K");
}
Console.ReadLine();
}
static double ConvertBytesToKilobytes(long bytes)
{
return (bytes / 1024f);
}
This value is in the PagedMemorySize64 property. The documentation mentions that this the "Page File Size" process performance counter and over here it is documented that this is referred to as "Commit Size" in Task Manager on Vista/2008 (and I would assume newer OSes).
This question already has answers here:
Read Big TXT File, Out of Memory Exception
(6 answers)
Closed 6 years ago.
The following code works fine with small txt files , but if we have large txt files its giving outofmemory exception at string[] array = File.ReadAllLines("hash.txt");
hash.txt file is a 500 mb
I tried few suggestions from internet but i didn't get that worked.
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
namespace Hash_Parser
{
internal class Program
{
private static List<string> users = new List<string>();
private static Dictionary<string, int> hash_original = new Dictionary<string, int>();
private static List<string> hash_found = new List<string>();
private static List<string> pass = new List<string>();
private static string hash_path = "split.txt";
private static void split()
{
Console.WriteLine("Splitting...");
StreamWriter streamWriter = new StreamWriter("user.txt");
StreamWriter streamWriter2 = new StreamWriter("hash.txt");
string[] array = File.ReadAllLines(Program.hash_path);
for (int i = 0; i < array.Length; i++)
{
string text = array[i];
string[] array2 = text.Split(new char[]
{
':'
}, 2);
if (array2.Count<string>() >= 2)
{
streamWriter.WriteLine(array2[0]);
streamWriter2.WriteLine(array2[1]);
}
}
streamWriter.Close();
streamWriter2.Close();
Console.WriteLine("Saved as user.txt and hash.txt");
}
private static void populate()
{
Console.WriteLine("Populating lists...");
Program.users.AddRange(File.ReadAllLines("user.txt"));
Program.pass.AddRange(File.ReadAllLines("pass.txt"));
Program.hash_found.AddRange(File.ReadAllLines("found.txt"));
int num = 0;
string[] array = File.ReadAllLines("hash.txt");
for (int i = 0; i < array.Length; i++)
{
string key = array[i];
Program.hash_original.Add(key, num);
num++;
}
}
private static void seek()
{
StreamWriter streamWriter = new StreamWriter("userpass.txt");
int num = 0;
int num2 = 100;
foreach (string current in Program.hash_found)
{
if (Program.hash_original.ContainsKey(current))
{
streamWriter.WriteLine(Program.users[Program.hash_original[current]] + ":" + Program.pass[num]);
}
num++;
if (num >= num2)
{
Console.Title = string.Concat(new object[]
{
"Processed: ",
num,
" : ",
Program.hash_found.Count
});
num2 += 1000;
}
}
Console.Title = string.Concat(new object[]
{
"Processed: ",
num,
" : ",
Program.hash_found.Count
});
streamWriter.Close();
}
private static void Main(string[] args)
{
Console.WriteLine("Split hash /split");
Console.WriteLine("'split.txt'\n");
Console.WriteLine("Parse hashes /parse");
Console.WriteLine("'user.txt' | 'found.txt' | 'hash.txt' | 'pass.txt'");
string a = Console.ReadLine();
if (a == "/split")
{
Program.split();
}
else
{
if (a == "/parse")
{
Program.populate();
Console.WriteLine("Processing...");
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
Program.seek();
stopwatch.Stop();
Console.WriteLine("Saved as userpass.txt");
Console.WriteLine("Time elapsed: " + stopwatch.Elapsed);
Console.ReadKey();
}
}
}
}
}
Thnaks for ur help.
Try this code :
foreach (var line in File.ReadLines(_filePath))
{
//Don't put "line" into a list or collection.
//Just make your processing on it.
}
Quoted Text: Just use File.ReadLines which returns an IEnumerable and doesn't load all the lines at once to the memory.
Quote Link : https://stackoverflow.com/a/13416225/3041974
I hope it helps.
Please be aware of process limits in .NET
http://www.codeproject.com/Articles/483475/Memory-Limits-in-a-NET-Process
For instance, a 32 bit system cannot have more than 4 GB of physical memory. Needless to say that 2^32 will give you a virtual address space with 4.294.967.296 different entries, and that’s precisely where the 4GB limit comes from. But even having those 4GB available on the system, your application will actually be able to see 2GB only. Why?
Because on 32 bits systems, Windows splits the virtual address space
into two equal parts: one for User Mode applications, and another one
for the Kernel (system applications). This behavior can be overridden
by using the "/3gb" flag in the Windows boot.ini config file. If we do
so, the system will then reserve 3GB for user applications, and 1 GB
for the kernel.
What is the process MEM Usage in Task Manager?
I wanted to test the overhead ascribed to a program by using await/async.
To test this, I wrote the following test class:
public class Entity : INotifyCompletion {
private Action continuation;
private int i;
public void OnCompleted(Action continuation) {
this.continuation = continuation;
}
public Entity GetAwaiter() {
return this;
}
public Entity GetResult() {
return this;
}
public bool IsCompleted { get { return true; } }
public void Execute() {
if (i > 0) Console.WriteLine("What");
}
}
And then I wrote a test harness. The test harness iterates through TestA and TestB 1600 times, measuring the latter 1500 times only (to allow the JIT to 'warm up'). set is a collection of Entity objects (but the implementation is irrelevant). There are 50,000 entities in the set. The test harness uses the Stopwatch class for testing.
private static void DoTestA() {
Entity[] objects = set.GetElements();
Parallel.For(0, objects.Length, async i => {
Entity e = objects[i];
if (e == null) return;
(await e).Execute();
});
}
private static void DoTestB() {
Entity[] objects = set.GetElements();
Parallel.For(0, objects.Length, i => {
Entity e = objects[i];
if (e == null) return;
e.Execute();
});
}
The two routines are identical, except one is awaiting the entity before calling Execute() (Execute() does nothing useful, it's just some dumb code to make sure the processor is really doing something for each Entity).
After executing my test in release mode targeting AnyCPU, I get the following output:
>>> 1500 repetitions >>> IN NANOSECONDS (1000ns = 0.001ms)
Method Avg. Min. Max. Jitter Total
A 1,301,465ns 1,232,200ns 2,869,000ns 1,567,534ns ! 1952.199ms
B 130,053ns 116,000ns 711,200ns 581,146ns ! 195.081ms
As you can see, the method with the await in it is about 10 times slower.
The thing is, as far as I know, there is nothing 'to' await - GetResult is always true. Does this mean that the state machine is executed even if the awaited 'thing' is already ready?
If so, is there any way around this? I'd like to use the semantics of async/await but this overhead is too high for my application...
EDIT: Adding full benchmark code after requested:
Program.cs
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.Diagnostics;
using System.Linq;
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace CSharpPerfTest {
public class Entity : INotifyCompletion {
private Action continuation;
private int i;
public void OnCompleted(Action continuation) {
this.continuation = continuation;
}
public Entity GetAwaiter() {
return this;
}
public Entity GetResult() {
return this;
}
public bool IsCompleted { get { return true; } }
public void Execute() {
if (i > 0) Console.WriteLine("What");
}
}
static class Program {
static ConcurrentSet<Entity> set;
const int MAX_ELEMENTS = 50000;
// Called once before all testing begins
private static void OnceBefore() {
set = new ConcurrentSet<Entity>();
Parallel.For(0, MAX_ELEMENTS, i => {
set.Add(new Entity());
});
}
// Called twice each repetition, once before DoTestA and once before DoTestB
private static void PreTest() {
}
private static void DoTestA() {
Entity[] objects = set.GetElements();
Parallel.For(0, objects.Length, async i => {
Entity e = objects[i];
if (e == null) return;
(await e).Execute();
});
}
private static void DoTestB() {
Entity[] objects = set.GetElements();
Parallel.For(0, objects.Length, i => {
Entity e = objects[i];
if (e == null) return;
e.Execute();
});
}
private const int REPETITIONS = 1500;
private const int JIT_WARMUPS = 10;
#region Test Harness
private static double[] aTimes = new double[REPETITIONS];
private static double[] bTimes = new double[REPETITIONS];
private static void Main(string[] args) {
Stopwatch stopwatch = new Stopwatch();
OnceBefore();
for (int i = JIT_WARMUPS * -1; i < REPETITIONS; ++i) {
Console.WriteLine("Starting repetition " + i);
PreTest();
stopwatch.Restart();
DoTestA();
stopwatch.Stop();
if (i >= 0) aTimes[i] = stopwatch.Elapsed.TotalMilliseconds;
PreTest();
stopwatch.Restart();
DoTestB();
stopwatch.Stop();
if (i >= 0) bTimes[i] = stopwatch.Elapsed.TotalMilliseconds;
}
DisplayScores();
}
private static void DisplayScores() {
Console.WriteLine();
Console.WriteLine();
bool inNanos = false;
if (aTimes.Average() < 10 || bTimes.Average() < 10) {
inNanos = true;
for (int i = 0; i < aTimes.Length; ++i) aTimes[i] *= 1000000;
for (int i = 0; i < bTimes.Length; ++i) bTimes[i] *= 1000000;
}
Console.WriteLine(">>> " + REPETITIONS + " repetitions >>> " + (inNanos ? "IN NANOSECONDS (1000ns = 0.001ms)" : "IN MILLISECONDS (1000ms = 1s)"));
Console.WriteLine("Method Avg. Min. Max. Jitter Total");
Console.WriteLine(
"A "
+ (String.Format("{0:N0}", (long) aTimes.Average()) + (inNanos ? "ns" : "ms")).PadRight(13, ' ')
+ (String.Format("{0:N0}", (long) aTimes.Min()) + (inNanos ? "ns" : "ms")).PadRight(13, ' ')
+ (String.Format("{0:N0}", (long) aTimes.Max()) + (inNanos ? "ns" : "ms")).PadRight(13, ' ')
+ (String.Format("{0:N0}", (long) Math.Max(aTimes.Average() - aTimes.Min(), aTimes.Max() - aTimes.Average())) + (inNanos ? "ns" : "ms")).PadRight(13, ' ')
+ ((long) aTimes.Sum() >= 10000 && inNanos ? "! " + String.Format("{0:f3}", aTimes.Sum() / 1000000) + "ms" : (long) aTimes.Sum() + (inNanos ? "ns" : "ms"))
);
Console.WriteLine(
"B "
+ (String.Format("{0:N0}", (long) bTimes.Average()) + (inNanos ? "ns" : "ms")).PadRight(13, ' ')
+ (String.Format("{0:N0}", (long) bTimes.Min()) + (inNanos ? "ns" : "ms")).PadRight(13, ' ')
+ (String.Format("{0:N0}", (long) bTimes.Max()) + (inNanos ? "ns" : "ms")).PadRight(13, ' ')
+ (String.Format("{0:N0}", (long) Math.Max(bTimes.Average() - bTimes.Min(), bTimes.Max() - bTimes.Average())) + (inNanos ? "ns" : "ms")).PadRight(13, ' ')
+ ((long) bTimes.Sum() >= 10000 && inNanos ? "! " + String.Format("{0:f3}", bTimes.Sum() / 1000000) + "ms" : (long) bTimes.Sum() + (inNanos ? "ns" : "ms"))
);
Console.ReadKey();
}
#endregion
}
}
If your function has a response time that 1ms for 50,000 calls is considered significant you should not be awaiting that code and instead be running it synchronously.
Using async code has a small overhead, it has to add a function calls for the state machine that drives it internally. If the work you are making async is also small in comparison to the overhead costs of running the state machine you should make the code you need to rethink if your code should be async.
Converted to an answer from the comments: apparently, this is not a clean benchmark test.
If you do not need asynchronous continuation, just don't use it. The code is always faster without it. If you do need it, then expect some overhead. You should understand what's going on behind the scene when you use a particular language/runtime feature.
Even if you remove Parallel.For, convert lambdas into methods and prevent inlining, there still will be some struct copying and await continuation closure allocation, to support the state machine functionality (an example of the generated code).
A more fair benchmark would be to test async/await vs. alternative implementation using callback closures and Task.ContinueWith, on a thread without synchronization context. I would not expect any significant difference in this case.
On a side note, you're passing an async void lambda Action into Parallel.For. You should be aware that the execution control will return to Parallel.For as soon as there's the 1st await inside the lambda, and then it's essentially a fire-and-forget call outside Parallel.For. I really can't think of any useful scenarios for this.
My current code:
UInt32 example = Convert.ToUInt32(tb_address.Text, 16);
Problem is if I'm peeking into an address like this:
0x83e3ba3c + 0x15
toUint32 will obviously return an error and say no.
Anyway I could go about having operators handled in it?
Picture of my app to maybe further understand it
Just split the string by using the '+'. After that trim the results, parse them and add them together.
Probably your best option would be to hide the code in a static class. something like this should work:
public static class HexAdder
{
private static UInt32 num1 = 0, num2 = 0;
private static void ParseString(string stringtoadd)
{
string[] temp = {"",""};
try
{
temp = stringtoadd.Split(" +".ToCharArray(), StringSplitOptions.RemoveEmptyEntries);
num1 = Convert.ToUInt32(temp[0], 16);
num2 = Convert.ToUInt32(temp[1], 16);
}
catch(Exception e)
{
MessageBox.Show("Invalid String Format\n- Added String = " + stringtoadd + "\n- First Part = " + temp[0] + "\n- Second Part = " + temp[1]);
}
}
public static UInt32 AddedToUint32(string stringtoadd)
{
ParseString(stringtoadd);
return num1 + num2;
}
}
This has basic validation and pops up a messagebox and returns 0 if there is an error.
Using it would look like this:
string test = "0x83e3ba3c + 0x15";
UInt32 example = HexAdder.AddedToUint32(test);
This way adding other operations is a simple matter of adding the appropriate static method(s).