We Keep Coding

KafkaConsumer.PlainSource Method (in C#.Net) Using large amount of CPU

below method using large amount of CPU can anyone help me to minimize the CPU usage with proper solution.
KafkaConsumer.PlainSource(
consumerSettings, subscription)
.RunForeach(result =>
{
_ActorRef.Tell(result.Message.Value);
}, materializer);

I'm running the SimpleProducer and SimpleConsumer samples baked into the Akka.Streams.Kafka repository - and the PlainSource is designed in a nearly identical fashion to yours:
KafkaConsumer.PlainSource(consumerSettings, subscription)
.RunForeach(result =>
{
Console.WriteLine($"Consumer: {result.Topic}/{result.Partition} {result.Offset}: {result.Message.Value}");
}, materializer);
Here's what my CPU utilization looks like - bearing in mind that the producer is continuously producing new events for my consumer:
This is extremely low resource consumption - which is what Akka.Streams and all of its plugins (such as Kafka) provide out of the box.
Your setup has no backpressure support (since IActorRef.Tell is non-blocking) and therefore this stream is going to run at full blast inside your system. Whatever your actors are doing is probably what's responsible for high CPU utilization.
Your other ticket is asking about how to add backpressure support to your Akka.Streams.Kafka application, so I'll help answer that too.

Why is my application becoming less responsive over time?

I'm debugging a C# application that becomes almost unresponsive after a few days. The application calculates memory/CPU usage every second and displays it in the footer of the main UI.
The cause for the unresponsiveness is because of the time it takes to fetch the RawValue of a PerformanceCounter ("Working Set - Private"). After a couple of days, it takes almost a second to fetch the RawValue, freezing the main UI thread. If I restart my computer, everything is fast again for a few days until it slowly becomes less responsive. If I recompile this application without the PerformanceCounter code (it's open source), it runs normally immediately.
To rule out that it's the application, here's some sample code that does the exact same thing:
static void Main(string[] args)
{
using (var memoryWorkingSetCounter = new PerformanceCounter("Process", "Working Set - Private", Process.GetCurrentProcess().ProcessName, true))
{
while (!Console.KeyAvailable)
{
var memoryWorkingSetSw = new Stopwatch();
memoryWorkingSetSw.Start();
var memoryWorkingSetValue = memoryWorkingSetCounter.RawValue;
memoryWorkingSetSw.Stop();
Console.WriteLine(memoryWorkingSetValue.ToString());
Console.WriteLine(memoryWorkingSetSw.Elapsed.ToString());
Thread.Sleep(TimeSpan.FromSeconds(1));
}
}
Console.Read();
}
I left this running for ~2.5 days and graphed the Elapsed time in milliseconds:
What could cause a performance counter in Windows to become slow over time? Could another app be not cleaning it up? Is there a way to debug which apps are also looking at this performance counter? I'm on Windows 10.

why you declare Stopwatch inside a loop?
remove any new declaration possible inside loops.
when you do so the memory is increasing overtime and you count on garbage collector to do the work

Get application pool's memory usage

I want to get my application pool's memory usage using Process class. This is my code so far:
ServerManager serverManager = new ServerManager();
ApplicationPoolCollection applicationPoolCollection = serverManager.ApplicationPools;
Process process = Process.GetProcessById(applicationPoolCollection.Where(p => p.Name == "mywebsite.com").First().WorkerProcesses[0].ProcessId);
I can get Process class but how do I know what is the amount of RAM it is using currently? I do see properties like PagedMemorySize64 and NonpagedSystemMemorySize64. Which one would give me the correct RAM that it is occupying at the moment?
I cannot use PerformanceCounter like so:
PerformanceCounter performanceCounter = new PerformanceCounter();
performanceCounter.CategoryName = "Process";
performanceCounter.CounterName = "Working Set";
performanceCounter.InstanceName = process.ProcessName;
Console.WriteLine(((uint)performanceCounter.NextValue() / 1024).ToString("N0"));
The reason being ProcessName would return w3p and therefore PerformanceCounter would write the total IIS RAM usage in Console which is not what I want.
I just need the application pool's RAM usage. Therefore I think my answer lies in Process class and not PerformanceCounter
Any help is appreciated.

If you need simply amount of ram visible to the process, which means amount of ram assigned to the process, but not necessary that every single bit of it is actually used by the process, you can use: WorkingSet. (Choose appropriate version for your .net)
Just invite your attention on the fact, that there is much more into process memory diagnostics, in case you might be concerned about allocated Virtual Memory, Commited Memory, Mapped Files and whatnot.

Extreme Memory Conditions Testing : How to saturate RAM?

I would like to write a small piece of program that launches threads, consumes available RAM memory in a linear fashion, until a certain level, and stops (ideally, pauses until "enough" memory is freed and continues creating threads after that, and so on.)
I tried the following, but the list.Add(new byte[]) requires contiguous RAM space and drops an OutOfMemoryException, which is NOT what I am trying to simulate.
EDIT :
I have a multi-threaded memory-hungry application that eats up a whole bunch of RAM GB's. All I want is to isolate/reproduce that situation in "Lab conditions" to tackle it, i.e write an adaptive mem-monitoring / thread-limiter draft. I am using x64 OS an x64 Platform.
To make it clear : The result I want to see is the Task Manager Memory Monitor going up straight due to the program.
static void Main(string[] args)
{
ComputerInfo ci = new ComputerInfo();
D("TOTAL PHYSICAL MEMORY : " + Math.Round(ci.TotalPhysicalMemory / Math.Pow(10,9),3) +" GB");
//########### Fill Memory ###############
var list = new List<byte[]>();
Thread FillMem= new Thread(delegate()
{
while (Process.GetCurrentProcess().PrivateMemorySize64 < MAX_MEM_LEVEL)
{
list.Add(new byte[1024 * 10000]); //<- I Need to change this
Thread.Sleep(100);
}
});
FillMem.Start();
//########### Show used Memory ###############
Thread MonitorMem = new Thread(delegate()
{
while (true)
{
D("PROCESS MEMORY : " + Math.Round(Process.GetCurrentProcess().PrivateMemorySize64 / Math.Pow(10, 6), 3) + " MB");
Thread.Sleep(1000);
}
});
MonitorMem.Start();
Console.Read();
}

The question is still quite confusing; it is not clear to me what you are trying to do here and why.
If you genuinely want to be consuming physical memory -- that is, telling the operating system no, really do not use this portion of the physical RAM chip that is installed in the machine for anything other than what I say -- then I would probably use the aptly-named AllocateUserPhysicalPages function from unmanaged code.
That will then reduce the amount of physical memory that is available for other uses, forcing more virtual memory pages to go out to the page file.
Other than making all the programs running on your machine a whole lot slower, I'm not sure what you intend to accomplish by this. Can you clarify?

The thing is that with C# you can not grow more then approximately 1.2 GB of RAM on 32 bit .NET framework. You can have even 8GB of RAM on your 64 bit machine, but if the process you run was compiled for 32bit architecture, it will lead to OutOfMemoryException as soon as it reaches approx 1.2GB.
For this kind of testing I would suggest choosing other type of languages/frameworks.
EDIT
A good link on subject:
is-there-a-memory-limit-for-a-single-net-process

If the problem that you're running into is that your process is running out of virtual memory space before the hardware is running out of physical memory space then you could just spin up a number (5 maybe?) processing with your code (and something to stop them at say 1-2 GB so they don't OOM themselves). It's probably not as good of a solution as a unmanaged call to allocate memory, but it would be easy enough to do.

about using cpucounter and ramcounter

how to detect cpuusage and ram usage of a remote host using c#??

You could use the C# PerformanceCounter class to get this done. It's relatively simple to create a read-only counter:
PerformanceCounter cpuCounter = new PerformanceCounter(category, counter, instance, machine)
The documentation is available here: http://msdn.microsoft.com/en-us/library/9ffskxdc(v=VS.90).aspx
I'm not 100% sure that these are the exact values you want, but these may work:
CPU:
category = Processor
counter = % Processor Time
instance = _Total
RAM:
category = Memory
counter = Available MBytes
instance = (use String.Empty)
Performance Counters can be very tricky if you want your app to work on all modern MS OSs. Things get even more complicated if you want your app to run as regular (non-admin) user.
But that should get you started.
-- Dan