I have a method that is accessed by many Tasks concurrently. I am reporting progress like this:
Console.WriteLine("Processed: " + Interlocked.Increment(ref progress).ToString());
Console.SetCursorPosition(0, Console.CursorTop - 1);
I reset the Console cursor position so next time it reports it overwrites the last number so I can have nice output on the screen.
I end up getting problem where it seems a thread reads new position wrong and it jumps up an extra line and messes up my other output information in the console.
I am not sure what I need to lock in order to lock the console so from time of reading to time of updating it cannot be read again and mess up the output (if that is actually the problem, I am not sure).
I'd like to be able to fully control what line it's outputting on so need to work out way to stop multiple threads messing up the position.
You can lock anything. Just create some object if there's no other object you think you could use, and then lock that.
var lck = new object();
void method()
{
lock(lck)
{
Console.WriteLine...
Console.SetCursorPosition...
}
}
Related
I have seen a few other posts very similar to this one, but the answers they give are not correctly answering the question. Sorry if there is something hidden away that I couldnt find...
I want to use Console.WriteLine() to print something above my current Console.ReadLine(), for example, my app prints "Hello world" and starts a thread that (in 5 seconds) will print "I just waited 5 seconds" above the line where I need to input something, like this:
Hello world
Please input something: _
Then 5 seconds will pass and it will look like this:
Hello world
I just waited 5 seconds
Please input something: _
So far I've tried using Console.SetCursorPosition(0,Console.CursorTop - 1) but this just prints over the line "Please input something: _" and if I use Console.CursorTop - 2 instead it crashes saying "[2] Out of range" (no idea why this is) and if I use Console.CursorTop - 2 it prints under "Please input something: _"... so my question is how do I print something ABOVE the line "Please input something: _"
Just moving the cursor is not good enough, the problem is that you are inserting text. That is possible, the Console.MoveBufferArea() method gives you access to the underlying screen buffer of the console and lets you move text and attributes to another line.
There are a couple of tricky corner-cases. One you already found, you have to force the console to scroll if the cursor is located at the end of the buffer. And the timer is a very difficult problem to solve, you can only really do this correctly if you can prevent Console.ReadLine() from moving the cursor at the exact same time that the timer's Elapsed event inserts the text. That requires a lock, you cannot insert a lock in Console.ReadLine().
Some sample code you can play with to get you there:
static string TimedReadline(string prompt, int seconds) {
int y = Console.CursorTop;
// Force a scroll if we're at the end of the buffer
if (y == Console.BufferHeight - 1) {
Console.WriteLine();
Console.SetCursorPosition(0, --y);
}
// Setup the timer
using (var tmr = new System.Timers.Timer(1000 * seconds)) {
tmr.AutoReset = false;
tmr.Elapsed += (s, e) => {
if (Console.CursorTop != y) return;
int x = Cursor.Left;
Console.MoveBufferArea(0, y, Console.WindowWidth, 1, 0, y + 1);
Console.SetCursorPosition(0, y);
Console.Write("I just waited {0} seconds", seconds);
Console.SetCursorPosition(x, y + 1);
};
tmr.Enabled = true;
// Write the prompt and obtain the user's input
Console.Write(prompt);
return Console.ReadLine();
}
}
Sample usage:
static void Main(string[] args) {
for (int ix = 0; ix < Console.BufferHeight; ++ix) Console.WriteLine("Hello world");
var input = TimedReadline("Please input something: ", 2);
}
Note the test on the Console.Top property, it ensures that nothing goes drastically wrong when the user typed too much text and forced a scroll or if Console.ReadLine() completed at the exact same time that the timer ticked. Proving that it is thread-safe in all possible cases is hard to do, there will surely be trouble when Console.ReadLine() moves the cursor horizontally at the exact same time that the Elapsed event handler runs. I recommend you write your own Console.ReadLine() method so you can insert the lock and feel confident it is always safe.
You can use carriage-return (\r, or U+000D) to return the cursor to the start of the current line, and then overwrite what's there. Something like
// A bunch of spaces to clear the previous output
Console.Write("\r ");
Console.WriteLine("\rI just waited 5 seconds");
Console.Write("Please input something: ");
However, if the user has started typing already this won't work anymore (as you may not overwrite all they have typed, and they will lose what they've typed on the screen, although it's still there in memory.
To properly solve this you actually need to modify the console's character buffer. You have to move everything above the current line one line up, and then insert your message. You can use Console.MoveBufferArea to move an area up. Then you need to save the current cursor position, move the cursor to the start of the line above, write your message, and reset the cursor position again to the saved one.
And then you have to hope that the user doesn't type while you're writing your message, because that would mess things up. I'm not sure you can solve that while using ReadLine, though, as you cannot temporarily lock something while ReadLine is active. To properly solve that you may have to write your own ReadLine alternative that reads individual keypresses and will lock on a common object when writing the resulting character to avoid having two threads writing to the console at the same time.
I am trying to serialize a thread (or Process) to a file and execute the thread on a different machine at some other time.
Actually what I have is something like this:
for (BigInteger i = 0; i < ABigIntegerVariable; i++)
{
// My Calculation
}
I want to suspend the computation and save its state, and resume it later with the saved state, possibly on a different machine.
note: I can't save data at the program closing cuz it contains object and It seems that it is note true to save an object
Thank you
Can't you just save your current loop iterator value and whatever calculation state is at the time you want to "move" it? It depends of course what exactly is happening inside that loop but maybe even crude serialization after each iteration would be enough for you to start at new location?
Of course your loop would have to start from saved data not from i = 0 but as I said, you didn't share any details about what is going on in //My Calculation so either put more details in question or try to figure it out on your own.
Also, as per comment from Sidewinder94, there's no problem with serialization of objects unless you are doing it wrong.
One additional thought: are those calculations dependent on your loop iterator or result of previous loop(s)? Because if not you could just split them into multiple threads/tasks and take advantage of parallel calculations.
I'm trying to process large amount of text files via Parallel.ForEach adding processed data to BlockingCollection.
The problem is that I want the Task taskWriteMergedFile to consume the collection and write them to result file at least every 800000 lines.
I guess that I can't test the collection size within the iteration because it is paralleled so I created the Task.
Can I convert while(true) loop in the task to EventWaitHandle in this case?
const int MAX_SIZE = 1000000;
static BlockingCollection<string> mergeData;
mergeData = new BlockingCollection<string>(new ConcurrentBag<string>(), MAX_SIZE);
string[] FilePaths = Directory.GetFiles("somepath");
var taskWriteMergedFile = new Task(() =>
{
while ( true )
{
if ( mergeData.Count > 800000)
{
String.Join(System.Environment.NewLine, mergeData.GetConsumingEnumerable());
//Write to file
}
Thread.Sleep(10000);
}
}, TaskCreationOptions.LongRunning);
taskWriteMergedFile.Start();
Parallel.ForEach(FilePaths, FilePath => AddToDataPool(FilePath));
mergeData.CompleteAdding();
You probably don't want to do it that way. Instead, have your task write each line to the file as it's received. If you want to limit the file size to 80,000 lines, then after the 80,000th line is written, close the current file and open a new one.
Come to think of it, what you have can't work because GetConsumingEnumerable() won't stop until the collection is marked as complete for adding. What would happen is that the thing would go through the sleep loop until there were 80,000 items in the queue, and then it would block on the String.Join until the main thread calls CompleteAdding. With enough data, you'd run out of memory.
Also, unless you have a very good reason, you shouldn't use ConcurrentBag here. Just use the default for BlockingCollection, which is ConcurrentQueue. ConcurrentBag is a rather special purpose data structure that won't perform as well as ConcurrentQueue.
So your task becomes:
var taskWriteMergedFile = new Task(() =>
{
int recordCount = 0;
foreach (var line in mergeData.GetConsumingEnumerable())
{
outputFile.WriteLine(line);
++recordCount;
if (recordCount == 80,000)
{
// If you want to do something after 80,000 lines, do it here
// and then reset the record count
recordCount = 0;
}
}
}, TaskCreationOptions.LongRunning);
That assumes, of course, that you've opened the output file somewhere else. It's probably better to open the output at the start of the task, and close it after the foreach has exited.
On another note, you probably don't want your producer loop to be parallel. You have:
Parallel.ForEach(FilePaths, FilePath => AddToDataPool(FilePath));
I don't know for sure what AddToDataPool is doing, but if it's reading a file and writing the data to the collection, you have a couple of problems. First, the disk drive can only do one thing at a time, so it ends up reading part of one file, then part of another, then part of another, etc. In order to read each chunk of the next file, it has to seek the head to the proper position. Disk head seeks are incredibly expensive--5 milliseconds or more. An eternity in CPU time. Unless you're doing heavy duty processing that takes much longer than reading the file, you're almost always better off processing one file at a time. Unless you can guarantee that the input files are on separate physical disks . . .
The second potential problem is that with multiple threads running, you can't guarantee the order in which things are written to the collection. That might not be a problem, of course, but if you expect all of the data from a single file to be grouped together in the output, that's not going to happen with multiple threads each writing multiple lines to the collection.
Just something to keep in mind.
I have created a Windows Form application that reads in a text file, rearranges the data, and writes to a new text file. I have noticed that it slows down exponentially as it runs. I have been using tracepoints, stopwatches, and datetime to figure out why each iteration is taking longer than the previous, but I can't figure it out. My best guess would be that it might have something to do with the way I'm initializing variables.
I'm not sure how helpful this snippet of code will be but maybe it will give some insight into my problem:
while (cuttedWords.Any())
{
var variable = cuttedWords.TakeWhile(x => x != separator).ToArray();
cuttedWords = cuttedWords.Skip(variable.Length + 1);
sortDataObject.SortDataMethod(variable, b);
if (sortDataObject.virtualPara)
{
if (!virtualParaUsed)
{
listOfNames = sortDataObject.findListOfNames(backgroundWords, ref IDforCounting, countParametersTable);
}
virtualParaUsed = true;
printDataObject.WriteFileVirtual(fileName, ID, sortDataObject.listNames[0], sortDataObject.listNames[1],
sortDataObject.unit, listOfNames, sortDataObject.virtualNames);
sortDataObject.virtualNames.Clear();
}
else
{
int[] indexes = checkedListBox1.CheckedIndices.Cast<int>().ToArray();
printDataObject.WriteFile(fileName, ID, sortDataObject.listNames[0], sortDataObject.listNames[1],
sortDataObject.unit, sortDataObject.hexValue[0], sortDataObject.stringShift, sortDataObject.sign,
sortDataObject.SFBinary[0], sortDataObject.wordValue, sortDataObject.conversions, sortDataObject.stringData, indexes, sortDataObject.conType);
}
decimal sum = ((decimal)IDforCounting) / countParametersTable * 100;
int sum2 = (int)sum;
backgroundWorker1.ReportProgress(sum2);
ID++;
IDforCounting++;
b++;
}
What is strange to me is that I know that each loop runs in a matter of milliseconds, but from the start of one loop to the start of the next, the time keeps increasing.
I apologize if this is not enough information to analyze my issue, but I'm not sure what else I can provide without showing my entire solution.
Thank you.
EDIT: A better questions might be: what is a good way to analyze performance if stopwatches aren't doing the trick. I'd rather not have to download a profiler.
If its taking longer and longer, on each iteration, its probably related to the initial cuttedWords.any().
What type is cuttedWords? If its a database-backed enumerable, it will re-issue the sql statement on every iteration, which may or may not be what you want.
On the other hand, if this is a producer-consumer scenario, it may be that cuttedWords is locked by the producer, causing the consumer to be thread-locked while waiting for the producer to complete its action.
Also, the .reportProgress will cause the backgroundworker to raise an event on the thread that created it, potentially causing UI updates, so maybe try removing that line and see if it helps any. Then replace it with some code that only calls reportProgress if the progress has actually changed.
Update: The answers from Andrew and Conrad were both equally helpful. The easy fix for the timing issue fixed the problem, and caching the bigger object references instead of re-building them every time removed the source of the problem. Thanks for the input, guys.
I'm working with a c# .NET API and for some reason the following code executes what I feel is /extremely/ slowly.
This is the handler for a System.Timers.Timer that triggers its elapsed event every 5 seconds.
private static void TimerGo(object source, System.Timers.ElapsedEventArgs e)
{
tagList = reader.GetData(); // This is a collection of 10 objects.
storeData(tagList); // This calls the 'storeData' method below
}
And the storeData method:
private static void storeData(List<obj> tagList)
{
TimeSpan t = (DateTime.UtcNow - new DateTime(1970, 1, 1));
long timestamp = (long)t.TotalSeconds;
foreach (type object in tagList)
{
string file = #"path\to\file" + object.name + ".rrd";
RRD dbase = RRD.load(file);
// Update rrd with current time timestamp and data.
dbase.update(timestamp, new object[1] { tag.data });
}
}
Am I missing some glaring resource sink? The RRD stuff you see is from the NHawk C# wrapper for rrdtool; in this case I update 10 different files with it, but I see no reason why it should take so long.
When I say 'so long', I mean the timer was triggering a second time before the first update was done, so eventually "update 2" would happen before "update 1", which breaks things because "update 1" has a timestamp that's earlier than "update 2".
I increased the timer length to 10 seconds, and it ran for longer, but still eventually out-raced itself and tried to update a file with an earlier timestamp. What can I do differently to make this more efficient, because obviously I'm doing something drastically wrong...
Doesn't really answer your perf question but if you want to fix the rentrancy bit set your timer.AutoRest to false and then call start() at the end of the method e.g.
private static void TimerGo(object source, System.Timers.ElapsedEventArgs e)
{
tagList = reader.GetData(); // This is a collection of 10 objects.
storeData(tagList); // This calls the 'storeData' method below
timer.Start();
}
Is there a different RRD file for each tag in your tagList? In your pseudo code you open each file N number of times. (You stated there is only 10 objects in the list thought.) Then you perform an update. I can only assume that you dispose your RRD file after you have updated it. If you do not you are keeping references to an open file.
If the RRD is the same but you are just putting different types of plot data into a single file then you only need to keep it open for as long as you want exclusive write access to it.
Without profiling the code you have a few options (I recommend profiling btw)
Keep the RRD files open
Cache the opened files to prevent you from having to open, write close every 5 seconds for each file. Just cache the 10 opened file references and write to them every 5 seconds.
Separate the data collection from data writing
It appears you are taking metric samples from some object every 5 seconds. If you do not having something 'tailing' your file, separate the collection from the writing. Take your data sample and throw it into a queue to be processed. The processor will dequeue each tagList and write it as fast as it can, going back for more lists from the queue.
This way you can always be sure you are getting ~5 second samples even if the writing mechanism is slowed down.
Use a profiler. JetBrains is my personal recommendation. Run the profiler with your program and look for the threads / methods taking the longest time to run. This sounds very much like an IO or data issue, but that's not immediately obvious from your example code.