Update: The answers from Andrew and Conrad were both equally helpful. The easy fix for the timing issue fixed the problem, and caching the bigger object references instead of re-building them every time removed the source of the problem. Thanks for the input, guys.
I'm working with a c# .NET API and for some reason the following code executes what I feel is /extremely/ slowly.
This is the handler for a System.Timers.Timer that triggers its elapsed event every 5 seconds.
private static void TimerGo(object source, System.Timers.ElapsedEventArgs e)
{
tagList = reader.GetData(); // This is a collection of 10 objects.
storeData(tagList); // This calls the 'storeData' method below
}
And the storeData method:
private static void storeData(List<obj> tagList)
{
TimeSpan t = (DateTime.UtcNow - new DateTime(1970, 1, 1));
long timestamp = (long)t.TotalSeconds;
foreach (type object in tagList)
{
string file = #"path\to\file" + object.name + ".rrd";
RRD dbase = RRD.load(file);
// Update rrd with current time timestamp and data.
dbase.update(timestamp, new object[1] { tag.data });
}
}
Am I missing some glaring resource sink? The RRD stuff you see is from the NHawk C# wrapper for rrdtool; in this case I update 10 different files with it, but I see no reason why it should take so long.
When I say 'so long', I mean the timer was triggering a second time before the first update was done, so eventually "update 2" would happen before "update 1", which breaks things because "update 1" has a timestamp that's earlier than "update 2".
I increased the timer length to 10 seconds, and it ran for longer, but still eventually out-raced itself and tried to update a file with an earlier timestamp. What can I do differently to make this more efficient, because obviously I'm doing something drastically wrong...
Doesn't really answer your perf question but if you want to fix the rentrancy bit set your timer.AutoRest to false and then call start() at the end of the method e.g.
private static void TimerGo(object source, System.Timers.ElapsedEventArgs e)
{
tagList = reader.GetData(); // This is a collection of 10 objects.
storeData(tagList); // This calls the 'storeData' method below
timer.Start();
}
Is there a different RRD file for each tag in your tagList? In your pseudo code you open each file N number of times. (You stated there is only 10 objects in the list thought.) Then you perform an update. I can only assume that you dispose your RRD file after you have updated it. If you do not you are keeping references to an open file.
If the RRD is the same but you are just putting different types of plot data into a single file then you only need to keep it open for as long as you want exclusive write access to it.
Without profiling the code you have a few options (I recommend profiling btw)
Keep the RRD files open
Cache the opened files to prevent you from having to open, write close every 5 seconds for each file. Just cache the 10 opened file references and write to them every 5 seconds.
Separate the data collection from data writing
It appears you are taking metric samples from some object every 5 seconds. If you do not having something 'tailing' your file, separate the collection from the writing. Take your data sample and throw it into a queue to be processed. The processor will dequeue each tagList and write it as fast as it can, going back for more lists from the queue.
This way you can always be sure you are getting ~5 second samples even if the writing mechanism is slowed down.
Use a profiler. JetBrains is my personal recommendation. Run the profiler with your program and look for the threads / methods taking the longest time to run. This sounds very much like an IO or data issue, but that's not immediately obvious from your example code.
Related
For testing reasons I want to be able to adjust what time Quartz.Net currently thinks it is so I do not necessarily have to wait hours, days, or weeks in order to check that my code is working.
For this purpose I created the following simple function (it is in F# but could be easily be done in C# or another language) :
let SimulateTime = fun () ->
currentTime <- DateTimeOffset.UtcNow
timeDifferenceInSeconds <- (currentTime - lastCheckedTime).TotalSeconds
simulatedTime <- simulatedTime.AddSeconds((timeDifferenceInSeconds *scaleTimeBy))
lastCheckedTime <- currentTime
simulatedTime
Where currentTime, lastCheckedTime, and simulatedTime would all be of type DateTimeOffset and both timeDifferenceInSeconds and scaleTimeBy are of type float.
I then change SystemTime.Now and SystemTime.UtcNow to use the above function as follows :
SystemTime.Now <-
Func<DateTimeOffset>(
fun () -> SimulateTime())
SystemTime.UtcNow <-
Func<DateTimeOffset>(
fun () -> SimulateTime())
Which was shown by Mark Seemann in a previous question of mine that can find here.
Now this mostly works except it seems like the longer function causes it to be off by a decently wide margin. What I mean by this is that all of my triggers will misfire. For example if I have a trigger set to occur every hour and set scaleTimeBy to 60.0 so that every second passed counts as a minute, it will never actually trigger on time. If I have a misfire policy, the trigger can then go off but the time it lists for when it activated will be as late as the half hour mark (so takes a full 30 seconds slower than what it should have been in this example).
However I can do this :
Console.WriteLine(SimulateTime())
Thread.Sleep(TimeSpan.FromSeconds(60.0))
Console.WriteLine(SimulateTime())
And the difference between the two times output to the screen in this example will be exactly an hour so the call doesn't seem like it should be adding as much of a time difference than it does.
Anyone have any advice on how to fix this issue or a better way of handling this problem?
Edit :
So the C# version of the SimulateTime function would be something like this :
public DateTimeOffset SimulateTime() {
currentTime = DateTimeOffset.UtcNow;
double timeDifference = (currentTime - lastCheckedTime).TotalSeconds;
simulatedTime = simulatedTime.AddSeconds(timeDifference * scaleTimeBy);
lastCheckedTime = currentTime
return simulatedTime;}
If that helps anyone with solving this problem.
So this issue is misfires caused by the fact that Quartz.net will idle and wait when it thinks it doesn't have any triggers occurring any time soon to avoid making too many calls. By default it waits about 30 seconds give or take if it doesn't have any triggers occurring in the time span. The idleWaitTime variable is a Timespan set in the QuartzSchedulerThread. Now when checking for triggers that might occur soon it also uses the BatchTimeWIndow from QuartzSchedulerResources.
Both idleWaitTime and BatchTimeWindow can be set in configuration/properties files where they'd be called "org.quartz.scheduler.idleWaitTime" and "org.quartz.scheduler.batchTriggerAcquisitionFireAheadTimeWindow."
Based off what it is called in BatchTimeWindow I thought it was just a bit of look ahead for grabbing a variable (which would like since if I'm speeding things up, I'd want a small idleWaitTime but I would want it to look further ahead for triggers because the few seconds your waiting is actually minutes so will trigger sooner than it thinks), but the description of "org.quartz.scheduler.batchTriggerAcquisitionFireAheadTimeWindow" on pages going over configuration properties implies that it can cause things to fire early and be less accurate. So to start here is the code for just modifying idleWaitTime
let threadpool = Quartz.Simpl.SimpleThreadPool()
let jobstore = Quartz.Simpl.RAMJobStore()
let idleWaitTime = TimeSpan.FromSeconds(30.0/scaleTimeBy)
let dbfailureretryinverval = TimeSpan(int64 15000)
Quartz.Impl.DirectSchedulerFactory.Instance.CreateScheduler("TestScheduler","TestInstance",threadpool,jobstore,idleWaitTime,dbfailureretryinverval)
let scheduler = Quartz.Impl.DirectSchedulerFactory.Instance.GetScheduler("TestScheduler")
You can create a Scheduler that has the idleWaitTime you want by using the DirectSchedulerFactory which probably could use a little bit better documentation. It takes also a bunch of stuff you may or may not want to modify depending on what you are working on. For threadpool I just use Quartz.net's default SimpleThreadPool because I do not care about messing with the threading at this time and would not want to explain how you go about doing so unless that was the whole point of the question. Information on jobstores is available here. I am using RAMJobStore here because it is simpler than AdoJobStore but it shouldn't matter for this example. The dbfailureretryinterval is another value that don't care about for this example so I just looked up what it is set to by default. Its value should matter the least for this example because not connecting to a database. For idleWaitTime might want to do more tests to figure out what is a good value for it, but I chose to go with just scaling its default value of 30 seconds by scaleTimeBy since that is what I'm using to scale how fast things are going by. So this should make it so if I am having the program simulate time going by at a much faster rate, then it should only remain idle for smaller periods of time. One important thing to note is that when create a scheduler in this way, it is not returned as well so need to make a separate call to get the scheduler I just created. I have no idea why this is this way, I'm guessing that if you are creating several Schedulers and not necessarily using all of them it is better this way.
Now after all that you are likely to still get a bit of a misfire rate. While it is now idling for much smaller units of time (only a few seconds so potentially an acceptable margin depending on what your use case is), it still has the issue of it is only then checking to see if it has a coming trigger in the next few fractions of a second.
So lets see if adding time to BatchTimeWindow helps matters?
let threadpool = Quartz.Simpl.SimpleThreadPool()
let threadexecutor = Quartz.Impl.DefaultThreadExecutor()
let jobstore = Quartz.Simpl.RAMJobStore()
let schedulepluginmap = System.Collections.Generic.Dictionary<String,Quartz.Spi.ISchedulerPlugin>()
let idleWaitTime = TimeSpan.FromSeconds(30.0/timeScale)
let maxBatchSize = 1
let batchTimeWindow = TimeSpan.FromSeconds(timeScale)
let scheduleexporter = Quartz.Simpl.RemotingSchedulerExporter()
Quartz.Impl.DirectSchedulerFactory.Instance.CreateScheduler("TestScheduler","TestInstance",threadpool,threadexecutor,jobstore,schedulepluginmap,idleWaitTime,maxBatchSize,batchTimeWindow,scheduleexporter)
let scheduler = Quartz.Impl.DirectSchedulerFactory.Instance.GetScheduler("TestScheduler")
Now this has even more variables that don't really care about for the purposes of this example and won't even bother going over because adjusting batchTimeWindow actually makes it worse. Like getting you back to misfiring by 30 minutes. So no, batchTimeWindow while looks like might be useful is not. Only modify idleWaitTime.
Ideally for this use would want a small wait time and a larger look ahead time, but the option for that does not seem like its available.
I am trying to create a script which does something say show a popup after 1 day. As far as I know I can do this using Thread.Stop or using System.Timers or many other ways. But the problem with each is that the computer has to be continuously running for each of these methods to give desired result at the time I want. Basically I want the script to start with installation of my program, wait for exactly 24 hours, then display a message box. If the computer is switched on at that time i.e. after 24 hours it should show the message box on next start up but only if 24 hours or more have passed. Please help, i am unable to find suitable solution for this. any help will be highly appreciated. I think that it may be achieved by getting the dateTime.Now and putting it in a file, and then compare current system time to the time in file every hour or so, and if 24 hours or more have passed, show the message box. Please help
If a message box is what you need to pop up, then you'll need a Windows Forms application. Here is the workflow of how it'll work:
Application will need to save data, either through a database or to a file.
When the application first loads, it will check either the database or the file for a TriggerDate entry.
If the TriggerDate exists, pull this date and compare it to today's date and time. If the current date and time is past our TriggerDate, display the message box. Recreate the TriggerDate with the current date and time plus 24 hours.
If TriggerDate does not exist, create it, filled with the current date and time plus 24 hours (e.g. DateTime.Now.AddHours(24)).
Use Threading.Thread.Sleep() to sleep for 5 minutes.
Jump to step 3 and repeat.
EDIT
Code will be something like this:
private const string TriggerFile = #"C:\TriggerData\trigger.txt";
private DateTime _triggerDate;
if (!File.Exists(TriggerFile))
{
using (StreamWriter sw = File.CreateText(TriggerFile))
{
sw.WriteLine(DateTime.Now.AddHours(24));
}
}
using (StreamReader sr = File.OpenText(TriggerFile))
{
_triggerDate = DateTime.Parse(sr.ReadToEnd());
}
while (true)
{
if (DateTime.Now >= _triggerDate)
{
MessageBox.Show(#"Alert!");
using (StreamWriter sw = File.CreateText(TriggerFile))
{
sw.WriteLine(DateTime.Now.AddHours(24));
_triggerDate = DateTime.Now.AddHours(24);
}
}
System.Threading.Thread.Sleep(60000*5); // Sleep for 5 minutes
}
You may not want to do while(true). You way want to implement a way to get out of the program.
If I understand correctly, what you can use is a way to save the "start date". You can save the start date of the timer and read the value from your script, that way you can calculate if 24 hours or more has passed since the moment you assigned a value to the start date. You can use a simply TXT file to save the value.
I see two ways of achieving this :
Easy way (but not strictly related to C#) is to use the task scheduler and create a task that will run your C# program every 24 hours. One way to do this from an installer would be to use the SchTasks.exe tool. This method allows you to just exit the program after your work is done without having to explicitely wait for the next run.
Harder way: Create a Windows Service, which is a program specifically designed to run permanently on the host computer. These programs are slightly more technical to write though. You can find more explanations about services here. However, you may not want to take this route as service cannot display user interfaces like popups.
I have created a Windows Form application that reads in a text file, rearranges the data, and writes to a new text file. I have noticed that it slows down exponentially as it runs. I have been using tracepoints, stopwatches, and datetime to figure out why each iteration is taking longer than the previous, but I can't figure it out. My best guess would be that it might have something to do with the way I'm initializing variables.
I'm not sure how helpful this snippet of code will be but maybe it will give some insight into my problem:
while (cuttedWords.Any())
{
var variable = cuttedWords.TakeWhile(x => x != separator).ToArray();
cuttedWords = cuttedWords.Skip(variable.Length + 1);
sortDataObject.SortDataMethod(variable, b);
if (sortDataObject.virtualPara)
{
if (!virtualParaUsed)
{
listOfNames = sortDataObject.findListOfNames(backgroundWords, ref IDforCounting, countParametersTable);
}
virtualParaUsed = true;
printDataObject.WriteFileVirtual(fileName, ID, sortDataObject.listNames[0], sortDataObject.listNames[1],
sortDataObject.unit, listOfNames, sortDataObject.virtualNames);
sortDataObject.virtualNames.Clear();
}
else
{
int[] indexes = checkedListBox1.CheckedIndices.Cast<int>().ToArray();
printDataObject.WriteFile(fileName, ID, sortDataObject.listNames[0], sortDataObject.listNames[1],
sortDataObject.unit, sortDataObject.hexValue[0], sortDataObject.stringShift, sortDataObject.sign,
sortDataObject.SFBinary[0], sortDataObject.wordValue, sortDataObject.conversions, sortDataObject.stringData, indexes, sortDataObject.conType);
}
decimal sum = ((decimal)IDforCounting) / countParametersTable * 100;
int sum2 = (int)sum;
backgroundWorker1.ReportProgress(sum2);
ID++;
IDforCounting++;
b++;
}
What is strange to me is that I know that each loop runs in a matter of milliseconds, but from the start of one loop to the start of the next, the time keeps increasing.
I apologize if this is not enough information to analyze my issue, but I'm not sure what else I can provide without showing my entire solution.
Thank you.
EDIT: A better questions might be: what is a good way to analyze performance if stopwatches aren't doing the trick. I'd rather not have to download a profiler.
If its taking longer and longer, on each iteration, its probably related to the initial cuttedWords.any().
What type is cuttedWords? If its a database-backed enumerable, it will re-issue the sql statement on every iteration, which may or may not be what you want.
On the other hand, if this is a producer-consumer scenario, it may be that cuttedWords is locked by the producer, causing the consumer to be thread-locked while waiting for the producer to complete its action.
Also, the .reportProgress will cause the backgroundworker to raise an event on the thread that created it, potentially causing UI updates, so maybe try removing that line and see if it helps any. Then replace it with some code that only calls reportProgress if the progress has actually changed.
I explain my situation.
I have a producer 1 to N consumers pattern. I'm using blocking collections and everything is working well. Doing some test I noticed this strange behavior:
I was testing how long my manipulation of data took in my consumers.
I noticed this strange things, below you'll find the code cleaned of my manipulation and which produce the strange behavior.
I have 4 consumers for 1 producer.
For most of data, the Console doesn't print anything, because ts=0 (its under a tick) but randomly (between every 1 to 5sec) it plots something like this (not in this very specific order, but of the same kind):
10000
20001
10000
30002
10000
40003
10000
10000
It is of the order of 10,000 ticks so around 1ms. Always a number in the format (N)000(N-1)
Note that the BlockingCollection I consume is filled depending on some network events which occurred completely at random times. Nothing regular from here.
The timing is almost perfect, always a multiple of 10,000 ticks.
What could be behind this ? Thks !
while(IsAlive)
{
DataToFieldMapping item;
try
{
_CollectionToConsume.TryTake(out item, -1);
}
catch
{
item = null;
}
if (item != null)
{
long ts = (DateTime.Now.Ticks - item.TimeStamp.Ticks);
if(ts>10)
Console.WriteLine(ts);
}
}
What's going on here is that DateTime.Now has a fairly limited precision. It's not giving you the time to the nearest tick. It is only updated every 10,000 ticks or so, which is why you generally see multiples of 10k ticks in your prints.
If you really want to get a better feel for the duration of those events, use the StopWatch class, which has a much higher precision. That said, StopWatch is simply a diagnostic tool (hence why it's in the Diagnostics namespace). You should only be using it to help you diagnose what's going on, and should be using it in production code.
On a side note, there really isn't any need to use a timer here at all. It appears that you're creating several consumers that are polling the BlockingCollection for new content. There is no reason to do this. They can simply block until the collection has items. (Hence the name, BlockingCollection.
The easiest way is for the consumers to simply do this:
foreach(var item in _CollectionToConsume.GetConsumingEnumerable())
ProcessItem(item);
Then just run that code in a background thread.
if you write the following and run, you'll see that ticks do not roll one to one, but rather in relatively large chunks b/c ticks resolution is actually much smaller.
for(int i =0; i< 100; i++)
{
Console.WriteLine(DateTime.Now.Ticks);
}
Use Stopwatch class to measure performance as that one uses a high-resolution timer which is much more suitable for the purpose.
I have a multithreaded (using a threadpool) C# program that reads from a text file containing logs and batch inserts them into a MongoDB collection. I want a consistent and precise way to measure how long it takes to insert the whole file into the collection.
I can't really call a thread.join (because it's a threadpool), and I can't use a stopwatch because they're running on separate threads.
What's the next best thing?
The current way I'm doing it is the timer on my smartphone. I repeatedly call db.collection.stats() and wait till the count is the same as the number of logs in the file...
If you're using C# 4.0+, I'd recommend you use the CountdownEvent class. Using that class, you can just create an instance with the number of logs for example as the counter:
var countdown = new CountdownEvent(numberOfLogs);
Then, each time you complete a write to MongoDB, you signal from the worker thread:
countdown.Signal(); // decrement counter
And then, in your main process (or another thread):
countdown.Wait(); // returns when the count is zero
// All writes complete
With mongostat (command line tool) you see exactly what goes on in the MongoDB server. It will give you inserts/queries etc per second. It won't automatically stop when it's done inserting, but it will definitely give you an insight to performance. The "inserts" will drop to 0 once you're done importing.