We Keep Coding

Faster way to search through a big string?

I am currently trying to make a program to find blocks of a specific color in a game save and move their position, however with some of the bigger saves my method of searching for the blocks in the save can start to take a bit. My current fastest method takes about 42 seconds to search for and move every block in a string about the size of 1MB. There are a lot of blocks in the save (Roughly one every 50-300 characters in the string, with a total of around 7k) so I'm not sure if string search algorithms would speed up or slow down this process.
So, I was wondering if I could get any tips of if anyone had any ideas on how to further speed up my code I would be very greatfull.
progressBar2.Maximum = blueprint.Length;
int i = 0;
while (i < blueprint.Length - 15)
{
progressBar2.Value = i;
try
{
if (!blueprint.Substring(i, 110).ToLower()
.Contains("\"color\""))
{
i += 100;
}
}
catch
{
return;
}
checkcolor(i, color, colortf, posset, axis);
i++;
}
I am currently optimizing the method checkcolor and it's the cause for most of the delay, but my current method runs it way more than needed.
I've tried adding a second if to skip at an interval of 10 as well as 100 but that caused it to take over 2 min, I've also tried different values to skip other then 100 but 100 seems to be the fastest.
Edit: I was making 2 new temporary strings just to check for a small bit of text millions of times, it's a lot faster to use .IndexOf which I did not know existed. Thanks for the help and sorry if this was off topic.

I would try to compare efficiency without creation substring and using ToLower():
if (!blueprint.IndexOf("\"color\"", StringComparison.OrdinalIgnoreCase) >= 0)

Calculating maximum power such that result is within limit

I'm currently working on an exponential falloff (http://www.acodersjourney.com/2016/02/26-handle-transient-errors-in-c/) implementation.
I'm calculating the delay to wait, such that:
var delay = (int) Math.Round(Math.Pow(timeBetweenAttempts, attempt), MidpointRounding.AwayFromZero);
Obviously, this delay starts to get very large, very quickly, even with the time between attempts being 10 milliseconds.
I would like to be able to do something akin:
var maxTimeBetweenAttempts = 5000; // 5 seconds is the hard limit
var nominalTimeBetweenAttempts = 10;
var maxNumberOfAttempts = // calculate the maximum number of raises that would hold below 5000.
Obviously this could be calculated using a loop, but I was wondering if there was a more elegant way to do this?

The inverse of exponentiation is logarithm so you can use that.
You would use
if (attempt < Math.Log(maxTimeBetweenAttempts) / Math.Log(nominalTimeBetweenAttempts))
{
Retry(++attempt);
}
To see if the attempt was below the threshold after rounding.
However, as this requires an extra calculation and is a little less obvious (as it uses logarithmic maths that most people haven't event seen since school) than just doing the exponential, I would personally recommend just calculating the power and testing that.
ie.
if(Math.Pow(nominalTimeBetweenAttempts, attempt) < maxTimeBetweenAttempts)
{
Retry(++attempt);
}
Edit:
Re-reading your question, you explicitly state that you want to know the maximum number of retries before they reach a certain length. That would be:
var maximumNumberOfAttempts = Math.Floor(Math.Log(maxTimeBetweenAttempts) / Math.Log(nominalTimeBetweenAttempts))

C# How to loop the maximum amount of times

When going through a really long array, or have a complicated calculations per each index, is there a way to yield after iterating through the array for the maximum amount of time. The maximum amount of time is the maximum time per each frame.
For example:
for(int i = 0; i < 100000; i++){
do something complicated;
if(maximum amount of time /*right before the user feels lag*/)
yield; (come back and resume i where it last yielded)
}
}
//order does not matter
So basically, what I want to achieve is high percent usage for the cpu, however, I do not want it to go beyond 100%, which the user will experience lag
edit:
Sorry for the little confusion. A more clear example might be 3d rendering in a program such as blender. When the user hits render, it calculates each pixels to determine what color it needs to be. When one looks at the cpu usage, it is close to 100%. however, it does not freeze while it calculates the pixels while it calculates the maximum amount as possible

If you are running your code on multiple CPUs (as implied by the multithreading tag), there should (in the usual case) be no need to stop executing the loop in order for your user interface to remain responsive. Perform the calculation on one or more background threads, and have those background threads update the UI thread as appropriate.
is there a way to yield after iterating through the array for the maximum amount of time
If by yield you mean just stop (and restart from the beginning next frame), then sure. You can pass a CancellationToken to your thread, and have it periodically check for a cancellation request. You can use a timer at the start of each frame to fire off that request, or more likely, use an existing mechanism that already does end-of-frame processing to trigger the thread to stop work.
If by yield you mean stop where I am and resume at that place at the start of the next frame, I would ask why stop given that you have multiple CPUs. If you must stop, you can use the CancellationToken as before, but just keep track of where you are in the loop, resuming from there instead of at the start.
So basically, what I want to achieve is high percent usage for the cpu, however, I do not want it to go beyond 100%, which the user will experience lag
You can never go over 100% CPU usage by definition. To avoid the feeling of lag when the CPU utilization is high, use thread priorities to ensure that the foreground thread has a higher priority than your background threads.

Unless I'm missing something....
double MAX_PROCESSTIME = 50.0;
DateTime loopStart = DateTime.Now();
for(int i = 0; i < 100000; i++){
// do something complicated;
double timePassed = (DateTime.Now() - loopStart).TotalMilliseconds;
if(timePassed > MAX_PROCESSTIME)
{
break;
}
}

How about you consider use a push model instead, to iterate in parallel and raising an event so the consumer just treat each item as they come?

Usually the solution to this problem is to move the work to a separate thread that can't interrupt the UI, and let the UI or a controller thread cancel the work when called for.
Another option is that I've read somewhere typical humans have a perception level of about 25 milliseconds; two events are perceived to occur at the same time as long as they are less than 25 milliseconds apart. Sadly, I can no longer find the original reference, but I did at least find a corroborating article. You can use this fact to set a timer for about that long and let the process run as much as you want until the timer goes off. You may also want to account for the atypical human as well, especially if your app is in an area catering to people that may have above average reflexes.

C# System CPU Usage and syncing with Windows Task Manager

this is a two part question, I wanted to post my code here on stack to help others with the same task.
Question 1:
I have a subset of code, which I believe, is correctly measuring CPU usage (across as many cores in the system, as per times retrieved) as per the measurement interval - I use 1 second in the thread call.
I had to decipher this from the very few articles on the web and from C++ code. My question is, for question 1, is this correct what I have done?
Sometimes the value returned is a minus figure which is why I multiply by -1. Again, I am assuming, since there is very little documentation, that this is what I should be doing.
I have the following code:
public static class Processor
{
[DllImport("kernel32.dll", SetLastError = true)]
static extern bool GetSystemTimes(out ComTypes.FILETIME lpIdleTime, out ComTypes.FILETIME lpKernelTime, out ComTypes.FILETIME lpUserTime);
private static TimeSpan _sysIdleOldTs;
private static TimeSpan _sysKernelOldTs;
private static TimeSpan _sysUserOldTs;
static Processor()
{
}
public static void Test()
{
ComTypes.FILETIME sysIdle, sysKernel, sysUser;
if(GetSystemTimes(out sysIdle, out sysKernel, out sysUser))
{
TimeSpan sysIdleTs = GetTimeSpanFromFileTime(sysIdle);
TimeSpan sysKernelTs = GetTimeSpanFromFileTime(sysKernel);
TimeSpan sysUserTs = GetTimeSpanFromFileTime(sysUser);
TimeSpan sysIdleDiffenceTs = sysIdleTs.Subtract(_sysIdleOldTs);
TimeSpan sysKernelDiffenceTs = sysKernelTs.Subtract(_sysKernelOldTs);
TimeSpan sysUserDiffenceTs = sysUserTs.Subtract(_sysUserOldTs);
_sysIdleOldTs = sysIdleTs;
_sysKernelOldTs = sysKernelTs;
_sysUserOldTs = sysUserTs;
TimeSpan system = sysKernelDiffenceTs.Add(sysUserDiffenceTs);
Double cpuUsage = (((system.Subtract(sysIdleDiffenceTs).TotalMilliseconds) * 100) / system.TotalMilliseconds);
if (cpuUsage < 0)
{
Console.WriteLine("CPU: " + ((int) (cpuUsage)*-1) + "%");
}
else
{
Console.WriteLine("CPU: " + (int) (cpuUsage) + "%");
}
Console.WriteLine("");
}
else
{
Console.WriteLine("Couldn't get CPU usage!");
Console.WriteLine("");
}
}
private static TimeSpan GetTimeSpanFromFileTime(ComTypes.FILETIME time)
{
return TimeSpan.FromMilliseconds((((ulong)time.dwHighDateTime << 32) + (uint)time.dwLowDateTime) * 0.000001);
}
}
Question 2:
Is there anyway for me to sync a thread, in my program, with that of the Windows Task Manager, for the purpose of matching measurement figure e.g CPU Usage with the above code?
What I mean is, if you open Windows Task Manager, you will notice that it polls every second - which in reality it doesn't need to be less than that. What I want to do is match the timing with my thread.
So when Windows Task Manager polls, my thread polls.
Some notes:
I didn't want to use Performance Counters or .NET built in methods. In fact, I believe - from what I have read, .NET doesn't have methods for calculating the CPU usage on a machine, that Performance counters are required for this otherwise.
Performance counters have overhead and in addition make the GC grow, not to mention the delay in calling the next result. While my software does not need to be real-time performance I do need it to be as responsive and use as little CPU time as possible. The above code can be called and returned in less than a millisecond. In fact on my development machine, the time-span difference shows 0ms. I don't believe Performance Counters are as responsive.
In case you are curious, my software is gathering a number of items, CPU, Memory, Event Log items etc. of which these all need to be gathered and stored, in SQL CE, before the next poll, 1 second away. Each task, item, however is on its own thread to facilitate this.
Also, the code above is not optimized in anyway and you will notice I have yet to comment it also. The reason being is I want to make sure it is correct before optimization etc.
Update 1
As per a coment I made down the way, I removed the extra "System" timespan as it is not required and modified the line that retrieves the "CPU Usage" and cast it appropriately.
int cpuUsage = (int)(((sysKernelDifferenceTs.Add(sysUserDifferenceTs).Subtract(sysIdleDifferenceTs).TotalMilliseconds) * 100.00) / sysKernelDifferenceTs.Add(sysUserDifferenceTs).TotalMilliseconds);
Though I am still unsure of the formula. While it seems to be highly accurate it does on occasion return a minus figure which is why I multiply it by -1 if that is the case. After all, there is no such thing a -2% CPU usage etc.
Update 2
So I did a simple test using "System.Diagnostics.PerformanceCounter". While incredibly handy and does exactly what it is intended to do it does create overhead.
Here are my observations:
It took the Performance Counter that much longer to initialize. In the order of roughly three seconds longer on my i7 2.6 Ghz.
The performance counter also seemed to add on another approx 5MB of RAM usage simply by using it. What I mean by this is: With the code above ,my app maxes out at 7.5MB ram. With the performance counter it "starts" at 12.5MB.
Over the space of 5 seconds, where my thread ran 5 times - once per second, the memory of my app had grown by 1 MB and this increase is consistent with time, although it does level out, in my case anyway, 3-4MB above starting. So where my app is usually 7.5MB ram with the code above, the PC code leveled out at 16.5 MB ram - an increase of 9MB over the code above. Note: The code above does not cause this increase.
So, if your application was built in a manner where resource usage and timing is key I would suggest against using Performance counters because of these reasons. Otherwise go ahead as it works without all the mess.
As for my app, performance counters will be detrimental to my software's purpose.

I think you have a bug in your formula. You want to basically compute CPU usage as this:
CPU Usage = KernelTimeDiff + UserTimeDiff
--------------------------------------------
KernelTimeDiff + UserTimeDiff + IdleTimeDiff
Thus, a quick mod to your code as follows:
// TimeSpan system = sysKernelDiffenceTs.Add(sysUserDiffenceTs);
//Double cpuUsage = (((system.Subtract(sysIdleDiffenceTs).TotalMilliseconds) * 100) / system.TotalMilliseconds);
TimeSpan totaltime = sysKernelDiffenceTs.Add(sysUserDiffenceTs);
totaltime = totaltime.Add(sysIdleDifferenceTs);
int cpuUsage = 100 - (sysIdleDifferenceTs.TotalMilliseconds * 100) / totaltime.TotalMilliseconds;
Console.WriteLine("CPU: " + cpuUsage + "%");
You originally declared cpuUsage as "Double". I'm not sure if you wanted floating point precision, but in your code, you definitely weren't getting anything other than integer precision because the assignment statement was just doing integer math. If you need higher precision from the computation, you could easily get it by mixing in some floating point:
Double cpuUsage = 100.0 - (sysIdleDifferenceTs.TotalMilliseconds * 100.0) /totaltime.TotalMilliseconds;
Also, in regards to being in sync with Task Manager. Task Manager, as I understand it, uses perf counters. (And I would suspect that GetSystemTimes is making perf counter calls under the hood, but perhaps not). And I'm not sure why you wouldn't use perf counters either. The "% Process Time" counter is an instant sample counter that doesn't require computing a diff with a previous result. (There's one per logical cpu). Use the PDH helper functions instead of the legacy registry key apis to get at it. You can do this from an unmanaged C/C++ DLL that exports a "GetCpuUsage" function back to your C# code. But I don't know why you couldn't just PInvoke the PDH functions from C# either. I don't know about this overhead that you speak of. I'm not sure I understand your reference to " the delay in calling the next result" either.

While loop execution time

We were having a performance issue in a C# while loop. The loop was super slow doing only one simple math calc. Turns out that parmIn can be a huge number anywhere from 999999999 to MaxInt. We hadn't anticipated the giant value of parmIn. We have fixed our code using a different methodology.
The loop, coded for simplicity below, did one math calc. I am just curious as to what the actual execution time for a single iteration of a while loop containing one simple math calc is?
int v1=0;
while(v1 < parmIn) {
v1+=parmIn2;
}

There is something else going on here. The following will complete in ~100ms for me. You say that the parmIn can approach MaxInt. If this is true, and the ParmIn2 is > 1, you're not checking to see if your int + the new int will overflow. If ParmIn >= MaxInt - parmIn2, your loop might never complete as it will roll back over to MinInt and continue.
static void Main(string[] args)
{
int i = 0;
int x = int.MaxValue - 50;
int z = 42;
System.Diagnostics.Stopwatch st = new System.Diagnostics.Stopwatch();
st.Start();
while (i < x)
{
i += z;
}
st.Stop();
Console.WriteLine(st.Elapsed.Milliseconds.ToString());
Console.ReadLine();
}

Assuming an optimal compiler, it should be one operation to check the while condition, and one operation to do the addition.

The time, small as it is, to execute just one iteration of the loop shown in your question is ... surprise ... small.
However, it depends on the actual CPU speed and whatnot exactly how small it is.
It should be just a few machine instructions, so not many cycles to pass once through the iteration, but there could be a few cycles to loop back up, especially if branch prediction fails.
In any case, the code as shown either suffers from:
Premature optimization (in that you're asking about timing for it)
Incorrect assumptions. You can probably get a much faster code if parmIn is big by just calculating how many loop iterations you would have to perform, and do a multiplication. (note again that this might be an incorrect assumption, which is why there is only one sure way to find performance issues, measure measure measure)
What is your real question?

It depends on the processor you are using and the calculation it is performing. (For example, even on some modern architectures, an add may take only one clock cycle, but a divide may take many clock cycles. There is a comparison to determine if the loop should continue, which is likely to be around one clock cycle, and then a branch back to the start of the loop, which may take any number of cycles depending on pipeline size and branch prediction)
IMHO the best way to find out more is to put the code you are interested into a very large loop (millions of iterations), time the loop, and divide by the number of iterations - this will give you an idea of how long it takes per iteration of the loop. (on your PC). You can try different operations and learn a bit about how your PC works. I prefer this "hands on" approach (at least to start with) because you can learn so much more from physically trying it than just asking someone else to tell you the answer.

The while loop is couple of instructions and one instruction for the math operation. You're really looking at a minimal execution time for one iteration. it's the sheer number of iterations you're doing that is killing you.
Note that a tight loop like this has implications on other things as well, as it bogs down one CPU and it blocks the UI thread (if it's running on it). Thus, not only it is slow due to the number of operations, it also adds a perceived perf impact due to making the whole machine look unresponsive.

If you're interested in the actual execution time, why not time it for yourself and find out?
int parmIn = 10 * 1000 * 1000; // 10 million
int v1=0;
Stopwatch sw = Stopwatch.StartNew();
while(v1 < parmIn) {
v1+=parmIn2;
}
sw.Stop();
double opsPerSec = (double)parmIn / sw.Elapsed.TotalSeconds;
And, of course, the time for one iteration is 1/opsPerSec.

Whenever someone asks about how fast control structures in any language you know they are trying to optimize the wrong thing. If you find yourself changing all your i++ to ++i or changing all your switch to if...else for speed you are micro-optimizing. And micro optimizations almost never give you the speed you want. Instead, think a bit more about what you are really trying to do and devise a better way to do it.
I'm not sure if the code you posted is really what you intend to do or if it is simply the loop stripped down to what you think is causing the problem. If it is the former then what you are trying to do is find the largest value of a number that is smaller than another number. If this is really what you want then you don't really need a loop:
// assuming v1, parmIn and parmIn2 are integers,
// and you want the largest number (v1) that is
// smaller than parmIn but is a multiple of parmIn2.
// AGAIN, assuming INTEGER MATH:
v1 = (parmIn/parmIn2)*parmIn2;
EDIT: I just realized that the code as originally written gives the smallest number that is a multiple of parmIn2 that is larger than parmIn. So the correct code is:
v1 = ((parmIn/parmIn2)*parmIn2)+parmIn2;
If this is not what you really want then my advise remains the same: think a bit on what you are really trying to do (or ask on Stackoverflow) instead of trying to find out weather while or for is faster. Of course, you won't always find a mathematical solution to the problem. In which case there are other strategies to lower the number of loops taken. Here's one based on your current problem: keep doubling the incrementer until it is too large and then back off until it is just right:
int v1=0;
int incrementer=parmIn2;
// keep doubling the incrementer to
// speed up the loop:
while(v1 < parmIn) {
v1+=incrementer;
incrementer=incrementer*2;
}
// now v1 is too big, back off
// and resume normal loop:
v1-=incrementer;
while(v1 < parmIn) {
v1+=parmIn2;
}
Here's yet another alternative that speeds up the loop:
// First count at 100x speed
while(v1 < parmIn) {
v1+=parmIn2*100;
}
// back off and count at 50x speed
v1-=parmIn2*100;
while(v1 < parmIn) {
v1+=parmIn2*50;
}
// back off and count at 10x speed
v1-=parmIn2*50;
while(v1 < parmIn) {
v1+=parmIn2*10;
}
// back off and count at normal speed
v1-=parmIn2*10;
while(v1 < parmIn) {
v1+=parmIn2;
}
In my experience, especially with graphics programming where you have millions of pixels or polygons to process, speeding up code usually involve adding even more code which translates to more processor instructions instead of trying to find the fewest instructions possible for the task at hand. The trick is to avoid processing what you don't have to.