We Keep Coding

Filter text in a variable

I have a variable (serial_and_username_and_subType) that contains this type of text:
CT-AF-23-GQG %username1% *subscriptionType*
DHR-345349-E %username2% *subscriptionType*
C3T-AF434-234-GQG %username3% *subscriptionType*
34-7-HHDHFD-DHR-345349-E %username4% *subscriptionType*
example: ST-NN1-CQ-QQQ-G12 %RandomDUDE12% *Lifetime*
after that, i have an IF instruction that checks if the user inputs something that is present in serial_and_username_and_subType.
if (userInput.Contains
(serial_and_username_and_subType))......
then, what i would like to do (but i am having troubles) is that when someone enters a serial, the program prints the corrispective username and subscription:
for example:
Please enter your Serial:
> ST-NN1-CQ-QQQ-G12
Welcome, RandomDUDE12!
You currently have a Lifetime subscription!
does anyone know a method or a way to obtain what i need?

You are already using Contains(). The other things you could use are
Substring()
Split()
IndexOf()
Split() is probably the easiest one as long as you can guarantee that neither % nor * are part of the serial, username or license:
var s = "ST-NN1-CQ-QQQ-G12 %RandomDUDE12% *Lifetime*";
var splitPercent = s.Split('%');
Console.WriteLine(splitPercent[1]);
var splitStar = s.Split('*');
Console.WriteLine(splitStar[1]);
This approach will work fine as long as you have few licenses only (maybe a few thousand are ok, because PCs are fast). If you have many licenses (like millions), you probably want to separate all that information so that it is not in a string, but a data structure. You would then use a dictionary and access the information directly, instead of iterating through all of them.

ASP.NET Caching, limiting the number of entries in cache

I'm trying to implement data caching for a web app in ASP.NET, this is for class and I've been asked to limit the number of entries in the ObjectCache, not by memory size but by the number of entries itself. This is quite easy since I can call ObjectCache.Count, but when the cache grows beyond the established limit (5, just for testing) I can't figure out how to remove the oldest element stored since it's alphabetically sorted.
This is being implemented in a Service, at the Data Access layer so I can't use any additional structure like a Queue to keep track of the insertions in the cache.
What can I do? Is there a way to filter or get the older element in the cache?
Here's the method code
public List<EventSummary> FindEvents(String keywords, long categoryId, int start, int count)
{
string queryKey = "FindEvent-" + start + ":" + count + "-" + keywords.Trim() + "-" + categoryId;
ObjectCache cache = MemoryCache.Default;
List<EventSummary> val = (List<EventSummary>)cache.Get(queryKey);
if (val != null)
return val;
Category evnCategory = CategoryDao.Find(categoryId);
List<Event> fullResult = EventDao.FindByEventCategoryAndKeyword(evnCategory, keywords, start, count);
List<EventSummary> summaryResult = new List<EventSummary>();
foreach (Event evento in fullResult)
{
summaryResult.Add(new EventSummary(evento.evnId, evento.evnName, evento.Category, evento.evnDate));
}
if (cache.Count() >= maxCacheSize)
{
//WHAT SHOULD I DO HERE?
}
cache.Add(queryKey, summaryResult, DateTime.Now.AddDays(cacheDays));
return summaryResult;
}

As mentioned in the comments, the Trim method from MemoryCache has a LRU (Least Recently Used) policy, which is the behavior you are looking for here. Unfortunately, the method is not based on an absolute number of objects to remove from the cache, but rather on a percentage, which is an int parameter. This just means that, if you try to hack your way around it and pass 1 / cache.Count() as the percentage, you have no control over how many objects have truly been removed from the cache, which is not an ideal scenario.
Another way to do it would just be to go with a DIY approach and simply not use the .NET caching utilities since, in our case, they do not seem to natively exactly fit your needs. I'm thinking of something along the lines of a SortedDictionary with the timecode of your cache objects as the key and a list of cache objects inserted into the cache at the given timecode as you values. It would be a good and, IMO, not too daring exercice to try and reproduce the .NET cache behavior you are already using, with the additionnal benefit of directly controlling the removal policy yourself.

As a side comment,not directly related to your question,
the biggest problem with caches in managed memory models is GC.
The moment you start storing over a few million entries you are asking for eventual GC pauses even with the most advanced non-blocking GCs.
It is hard to cache over 16 Gb, without pausing every now and then for 5-6 seconds (that is stop-all).
I have previously described here: https://stackoverflow.com/a/30584575/1932601
the caching of objects as-is is eventually a bad choice if you need to store very many expiring entries (say 100 million chat messages)
Take a look at what we did to store hundreds of millions of objects for a long time without killing the GC.
https://www.youtube.com/watch?v=Dz_7hukyejQ

Thread-safe increment/decrement with StackExchange.Redis

I'm using the StackExchange.Redis package as a shared store for a multi-threaded application that will in turn be running on multiple servers simultaneously...so multi-multi-threaded ;)
One of the simplest use cases I have is that I want to keep a basic count on a series of keys (i.e. KEY1=4 KEY2=7, KEY3=13, etc). And I'll have business rules in place to enforce the max permissible value allowed for a given key. So for example, say KEY1 can only ever go as high as 5...if two threads fire at exactly the same time trying to increment it, I only want to allow one of them to succeed.
I believe I can accomplish this with a transaction by first fetching the current value and then making it condition that the value hasn't changed. Will this work as I am expecting? Is there a more elegant way to do this?
public void Increment(string key) {
IDatabase db = redisConn.GetDatabase();
var val = db.StringGet(key);
int maxVal = 5;
if (Convert.ToInt32(val) < maxVal) {
var trans = db.CreateTransaction();
trans.AddCondition(Condition.StringEqual(key, val));
trans.StringIncrementAsync(key);
trans.Execute();
}
}
PS: Love this package, it's a pleasure to work with

Yes, that should work fine. But it may be both easier and more efficient to use a Lua script via ScriptEvaluate.
Note: with your current code, you may want to check the response of Execute and "redo from start" if false. If a thread-race happens when moving from 2 to 3, with your current code: the update would be discarded. This wouldn't be an issue with a Lua script.

I can't figure out what is slowing my program down

I have created a Windows Form application that reads in a text file, rearranges the data, and writes to a new text file. I have noticed that it slows down exponentially as it runs. I have been using tracepoints, stopwatches, and datetime to figure out why each iteration is taking longer than the previous, but I can't figure it out. My best guess would be that it might have something to do with the way I'm initializing variables.
I'm not sure how helpful this snippet of code will be but maybe it will give some insight into my problem:
while (cuttedWords.Any())
{
var variable = cuttedWords.TakeWhile(x => x != separator).ToArray();
cuttedWords = cuttedWords.Skip(variable.Length + 1);
sortDataObject.SortDataMethod(variable, b);
if (sortDataObject.virtualPara)
{
if (!virtualParaUsed)
{
listOfNames = sortDataObject.findListOfNames(backgroundWords, ref IDforCounting, countParametersTable);
}
virtualParaUsed = true;
printDataObject.WriteFileVirtual(fileName, ID, sortDataObject.listNames[0], sortDataObject.listNames[1],
sortDataObject.unit, listOfNames, sortDataObject.virtualNames);
sortDataObject.virtualNames.Clear();
}
else
{
int[] indexes = checkedListBox1.CheckedIndices.Cast<int>().ToArray();
printDataObject.WriteFile(fileName, ID, sortDataObject.listNames[0], sortDataObject.listNames[1],
sortDataObject.unit, sortDataObject.hexValue[0], sortDataObject.stringShift, sortDataObject.sign,
sortDataObject.SFBinary[0], sortDataObject.wordValue, sortDataObject.conversions, sortDataObject.stringData, indexes, sortDataObject.conType);
}
decimal sum = ((decimal)IDforCounting) / countParametersTable * 100;
int sum2 = (int)sum;
backgroundWorker1.ReportProgress(sum2);
ID++;
IDforCounting++;
b++;
}
What is strange to me is that I know that each loop runs in a matter of milliseconds, but from the start of one loop to the start of the next, the time keeps increasing.
I apologize if this is not enough information to analyze my issue, but I'm not sure what else I can provide without showing my entire solution.
Thank you.
EDIT: A better questions might be: what is a good way to analyze performance if stopwatches aren't doing the trick. I'd rather not have to download a profiler.

If its taking longer and longer, on each iteration, its probably related to the initial cuttedWords.any().
What type is cuttedWords? If its a database-backed enumerable, it will re-issue the sql statement on every iteration, which may or may not be what you want.
On the other hand, if this is a producer-consumer scenario, it may be that cuttedWords is locked by the producer, causing the consumer to be thread-locked while waiting for the producer to complete its action.
Also, the .reportProgress will cause the backgroundworker to raise an event on the thread that created it, potentially causing UI updates, so maybe try removing that line and see if it helps any. Then replace it with some code that only calls reportProgress if the progress has actually changed.

Back to basics; for-loops, arrays/vectors/lists, and optimization

I was working on some code recently and came across a method that had 3 for-loops that worked on 2 different arrays.
Basically, what was happening was a foreach loop would walk through a vector and convert a DateTime from an object, and then another foreach loop would convert a long value from an object. Each of these loops would store the converted value into lists.
The final loop would go through these two lists and store those values into yet another list because one final conversion needed to be done for the date.
Then after all that is said and done, The final two lists are converted to an array using ToArray().
Ok, bear with me, I'm finally getting to my question.
So, I decided to make a single for loop to replace the first two foreach loops and convert the values in one fell swoop (the third loop is quasi-necessary, although, I'm sure with some working I could also put it into the single loop).
But then I read the article "What your computer does while you wait" by Gustav Duarte and started thinking about memory management and what the data was doing while it's being accessed in the for-loop where two lists are being accessed simultaneously.
So my question is, what is the best approach for something like this? Try to condense the for-loops so it happens in as little loops as possible, causing multiple data access for the different lists. Or, allow the multiple loops and let the system bring in data it's anticipating. These lists and arrays can be potentially large and looping through 3 lists, perhaps 4 depending on how ToArray() is implemented, can get very costy (O(n^3) ??). But from what I understood in said article and from my CS classes, having to fetch data can be expensive too.
Would anyone like to provide any insight? Or have I completely gone off my rocker and need to relearn what I have unlearned?
Thank you

The best approach? Write the most readable code, work out its complexity, and work out if that's actually a problem.
If each of your loops is O(n), then you've still only got an O(n) operation.
Having said that, it does sound like a LINQ approach would be more readable... and quite possibly more efficient as well. Admittedly we haven't seen the code, but I suspect it's the kind of thing which is ideal for LINQ.

For referemce,
the article is at
What your computer does while you wait - Gustav Duarte
Also there's a guide to big-O notation.
It's impossible to answer the question without being able to see code/pseudocode. The only reliable answer is "use a profiler". Assuming what your loops are doing is a disservice to you and anyone who reads this question.

Well, you've got complications if the two vectors are of different sizes. As has already been pointed out, this doesn't increase the overall complexity of the issue, so I'd stick with the simplest code - which is probably 2 loops, rather than 1 loop with complicated test conditions re the two different lengths.
Actually, these length tests could easily make the two loops quicker than a single loop. You might also get better memory fetch performance with 2 loops - i.e. you are looking at contiguous memory - i.e. A[0],A[1],A[2]... B[0],B[1],B[2]..., rather than A[0],B[0],A[1],B[1],A[2],B[2]...
So in every way, I'd go with 2 separate loops ;-p

Am I understanding you correctly in this?
You have these loops:
for (...){
// Do A
}
for (...){
// Do B
}
for (...){
// Do C
}
And you converted it into
for (...){
// Do A
// Do B
}
for (...){
// Do C
}
and you're wondering which is faster?
If not, some pseudocode would be nice, so we could see what you meant. :)
Impossible to say. It could go either way. You're right, fetching data is expensive, but locality is also important. The first version may be better for data locality, but on the other hand, the second has bigger blocks with no branches, allowing more efficient instruction scheduling.
If the extra performance really matters (as Jon Skeet says, it probably doesn't, and you should pick whatever is most readable), you really need to measure both options, to see which is fastest.
My gut feeling says the second, with more work being done between jump instructions, would be more efficient, but it's just a hunch, and it can easily be wrong.

Aside from cache thrashing on large functions, there may be benefits on tiny functions as well. This applies on any auto-vectorizing compiler (not sure if Java JIT will do this yet, but you can count on it eventually).
Suppose this is your code:
// if this compiles down to a raw memory copy with a bitmask...
Date morningOf(Date d) { return Date(d.year, d.month, d.day, 0, 0, 0); }
Date timestamps[N];
Date mornings[N];
// ... then this can be parallelized using SSE or other SIMD instructions
for (int i = 0; i != N; ++i)
mornings[i] = morningOf(timestamps[i]);
// ... and this will just run like normal
for (int i = 0; i != N; ++i)
doOtherCrap(mornings[i]);
For large data sets, splitting the vectorizable code out into a separate loop can be a big win (provided caching doesn't become a problem). If it was all left as a single loop, no vectorization would occur.
This is something that Intel recommends in their C/C++ optimization manual, and it really can make a big difference.

... working on one piece of data but with two functions can sometimes make it so that code to act on that data doesn't fit in the processor's low level caches.
for(i=0, i<10, i++ ) {
myObject object = array[i];
myObject.functionreallybig1(); // pushes functionreallybig2 out of cache
myObject.functionreallybig2(); // pushes functionreallybig1 out of cache
}
vs
for(i=0, i<10, i++ ) {
myObject object = array[i];
myObject.functionreallybig1(); // this stays in the cache next time through loop
}
for(i=0, i<10, i++ ) {
myObject object = array[i];
myObject.functionreallybig2(); // this stays in the cache next time through loop
}
But it was probably a mistake (usually this type of trick is commented)
When data is cycicly loaded and unloaded like this, it is called cache thrashing, btw.
This is a seperate issue from the data these functions are working on, as typically the processor caches that separately.

I apologize for not responding sooner and providing any kind of code. I got sidetracked on my project and had to work on something else.
To answer anyone still monitoring this question;
Yes, like jalf said, the function is something like:
PrepareData(vectorA, VectorB, xArray, yArray):
listA
listB
foreach(value in vectorA)
convert values insert in listA
foreach(value in vectorB)
convert values insert in listB
listC
listD
for(int i = 0; i < listB.count; i++)
listC[i] = listB[i] converted to something
listD[i] = listA[i]
xArray = listC.ToArray()
yArray = listD.ToArray()
I changed it to:
PrepareData(vectorA, vectorB, ref xArray, ref yArray):
listA
listB
for(int i = 0; i < vectorA.count && vectorB.count; i++)
convert values insert in listA
convert values insert in listB
listC
listD
for(int i = 0; i < listB.count; i++)
listC[i] = listB[i] converted to something
listD[i] = listA[i]
xArray = listC.ToArray()
yArray = listD.ToArray()
Keeping in mind that the vectors can potentially have a large number of items. I figured the second one would be better, so that the program wouldnt't have to loop n times 2 or 3 different times. But then I started to wonder about the affects (effects?) of memory fetching, or prefetching, or what have you.
So, I hope this helps to clear up the question, although a good number of you have provided excellent answers.

Thank you every one for the information. Thinking in terms of Big-O and how to optimize has never been my strong point. I believe I am going to put the code back to the way it was, I should have trusted the way it was written before instead of jumping on my novice instincts. Also, in the future I will put more reference so everyone can understand what the heck I'm talking about (clarity is also not a strong point of mine :-/).
Thank you again.