I have an array of items that are time sensitive. After an amount of time, the last item needs to fall off and a new item is put at the beginning.
What is the best way to do this?
I would suggest using a queue, just a special instance of an array or list. When your timed event occurs, pop the last item from the queue, and then push your new item on.
Probably the easiest way to do this with an array is to use a circular index. Rather than always looking at array[n], you would reference array[cIndex] (where cIndex referrs to the item in the array being indexed (cIndex is incremented based on the arraySize (cIndex % arraySize)).
When you choose to drop the oldest item in the array, you would simply reference the element located at ((cIndex + (arraySize - 1)) % arraySize).
Alternatively, you could use a linkedList approach.
Use a Queue instead.
By using a Queue, preferably one implemented using a linked-list.
Have a look at using a Queue rather than a simple array.
A queue would work if there a fixed number of items.
Given that the 'amount of time' is known, how about a SortedDictionary with a DateTime key and override the Add method to remove all items with keys that are too old.
LinkedList<T> has AddFirst and RemoveLast members that should work perfectly.
EDIT: Looking at the Queue docs, it seems they use an internal array. As long as the implementation uses a circular-array type algorithm performance should be fine.
In csharp 3 you can do:
original = new[] { newItem }.Concat(
original.Take(original.Count() - 1)).ToArray()
But you are probably better off using a specialised datastructure
Queue is great for FIFO arrays. For generic array handling, use List(T)'s
Insert(0, x) and RemoveAt(0) methods to put or remove items in front of the list, for example.
Technically you need a deque. A queue has items pushed and popped off one end only. A deque is open at both ends.
Most languages will allow array manipulation, just remove the first element and put another one on the end.
Alternatively you can shift every element, by looping. Just replace each element (starting from the oldest) with its neighbour. Then place the new item in the last element.
If you know that your deque won't go above a certain size, then you can make it circular. You'll need two pointers to tell you where the two ends are though. Adding and removing items, will increase/decrease your pointers accordingly. You'll have to detect a buffer overflow condition (i.e. your pointers 'cross'). And you'll have to use modular arithmetic so your pointers go in a circle around the array.
Or you could time stamp each element in the array and remove them when they become too 'old'. You can either do this by keeping a separate array indexed in the same way, or by having an array of two element arrays, with the time stamp stored in one of the sub-elements.
If you're looking for the fastest way of doing this, it's going to be a circular array: you keep track of your current position in the array (ndx), and the end of the array (end), so when you insert an item, you implicitly eliminate the oldest item.
A circular array is the fastest implementation of a fixed-size queue that I know of.
For example, in C/C++ it would look like this for ints (quitting when you get a 0):
int queue[SIZE];
int ndx=0; // start at the beginning of the array
int end=SIZE-1;
int newitem;
while(1){
cin >> newitem;
if(!newitem) // quit if it's a 0
break;
if(ndx>end) // need to loop around the end of the array
ndx=0;
queue[ndx] = newitem;
ndx++
}
Lots of optimization could be done, but if you want to built it yourself, this is the fastest route.
If you don't care about performance, use a shipped Queue object because it should be generalized.
It may or may not be optimized, and it may not support a fixed size list, so be sure to check the documentation on it before using.
Related
Trying to create a type of low pass filter where I constantly average the previous 10 float values. Not sure whether to use builtin arrays (native .NET arrays), the .NET Stack operator, or perhaps an ArrayList or List.
In pseudocode I need to
1- Define the array or Stack containing 10 floats
2- Every update Push a new value to the Array, Stack, List
3- Check the length and if greater than10 remove the first or oldest float value from the Array, Stack, List
4- Get the average of all float values in the Array, Stack, List
5- Repeat steps 2-4
1st question Should I be using built in Arrays, Stack, ArrayList or List instead? I notice in the Stack documentation there is no method for removing the oldest (bottom)item from the stack, but perhaps I am missing something
https://msdn.microsoft.com/en-us/library/3278tedw(v=vs.100).aspx
What I need is all the functionality of the (Javascript only) Array class but in C#.
2nd Can anyone help with actual syntax using either approach? Any help appreciated!
You've probably missed the Queue data structure. With it, you can put new elements to the end and delete old ones from the front.
Additionally, you don't need to look at all the elements in the array to compute the average if there are constantly 10 elements. Knowing the previous average you can compute the new average like this:
newAvg = oldAvg + (newElem - deletedElem)/10;
or more shortly:
avg += (newElem - deletedElem)/10;
You could use the generic Queue<T> collection to store the values:
var queue = new Queue<float>();
queue.Enqueue(1.0f); // pushes new item
queue.Dequeue(); // removes oldest item
To obtain the average, use the LINQ Average() extension method:
var average = queue.Average();
Stacks are LIFO(Last In First Out).
If you want to remove the oldest value, then you'd want to use a queue (First In First Out).
We have the Enqueue and Dequeue methods to work on a queue in C#.
But is there a way we can remove the last item in the queue? Dequeue actually removes the item from the top.
The entire purpose of a Queue is to walk the collection from top to bottom. If you want a collection type that allows removing from the back too (or any place in the collection), use another collection type, a List<T> for example.
Maybe you should consider using a Stack instead of a Queue.
Stack works similar to a queue but instead of the first in first out behaviour a stack is used when you need a last in first out solution for your objects
This is an expensive operation, so I only do this rarely. People here getting mad and not answering... all you do is just use a second Queue to move everything to:
// original queue must already be filled up
Queue<Waypoint> waypoints = new Queue<Waypoint>();
// fill it up...
Queue<Waypoint> temp = new Queue<Waypoint>();
while (waypoints.Count > 0) // stop one short of the end
{
temp.Enqueue(waypoints.Dequeue());
}
Waypoint wp = waypoints.Dequeue();
// get the last one and do what you desire to it...
// I modified mine and then added it back, this is not needed
temp.Enqueue(new Waypoint(wp.pos, (p.vectorPath[0] - wp.pos).normalized));
// necessary: you have waypoints empty now so you have to swap it for temp:
waypoints = temp;
Like Patrick stated the queue is meant to be first in first out. Think of it like a printer queue, the first page that goes to into the printer queue is the first to be printed out.
Look into using a List instead of an Array, I found lists a bit easier to manipulate with a bunch of extended methods that arrays do not have.
http://www.dotnetperls.com/list
Here is a link to how to remove the last item in a list which has a few different ways of accomplishing it.
How to remove the last element added into the List?
hope this helps!
I need to store a set of elements. What I need is functionality to
remove (single) elements and
add (sets of) elements and
each object should only be in the set once and
get a random element from the set
I chose the HashSet (C#) since it sports fast methods for removing elements (hashSet.remove(element)), adding sets (hashSet.UnionWith(anotherHashSet)) and the nature of a HashSet guarantees that there are not duplicates, so requirements 1 to 3 are taken care of.
The only way I found to get a random element is
Object object = hashSet.ElementAt(rnd.Next(hashSet.Count));
But this is very slow, since I call it once for every pixel of my map (creating a random flood fill from multiple starting points; mapsize 500x500 at the moment but I'd like to go bigger) and the hashset holds rather many items. (A quick test shows it blows up to 5752 entries before shrinking again.)
Profiling (CPU sampling) tells me my ElementAt calls take over 50%.
I realize 500x500 operations over a big hashset is no easy task, but other operations (Remove and UnionWith) are called as often as ElementAt, so the main problem seems to be the operation and not the number of calls.
I vaguely understand why getting a certain element from a HashSet is very expensive (when compared to getting it from a list or another ordered data structure, but I just want a random pick. Can it really be so hard and is there no way around it? Is there a better data structure for my purpose?
Changing everything to Lists doesn't help because now other methods become bottlenecks and it takes even longer.
Casting the HashSet to an array and pick my random element from there expectedly doesn't help because while picking a random element from an array is quick, casting the hashset to the array in the first place takes longer than running hashSet.ElementAt by itself.
If you want to understand better what I am trying to do: A link to my question and the answer.
I think that OrderedDictionary might suit your purposes:
var dict = new OrderedDictionary();
dict.Add("My String Key", "My String");
dict.Add(12345, 54321);
Console.WriteLine(dict[0]); // Prints "My String"
Console.WriteLine(dict[1]); // Prints 54321
Console.WriteLine(dict["My String Key"]); // Prints "My String"
Console.WriteLine(dict[(object)12345]); // Prints 54321 (note the need to cast!)
This has fast add and remove, and O(1) indexing. It only works with object keys and values though - there's no generic version.
[EDIT] Many years later: We now have the strongly-typed generic SortedDictionary<TKey, TValue> which might be better.
The basic problem is the indexing.
In an array or a list, the data is indexed by its coördinate - usually just a simple int index. In a HashSet, you pick the index yourself - the key. The side-effect is, though, that there is no "coördinate" - the question "element at index 3" doesn't make sense, really. The way it's actually implemented is that the whole HashSet is enumerated, item after item, and the n-th item is returned. This means that to get the 1000th item, you have to enumerate all the 999 items before that as well. This hurts.
The best way to solve this would be to pick the random based on an actual key of the HashSet. Of course, this only works if it's reasonable to pick random keys just like that.
If you can't pick the key at random in a satisfactory way, you'll probably want to keep two separate lists - whenever you add a new item to a HashSet, add its key to a List<TKey>; you can then easily pick a random key from the List, and follow it. Depending on your requirements, duplicates may not be much of a problem.
And of course, you could save on the ElementAt enumerations if you only do the enumeration once - for example, before searching the HashSet, you could convert it to List. This only makes sense if you're picking multiple random indices at once, of course (e.g. if you pick 5 indices at random at once, you'll save about 1/5th of the time on average) - if you're always picking one, then modifying the HashSet and picking another, it's not going to help.
Depending on your exact use case, it might also be worth having a look at SortedSet. It works in a similar way to HashSet, but it maintains order in the keys. The helpful part is that you can use the GetViewBetween method to get a whole range of keys - you could use this quite effectively if your keys are sparse, but well balanced between arbitrary ranges. You'd just first pick a range at random, then get the items in range with GetViewBetween, and pick a random one out of those as well. In effect, this will allow you to partition the search results, and should save quite a bit of time.
What data structure could I use in C# to allow quick insertion/deletion as well as uniform random selection? A List has slow deletion by element (since it needs to find the index of the element each time), while a HashSet does not seem to allow random selection of an element (without copying to a list.)
The data structure will be updated continuously, so insertion and deletion need to be online procedures. It seems as if there should be a way to make insertion, deletion, and random selection all O(log n).
A binary search tree with arbitrary integer keys assigned to the objects would solve all of these problems, but I can't find the appropriate class in the C# standard library. Is there a canonical way to solve this without writing a custom binary search tree?
There is already a BST in the C# BCL, it's called a SortedDictionary<TKey, TValue>, if you don't want Key Value Pairs, but instead want single items, you can use the SortedSet<T> (SortedSet is in .NET 4.0).
It sounds like from your example you'd want a SortedDictionary<int, WhateverValueType>. Though I'm not sure exactly what you are after when you say "uniform random selection".
Of course, the Dictionary<TKey, TValue> is O(1) which is much faster. So unless you have a need for sorted order of the keys, I'd use that.
UPDATE: From the sounds of your needs, you're going to have a catch-22 on efficiency. To be able to jump into a random contiguous index in the data structure, how often will you be inserting/deleting? If not often, you could use an array and just Sort() after (O(n log n)), or always insert/delete in order (O(n)).
Or, you could wrap a Dictionary<int, YourType> and keep a parallel List<int> and update it after every Add/Delete:
_dictionary.Add(newIndex, newValue);
_indexes.Add(newIndex);
And then just access a random index from the list on lookups. The nice thing is that in this method really the Add() will be ~ O(1) (unless List resizes, but you can set an initial capacity to avoid some of that) but you would incurr a O(n) cost on removes.
I'm afraid the problem is you'll either sacrifice times on the lookups, or on the deletes/inserts. The problem is all the best access-time containers are non-contiguous. With the dual List<int>/Dictionary<int, YourValue> combo, though, you'd have a pretty good mix.
UPDATE 2: It sounds like from our continued discussion that if that absolute performance is your requirement you may have better luck rolling your own. Was fun to think about though, I'll update if I think of anything else.
Binary search trees and derived structures, like SortedDictionary or SortedSet, operate by comparing keys.
Your objects are not comparable by itself, but they offer object identity and a hash value. Therefore, a HashSet is the right data structure. Note: A Dictionary<int,YourType> is not appropriate because removal becomes a linear search (O(n)), and doesn't solve the random problem after removals.
Insert is O(1)
Remove is O(1)
RandomElement is O(n). It can easily be implemented, e.g.
set.ElementAt(random.Next(set.Count))
No copying to an intermediate list is necessary.
I realize that this question is over 3 years old, but just for people who come across this page:
If you don't need to keep the items in the data set sorted, you can just use a List<ItemType>.
Insertion and random selection are O(1). You can do deletion in O(1) by just moving the last item to the position of the item you want to delete and removing it from the end.
Code:
using System; // For the Random
using System.Collections.Generic; // The List
// List:
List<ItemType> list = new List<ItemType>();
// Add x:
ItemType x = ...; // The item to insert into the list
list.Add( x );
// Random selection
Random r = ...; // Probably get this from somewhere else
int index = r.Next( list.Count );
ItemType y = list[index];
// Remove item at index
list[index] = list[list.Count - 1]; // Copy last item to index
list.RemoveAt( list.Count - 1 ); // Remove from end of list
EDIT: Of course, to remove an element from the List<ItemType> you'll need to know its index. If you want to remove a random element, you can use a random index (as done in the example above). If you want to remove a given item, you can keep a Dictionary<ItemType,int> which maps the items to their indices. Adding, removing and updating these indices can all be done in O(1) (amortized).
Together this results in a complexity of O(1) (amortized) for all operations.
This question already has answers here:
How to find the index of an element in an array in Java?
(15 answers)
Closed 6 years ago.
I was asked this question in an interview. Although the interview was for dot net position, he asked me this question in context to java, because I had mentioned java also in my resume.
How to find the index of an element having value X in an array ?
I said iterating from the first element till last and checking whether the value is X would give the result. He asked about a method involving less number of iterations, I said using binary search but that is only possible for sorted array. I tried saying using IndexOf function in the Array class. But nothing from my side answered that question.
Is there any fast way of getting the index of an element having value X in an array ?
As long as there is no knowledge about the array (is it sorted? ascending or descending? etc etc), there is no way of finding an element without inspecting each one.
Also, that is exactly what indexOf does (when using lists).
How to find the index of an element having value X in an array ?
This would be fast:
int getXIndex(int x){
myArray[0] = x;
return 0;
}
A practical way of finding it faster is by parallel processing.
Just divide the array in N parts and assign every part to a thread that iterates through the elements of its part until value is found. N should preferably be the processor's number of cores.
If a binary search isn't possible (beacuse the array isn't sorted) and you don't have some kind of advanced search index, the only way I could think of that isn't O(n) is if the item's position in the array is a function of the item itself (like, if the array is [10, 20, 30, 40], the position of an element n is (n / 10) - 1).
Maybe he wants to test your knowledge about Java.
There is Utility Class called Arrays, this class contains various methods for manipulating arrays (such as sorting and searching)
http://download.oracle.com/javase/6/docs/api/java/util/Arrays.html
In 2 lines you can have a O(n * log n) result:
Arrays.sort(list); //O(n * log n)
Arrays.binarySearch(list, 88)); //O(log n)
Puneet - in .net its:
string[] testArray = {"fred", "bill"};
var indexOffset = Array.IndexOf(testArray, "fred");
[edit] - having read the question properly now, :) an alternative in linq would be:
string[] testArray = { "cat", "dog", "banana", "orange" };
int firstItem = testArray.Select((item, index) => new
{
ItemName = item,
Position = index
}).Where(i => i.ItemName == "banana")
.First()
.Position;
this of course would find the FIRST occurence of the string. subsequent duplicates would require additional logic. but then so would a looped approach.
jim
It's a question about data structures and algorithms (altough a very simple data structure). It goes beyond the language you are using.
If the array is ordered you can get O(log n) using binary search and a modified version of it for border cases (not using always (a+b)/2 as the pivot point, but it's a pretty sophisticated quirk).
If the array is not ordered then... good luck.
He can be asking you about what methods you have in order to find an item in Java. But anyway they're not faster. They can be olny simpler to use (than a for-each - compare - return).
There's another solution that's creating an auxiliary structure to do a faster search (like a hashmap) but, OF COURSE, it's more expensive to create it and use it once than to do a simple linear search.
Take a perfectly unsorted array, just a list of numbers in memory. All the machine can do is look at individual numbers in memory, and check if they are the right number. This is the "password cracker problem". There is no faster way than to search from the beginning until the correct value is hit.
Are you sure about the question? I have got a questions somewhat similar to your question.
Given a sorted array, there is one element "x" whose value is same as its index find the index of that element.
For example:
//0,1,2,3,4,5,6,7,8,9, 10
int a[10]={1,3,5,5,6,6,6,8,9,10,11};
at index 6 that value and index are same.
for this array a, answer should be 6.
This is not an answer, in case there was something missed in the original question this would clarify that.
If the only information you have is the fact that it's an unsorted array, with no reletionship between the index and value, and with no auxiliary data structures, then you have to potentially examine every element to see if it holds the information you want.
However, interviews are meant to separate the wheat from the chaff so it's important to realise that they want to see how you approach problems. Hence the idea is to ask questions to see if any more information is (or could be made) available, information that can make your search more efficient.
Questions like:
1/ Does the data change very often?
If not, then you can use an extra data structure.
For example, maintain a dirty flag which is initially true. When you want to find an item and it's true, build that extra structure (sorted array, tree, hash or whatever) which will greatly speed up searches, then set the dirty flag to false, then use that structure to find the item.
If you want to find an item and the dirty flag is false, just use the structure, no need to rebuild it.
Of course, any changes to the data should set the dirty flag to true so that the next search rebuilds the structure.
This will greatly speed up (through amortisation) queries for data that's read far more often than written.
In other words, the first search after a change will be relatively slow but subsequent searches can be much faster.
You'll probably want to wrap the array inside a class so that you can control the dirty flag correctly.
2/ Are we allowed to use a different data structure than a raw array?
This will be similar to the first point given above. If we modify the data structure from an array into an arbitrary class containing the array, you can still get all the advantages such as quick random access to each element.
But we gain the ability to update extra information within the data structure whenever the data changes.
So, rather than using a dirty flag and doing a large update on the next search, we can make small changes to the extra information whenever the array is changed.
This gets rid of the slow response of the first search after a change by amortising the cost across all changes (each change having a small cost).
3. How many items will typically be in the list?
This is actually more important than most people realise.
All talk of optimisation tends to be useless unless your data sets are relatively large and performance is actually important.
For example, if you have a 100-item array, it's quite acceptable to use even the brain-dead bubble sort since the difference in timings between that and the fastest sort you can find tend to be irrelevant (unless you need to do it thousands of times per second of course).
For this case, finding the first index for a given value, it's probably perfectly acceptable to do a sequential search as long as your array stays under a certain size.
The bottom line is that you're there to prove your worth, and the interviewer is (usually) there to guide you. Unless they're sadistic, they're quite happy for you to ask them questions to try an narrow down the scope of the problem.
Ask the questions (as you have for the possibility the data may be sorted. They should be impressed with your approach even if you can't come up with a solution.
In fact (and I've done this in the past), they may reject all your possibile approaches (no, it's not sorted, no, no other data structures are allowed, and so on) just to see how far you get.
And maybe, just maybe, like the Kobayashi Maru, it may not be about winning, it may be how you deal with failure :-)