We Keep Coding

The faster way to get random numbers [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 5 years ago.
Improve this question
I have a problem
I need to get random numbers, but except numbers that I already have.
My Code:
List<int> current_numbers = repository.GetCurrentNumbers();
Random rnd = new Random(42);
var new_values = Enumerable.Range(10000000,99999999)
.Except(current_numbers)
.OrderBy(o=> rnd.Next())
.Take(amount)
.ToList();
But this code is VERY SLOWLY
When I tried to use select instead OrderBy - I got DUPLICATES.
In my case, its must be without duplicates.
UPDATED:
With OrderBy -- I have problem with memory :)
Range must be like this 1M - 99M
Thank you.

Since you want numbers from such a large range, you probably want to use a HashSet to eliminate duplicates.
HashSet<int> current_numbers = new HashSet<int>(repository.GetCurrentNumbers());
HashSet<int> newValues = new HashSet<int>();
while (newValues.Count < amount)
{
var next = rnd.Next(10000000,99999999);
if (!current_numbers.Contains(next))
newValues.Add(next);
}
Converting current_numbers to a HashSet will speed up the process, because a call to Contains would take O(n) time if current_numbers is not stored as a HashSet.

Use a HashSet<T> instead of a List and then test using Contains - if you take a peek at Reference Source, you'll note that Except will build those existing numbers into a Set.
Since OrderBy tries to sort the whole collection, you're losing the benefits of lazily executed enumerables by using OrderBy at all - instead, using a regular loop and generate random numbers
var random = new Random(); // Default constructor or you'll get the same sequence because of a constant seed
var result = new HashSet<int>();
var currentNumbers = new HashSet<int>(repository.GetCurrentNumbers());
while(result.Count < amount)
{
var next = random.Next(1000000,99000000);
if(currentNumbers.Contains(next)) continue;
result.Add(next);
}
Or write you own generator
public static IEnumerable<int> GenerateRandom()
{
var random = new Random();
while(true) { yield return random.Next(1000000,99000000); }
}
// Later...
var newValues = MyClass.GenerateRandom()
.Where(next => !currentNumbers.Contains(next))
.Distinct()
.Take(amount)
.ToList();

To avoid creating that huge list of numbers you can instead keep track of the numbers you have and randomly pick where the next number should come from. First you need an ordered list of the used number. Then add the lower and upper bounds to it. Then keep track of the index of the lower and upper bounds. Then iterate until you have enough number and each time randomly pick an index between the lower and upper bound indexes. Check that the difference between the number at that index and the next is 1 and if so increment the index until it isn't or you hit the upper bound. If the upper bound was hit then just walk the upper bound down and try again. When you do find a gap in the used numbers randomly pick a number in the gap and add it to your return list and to the used list at the proper index. Then make sure to walk the lower bound index up if needed.
var used = repository.GetCurrentNumbers().OrderBy(x => x).ToList();
used.InsertAt(0, 999999) // This is the lower bounds;
used.Add(100000000); // This is the upper bounds;
var random = new Random();
var values = new List<int>();
int lowerIndex = 0;
int upperIndex = used.Length - 1;
while(values.Count < amount) {
int ind = random.Next(lowerIndex, upperIndex);
while(ind < upperIndex && used[ind+1] - used[ind] == 1) ind++;
if(ind == upperIndex){
while(used[upperIndex] - used[upperIndex-1] == 1) upperIndex--;
continue;
}
int val = random.Next(used[ind]+1, used[ind+1]);
values.Add(val);
used.InsertAt(ind+1, val);
while(used[lowerIndx+1] - used[lowerIndex] == 1) lowerIndex++;
}
This works best if amount isn't a very large number and your overall range is large and the initial used numbers is also sparse.

how to create a random int[] [duplicate]

This question already has answers here:
Most efficient way to randomly "sort" (Shuffle) a list of integers in C#
(13 answers)
Closed 5 years ago.
I need to create a random array of int with certain parameters.
int[] test = new int[80];
Random random = new Random();
I need to assign 1's and 0's, randomly to the array. I know how to do that.
test[position] = Random.Next(0,2);//obviously with a for loop
But I need to have exactly 20 1's, but they need to be randomly positioned in the array. I don't know how to make sure that the 1's are randomly positioned, AND that there are exactly 20 1's. The rest of the positions in the array would be assigned 0.

I think you need to turn your thinking around.
Consider:
var cnt = 20;
while (cnt > 0) {
var r = Random.Next(0, 80);
if (test[r] != 1) {
test[r] = 1;
cnt--;
}
}
Expanding explanation based on comments from CodeCaster. Rather than generate a random value to place in the array, this code generates and index to set. Since C# automatically initializes the test array to 0 these values are already set. So all you need is to add your 1 values. The code generates a random index, tests it to see if it isn't 1, if so it sets the array element and decrements a count (cnt). Once count reaches zero the loop terminates.
This won't properly function if you need more values than 0 and 1 that is true. Of course the questions explicitly stated that these were the needed values.
"This causes horrible runtime performance". What!? Can you produce any prove of that? There is a chance that the index generated will collide with an existing entry. This chance increases as more 1's are added. Worst case is there is a 19/80 (~23%) chance of collision.

If you know you need exactly 20 of one value, a better way to go about this is to pre-populate the array with the required values and then shuffle it.
Something like this should work:
int[] array = new int[80];
for (int i = 0; i < 80; i++)
{
int val = 0;
if (i < 20)
{
val = 1;
}
array[i] = val;
}
Random rnd = new Random();
int[] shuffledArray = array.OrderBy(x => rnd.Next()).ToArray();

You can do
for (int i = 0; i < 20; i++)
{
var rand = random.Next(0,80);
if (test[rand] == 1)
{
i--;
continue;
}
test[rand] = 1;
}
Remaining are automatically zero.

Random selection for a variable set with a guarantee of every item at least once

I am working on a game in c# but that detail is not really neccessary to solve my problem.
At I high level here is what I want:
I have a set that could have any number of items in it.
I want to randomly select 10 items from that set.
If the set has less than 10 items in then I expect to select the same
item more than once.
I want to ensure every item is selected at least once.
What would be the algorithm for this?
Sorry I'm not sure if this is an appropriate place to ask, but I've got no pen and paper to hand and I can't quite get my head round what's needed so appreciate the help.
In addition I might also want to add weights to the items to
increase/decrease chance of selection, so if you are able to
incorporate that into your answer that would be fab.
Finally thought I should mention that my set is actually a List<string>, which might be relevent if you prefer to give a full answer rather than psuedo code.

This is what I use to randomize an array. It takes an integer array and randomly sorts that list a certain amount of times determined by the random number (r).
private int[] randomizeArray(int[] i)
{
int L = i.Length - 1;
int c = 0;
int r = random.Next(L);
int prev = 0;
int curr = 0;
int temp;
while (c < r)
{
curr = random.Next(0, L);
if (curr != prev)
{
temp = i[prev];
i[prev] = i[curr];
i[curr] = temp;
c++;
}
}
return i;
}

If you look for effective code, my answer isnt it. In theory, create some collection you can remove from that will mirror your set. Then select random member of the object from it ...and remove, this will garantee items wont repeat(if possible).
Random rnd = new Random();
List<int> indexes = new List<int>(items.Count);
for (int i = 0; i < items.Count; i++)
indexes.Add(i);
List<string> selectedItems = new List<string>(10);
int tmp;
for(int i = 0; i < 10; i++)
{
tmp = rnd.Next(1,10000); //something big
if(indexes.Count > 0)
{
selectedItems.Add(yourItems[indexes[tmp%indexes.Count]]);
indexes.RemoveAt(tmp%indexes.Count);
}
else
selectedItems.Add(yourItems[rnd.Next(0,9)]); //you ran out of unique items
}
where items is your list and yourItems is list of selected items, you dont need to store them if you want process them right away

Perhaps shuffle the collection and pick elements from the front until you have the required amount.
Once you've gone through all the elements, you should perhaps shuffle it again, so you don't just repeat the same sequence.
The basic algorithm for shuffling: (in pseudo-code)
for i from n − 1 downto 1 do
j ← random integer with 0 ≤ j ≤ i
exchange a[j] and a[i]
With the above algorithm (or a minor variation), it's possible to just shuffle until you reach the required number of elements, no need to shuffle the whole thing.

Given a list of length n select k random elements using C#

I found this post:
Efficiently selecting a set of random elements from a linked list
But this means that in order to approach true randomness in the sample I have to iterate over all elements, throw them in memory with a random number, and then sort. I have a very large set of items here (millions) - is there a more efficient approach to this problem?

I would suggest simply shuffling elements as if you were writing a modified Fisher-Yates shuffle, but only bother shuffling the first k elements. For example:
public static void PartialShuffle<T>(IList<T> source, int count, Random random)
{
for (int i = 0; i < count; i++)
{
// Pick a random element out of the remaining elements,
// and swap it into place.
int index = i + random.Next(source.Count - i);
T tmp = source[index];
source[index] = source[i];
source[i] = tmp;
}
}
After calling this method, the first count elements will be randomly picked elements from the original list.
Note that I've specified the Random as a parameter, so that you can use the same one repeatedly. Be careful about threading though - see my article on randomness for more information.

Take a look at this extension method http://extensionmethod.net/csharp/ienumerable-t/shuffle. You could add Skip() Take() type to page the values out the final list.

If the elements can be in memory, put them in memory first
List<Element> elements = dbContext.Select<Element>();
Now you know the number of elements. Create a set of unique indexes.
var random = new Random();
var indexes = new HashSet<int>();
while (indexes.Count < k) {
indexes.Add(random.Next(elements.Count));
}
Now you can read the elements from the list
var randomElements = indexes.Select(i => elements[i]);
I assume that the DB contains unique elements. If this is not the case, you will have to create a HashSet<Elements> instead or to append .Distinct() when querying from the DB.
UPDATE
As Patricia Shanahan says, this method will work well if k is small compared to n. If it is not the case, I suggest selecting a set n - k indexes to be excluded
var random = new Random();
var indexes = new HashSet<int>();
IEnumerable<Element> randomElements;
if (k <= elements.Count / 2) {
while (indexes.Count < k) {
indexes.Add(random.Next(elements.Count));
}
randomElements = indexes.Select(i => elements[i]);
} else {
while (indexes.Count < elements.Count - k) {
indexes.Add(random.Next(elements.Count));
}
randomElements = elements
.Select((e,i) => indexes.Contains(i) ? null : elements[i])
.Where(e => e != null);
}

C# creating a list of random unique integers

I need to create a list with one billion integers and they must all be unique. I also need this to be done extremely fast.
Creating a list and adding random numbers one by one and checking to see if each one is a duplicate is extremely slow.
It seems to be quite fast if I just populate a list with random numbers without checking if they are duplicates and then using distinct().toList(). I repeat this until there are no more duplicates. However the extra memory used by creating a new list is not optimal. Is there a way to get the performance of distinct() but instead of creating a new list it just modifies the source list?

Do the integers need to be in a certain range? If so, you could create an array or list with all numbers in that range (for example from 1 to 1000000000) and shuffle that list.

I found this the fastest while maintaining randomness:
Random rand = new Random();
var ints = Enumerable.Range(0, numOfInts)
.Select(i => new Tuple<int, int>(rand.Next(numOfInts), i))
.OrderBy(i => i.Item1)
.Select(i => i.Item2);
...basically assigning a random id to each int and then sorting by that id and selecting the resulting list of ints.

You can track duplicates in a separate HashSet<int>:
var set = new HashSet<int>();
var nums = new List<int>();
while(nums.Count < 1000000000) {
int num;
do {
num = rand.NextInt();
} while (!set.Contains(num));
set.Add(num);
list.Add(num);
}
You need a separate List<int> to store the numbers because a hashset will not preserve your random ordering.

Taking the question literally (a list with one billion integers and they must all be unique):
Enumerable<int>.Range(0, 1000000000)
But along the lines of CodeCaster's answer, you can create the list and shuffle it at the same time:
var count = 1000000000;
var list = new List<int>(count);
var random = new Random();
list.Add(0);
for (var i = 1; i < count; i++)
{
var swap = random.Next(i - 1);
list.Add(list[swap]);
list[swap] = i;
}

If the amount of possible integers from which you draw is significantly larger (say factor 2) than the amount of integers you want you can simply use a HashSet<T> to check for duplicates.
List<int> GetUniqueRandoms(Random random, int count)
{
List<int> result = new List<int>(count);
HashSet<int> set = new HashSet<int>(count);
for(int i = 0; i < count; i++)
{
int num;
do
{
num = random.NextInt();
while(!set.Add(num));
result.Add(num);
}
return result;
}
This allocates the collections with the correct capacity to avoid reallocation during growth. Since your collections are large this should be a large improvement.
You can also use Distinct a single time:
IEnumerable<int> RandomSequence(Random random)
{
while(true)
{
yield return random.NextInt();
}
}
RandomSequence(rand).Distinct().Take(1000000000).ToList();
But with both solutions you need enough memory for a HashSet<int> and a List<int>.
If the amount of possible integers from which you draw is about as large as the amount of integers you want, you can create an array that contains all of them, shuffle them and finally cut off those you're not interested in.
You can use Jon Skeet's shuffle implementation.

What if you created the list in a sorted but still random fashion (such as adding a random number to the last element of the list as the next element), and then shuffled the list with a Fisher-Yates-Durstenfeld? That would execute in linear time overall, which is pretty much as good as it gets for list generation. However, it might have some significant bias that would affect the distribution.