We Keep Coding

Is there a more efficient way to generate 40 million alphanumeric unique random strings in C# [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 2 years ago.
Improve this question
I've created a windows-forms-based small application to generate random unique alphanumeric strings with length=8. Application is working fine with small count but it got stuck for like forever when I try to generate 40 million (as per my requirement) strings. Please help me to make it efficient so that the strings could be generated quickly.
following is the complete code I've used for it.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.Threading.Tasks;
namespace RandomeString
{
public partial class Form1 : Form
{
private const string Letters = "abcdefghijklmnpqrstuvwxyz";
private readonly char[] alphanumeric = (Letters + Letters.ToLower() + "abcdefghijklmnpqrstuvwxyz0123456789").ToCharArray();
private static Random random = new Random();
private int _ticks;
public Form1()
{
InitializeComponent();
}
private void button1_Click(object sender, EventArgs e)
{
if (string.IsNullOrEmpty(textBox1.Text) || string.IsNullOrWhiteSpace(textBox2.Text))
{
string message = "Please provide required length and numbers of strings count.";
string title = "Input Missing";
MessageBoxButtons buttons = MessageBoxButtons.OK;
DialogResult result = MessageBox.Show(message, title, buttons, MessageBoxIcon.Warning);
}
else
{
int ValuesCount;
ValuesCount = Convert.ToInt32(textBox2.Text);
for (int i = 1; i <= ValuesCount; i++)
{
listBox1.Items.Add(RandomString(Convert.ToInt32(textBox1.Text)));
}
}
}
public static string RandomString(int length)
{
const string chars = "abcdefghijklmnpqrstuvwxyz0123456789";
return new string(Enumerable.Repeat(chars, length)
.Select(s => s[random.Next(s.Length)]).ToArray());
}
private void button2_Click(object sender, EventArgs e)
{
try
{
StringBuilder sb = new StringBuilder();
foreach (object row in listBox1.Items)
{
sb.Append(row.ToString());
sb.AppendLine();
}
sb.Remove(sb.Length - 1, 1); // Just to avoid copying last empty row
Clipboard.SetData(System.Windows.Forms.DataFormats.Text, sb.ToString());
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
private void timer1_Tick(object sender, EventArgs e)
{
_ticks++;
this.Text = _ticks.ToString();
}
}
}

One way to speed things up is to avoid LINQ. For example, take a look at these two implementations:
public static string LinqStuff(int length)
{
const string chars = "abcdefghijklmnpqrstuvwxyz0123456789";
return new string(Enumerable.Repeat(chars, length)
.Select(s => s[random.Next(s.Length)]).ToArray());
}
public static string ManualStuff(int length)
{
const string chars = "abcdefghijklmnpqrstuvwxyz0123456789";
const int clength = 35;
var buffer = new char[length];
for(var i = 0; i < length; ++i)
{
buffer[i] = chars[random.Next(clength)];
}
return new string(buffer);
}
Running it through this:
private void TestThis(long iterations)
{
Console.WriteLine($"Running {iterations} iterations...");
var sw = new Stopwatch();
sw.Start();
for (long i = 0; i < iterations; ++i)
{
LinqStuff(20);
}
sw.Stop();
Console.WriteLine($"LINQ took {sw.ElapsedMilliseconds} ms.");
sw.Reset();
sw.Start();
for (long i = 0; i < iterations; ++i)
{
ManualStuff(20);
}
sw.Stop();
Console.WriteLine($"Manual took {sw.ElapsedMilliseconds} ms.");
}
With this:
TestThis(50_000_000);
Yielded these results:
LINQ took 28272 ms.
Manual took 9449 ms.
So by using LINQ, you increase the time it takes to generate strings by 3 times.
You could tweak this more and squeeze out a few more seconds, probably (for example, send in the same char[] buffer to each call)

Don't use linq
pre-allocate the memory
don't put it in to a UI control
use as many cores and threads as you can.
use direct memory.
Write the results to a file, instead of using the clipboard
This could likely be done quicker and even more efficiently, see notes. However, I can generate the chars in under 200ms
Note : Span<T> would give better results, however due to the lamdas it's just easier to take the small hit from fixed and use pointers
private const string Chars = "abcdefghijklmnpqrstuvwxyz0123456789";
private static readonly ThreadLocal<Random> _random =
new ThreadLocal<Random>(() => new Random());
public static unsafe void Do(byte[] array, int index)
{
var r = _random.Value;
fixed (byte* pArray = array)
{
var pLen = pArray + ((index + 1) * 1000000);
int i = 1;
for (var p = pArray + (index * 1000000); p < pLen; p++ ,i++)
if ((i % 9) == 0) *p = (byte)'\r';
else if ((i % 10) == 0) *p = (byte)'\n';
else *p = (byte)Chars[r.Next(35)];
}
}
public static async Task Main(string[] args)
{
var array = new byte[40000000 * ( 8 + 2)];
var sw = Stopwatch.StartNew();
Parallel.For(0, 39, (index) => Do(array, index));
Console.WriteLine(sw.Elapsed);
sw = Stopwatch.StartNew();
await using (var fs = new FileStream(#"D:\asdasd.txt", FileMode.Create,FileAccess.Write,FileShare.None, 1024*1024,FileOptions.Asynchronous|FileOptions.SequentialScan))
await fs.WriteAsync(array,0, array.Length);
Console.WriteLine(sw.Elapsed);
}
Output
00:00:00.1768141
00:00:00.4369418
Note 1 : I haven't really put much thought into this apart from the raw generation, obviously there are other considerations.
Note 2 : Also this will end up on the large object heap, so buyer beware. You would need to generate them straight to file so save this from ending up on the LOB
Note 3 : I give no guarantees about the random distribution, likely a different random number generator would be better overall
Note 4 : I used 40 index's because the math was easy, you would get slightly better results if you could match your threads to cores

How to add a pause between executing tasks in C#

I am currently writing a program which requires me to have a pause between executing tasks.
So I have 4 things.
Read Limit
Delay Between Each Read
Total Reads
Global delay (pause the program for 'x' seconds after a task is finished)
Basically, one task is considered the "Read Limit". So, for example, if I have these settings:
Read Limit (10)
Delay Between Each Read (20)
Total Reads (100)
Global Delay (30)
The program has to read 10 lines from the file based on "Read Limit" and between reading each line, there is a delay of 20 seconds based on "Delay Between Each Read". After it reads 10 lines, it is paused for 30 seconds based on "Global Delay". When the global delay is over, it starts again where it stopped and continues doing this until the limit of 100 is reached based on "Total Reads".
I have tried using System.Threading.Thread.Sleep() but I couldn't make it work. How can I achieve this with C#?
Thanks in advance.
//update with some of my code.
I load the file like this:
private void btnLoadFile_Click(object sender, EventArgs e)
{
OpenFileDialog ofd = new OpenFileDialog();
if (ofd.ShowDialog() == DialogResult.OK)
{
string[] lines = System.IO.File.ReadAllLines(ofd.FileName);
}
}
I have 4 global variables:
public int readLimit = 0;
public int delayBetweenRead = 0;
public int totalReads = 0;
public int globalDelay = 0;
public int linesRead = 0;
And I want to make the function like this:
private void doTask()
{
while (linesRead <= readLimit)
{
readLine(); // read one line
doDelay(); // delay between each line
readLine(); // read another line and so on, until readLimit or totalReads is reached
globalDelay(); // after readLimit is reached, call globalDelay to wait
linesRead++;
}
}

This might be of interest - here's the way to do this with Microsoft's Reactive Framework (NuGet "Rx-Main").
int readLimit = 10;
int delayBetweenRead = 20;
int globalDelay = 30;
int linesRead = 100;
var subscription =
Observable
.Generate(0, n => n < linesRead, n => n + 1, n => n,
n => TimeSpan.FromSeconds(n % readLimit == 0 ? globalDelay : delayBetweenRead))
.Zip(System.IO.File.ReadLines(ofd.FileName), (n, line) => line)
.Subscribe(line =>
{
/* do something with each line */
});
If you need to stop the reading before it finishes naturally just call subscription.Dispose();.

What you do you mean by
I have tried using System.Threading.Thread.Sleep() but I couldn't make it work
Here is an example of achieving what you described with Thread.Sleep:
using (var fs = new FileStream("C:\\test.txt", FileMode.Open, FileAccess.Read))
{
using (var sr = new StreamReader(fs))
{
int nRead = 0;
while (nRead < settings.Total)
{
for (int i = 0; i < settings.ReadLimit && nRead < settings.Total; ++i, nRead++)
{
Console.WriteLine(sr.ReadLine());
if (i + 1 < settings.ReadLimit)
{
Thread.Sleep(settings.Delay * 1000);
}
}
if (nRead < settings.Total)
{
Thread.Sleep(settings.GlobalDelay * 1000);
}
}
}
}

Problems reading and writing to a text file in C#

Ok so I have looked at every other question that relates to my problem and can't see where I am going wrong. My objective is to write the results of a dice simulator to a text file then read that contents into a listbox in C#. Here is my code so far:
namespace WpfApplication1
{
public MainWindow()
{
InitializeComponent();
}
private void button1_Click(object sender, RoutedEventArgs e)
{
try
{
Random RandomValue = new Random();
Random RandomValue2 = new Random();
using (StreamWriter outputFile = new StreamWriter("C:\\Example.txt"))
{
for (int i = 0; i < 100; i++)
{
int face1 = RandomValue.Next(1, 7);
int face2 = RandomValue2.Next(1, 7);
int toss = face1 + face2;
outputFile.WriteLine(toss);
}
outputFile.Close();
}
}
catch (Exception ex)
{
listBox1.Items.Add(ex);
}
}
private void button2_Click(object sender, RoutedEventArgs e)
{
try
{
using (StreamReader inputFile = new StreamReader("C:\\Example.txt"))
{
String toss;
for (int i = 0; i > 100; i++)
{
toss = inputFile.ReadLine();
listBox1.Items.Add(toss);
}
catch (Exception ex)
{
listBox1.Items.Add(ex);
}
}
}
}
}
When I try running the program I receive the error: System.UnauthorizedAccessException: Access to the path 'C:\Example.txt' is denied.
I think that where I am going wrong is actually creating the text file. When I try to read from it I receive the same message. I am extremely new to C# and have the feeling that I am not calling the file correctly. Any insight would be extremely welcome!

Alright everyone, first I want to thank you all for your help. I found where I was going wrong and why it was not working correctly. First, running C# as an administrator solved the access issues, thank you Selman22. After that it was writing to the file just fine but would not read the file and move the digits to the listbox. After reviewing the code more closely I realized I was declaring the for loop to run as long as I was greater than 100 rather than smaller:
for (int i = 0; i > 100; i++)
After changing the for statement to:
for (int i = 0; i < 100; i++)
It now runs smoothly. Thank you all again!

Trying to find correct way to infinitely loop in my code-behind file

I'm attempting to rotate through a series of numbered pictures sequentially in a C# code-behind section of a Master Content Page for a project I'm working with. I'm trying to start at 1, move to 5, and then start the process again. This is what I have so far. It doesn't work because you need additional conditions in the if section.
int i = 1;
restart:
while (i < 6)
{
Image1.ImageUrl = "~/Images/" + i.ToString() + ".png";
i++;
if (i == 5)
{
restart;
}
}
EDIT:
After trying to prepend "GOTO" to the label, I can't get the page to load. Here is the rest of the file:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
namespace Clark
{
public partial class Home : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
if (!IsPostBack)
{
SetImageURL(); // To ensure image loads on page load
}
}
protected void Timer1_Tick(object sender, EventArgs e)
{
SetImageURL(); // Refactored from previous logic
}
private void SetImageURL()
{
// The random function will choose between 1&5 which
// correspond to images 1 through 5
//Random _rand = new Random();
//int i = _rand.Next(1, 5);
//Image1.ImageUrl = "~/Images/" + i.ToString() + ".png";
int i = 1;
restart:
while (i < 6)
{
Image1.ImageUrl = "~/Images/" + i.ToString() + ".png";
i++;
if (i == 5)
{
restart;
}
}
/*
while (true)
{
int i = 1;
do
{
Image1.ImageUrl = "~/Images/" + i.ToString() + ".png";
i++;
} while (i < 6);
} */
}
}
}

Just use nested loops.
do
{
for (int i = 1; i < 6; i++;
{
Image1.ImageUrl = "~/Images/" + i.ToString() + ".png";
}
}
while (/* additional conditions */)
I chose the do/while architecture because it maps best to what you already have, but just a plain-ol' while would do the trick too.
Although that all said, remember that unless this is just an example, this is a pretty silly thing to do. Your ImageUrl won't be read while you're in this loop, so you could just calculate out what the result will actually be and not have to waste the time and resources to run what could turn into an infinite loop like this.
Edit:
The question has evolved.
Instead of using loops, store an Application variable:
object imageIndexObj = Application["imageIndex"];
int imageIndex;
if(imageIndexObj == null)
imageIndex = 1;
else
imageIndex = (int)imageIndexObj;
Image1.ImageUrl = "~/Images/" + (imageIndex % 5) + ".png";
Application["imageIndex"] = imageIndex + 1;
There are some implications here, particularly when dealing with multiple-server farms. But this should suffice for your case. If you hit a user-load wherein multiple servers are required, then using a workflow like this will likely become irrelevant, since the result will be more random anyway.
You could also use Session in place of Application here, if you want the five images to cycle for each session, rather than for each request. That's your call. The code's the same other than that, though.

Speedup my application

I am creating a file encryption application using Advanced Encryption Standard Algorithm, but I find that my application is so slow especially when I am encrypting a large amount of data,
ex. 80MB of file size, 30 mins. was already past but my application is not yet done encrypting my file with the size of 80mb.
I am using ECB(Electronic Code Book) mode in my Encryption Algorithm
how can I speed up my application in encrypting a large amount of data? I do some research and I find this, http://en.wikipedia.org/wiki/Speedup, but I'm not sure if this is the answer to my problem... or is it effective if I will used BackgroundWorker? by the way, I am using Visual Studio 2008 in the development of my project.
here is my codes in encrypting a file..
private void cmdEncrypt_Click(object sender, EventArgs e)
{
AESECB aes = new AESECB();
FileInfo fInfo = new FileInfo(txtFileSource.Text);
if (txtFileSource.Text == "")
{
MessageBox.Show("Please Select a File", "Message", MessageBoxButtons.OK, MessageBoxIcon.Warning);
return;
}
if (txtSecretKey.Text == "")
{
MessageBox.Show("Please Enter the password", "Message", MessageBoxButtons.OK, MessageBoxIcon.Warning);
return;
}
if (fInfo.Exists == false)
{
MessageBox.Show("File Not Found!", "Message", MessageBoxButtons.OK, MessageBoxIcon.Warning);
return;
}
byte[] bytePadding = aes.filePadding(txtFileSource.Text);
byte[] fByte = aes.getFileByte(txtFileSource.Text);
int farrLength = (bytePadding.Length + fByte.Length);
byte[] newFbyte = new byte[farrLength];
byte[] encryptedFByte = new byte[farrLength];
int counterBytePadding =0;
byte firstByte = 0;
for (int i = 0; i < farrLength; i++)
{
if (i < fByte.Length)
{
newFbyte[i] = fByte[i];
}
else
{
newFbyte[i] = bytePadding[counterBytePadding];
counterBytePadding++;
}
}
int plainFileBlock = newFbyte.Length / 16;
progressBar1.Maximum = plainFileBlock-1;
progressBar1.Visible = true;
int counter = 0;
int counter2 = 0;
for (int j = 0; j < plainFileBlock; j++)
{
byte[] encfbyte = aes.fileEncrypt(txtSecretKey.Text, newFbyte, counter);
for (int k = 0; k < 16; k++)
{
encryptedFByte[counter2] = encfbyte[k];
counter2++;
}
progressBar1.Value = j;
counter = counter + 16;
}
progressBar1.Visible = false;
int bytesToRead = encryptedFByte.Length;
string newPath = txtFileSource.Text + ".aesenc";
using (FileStream newFile = new FileStream(newPath, FileMode.Create, FileAccess.Write))
{
newFile.Write(encryptedFByte, 0, bytesToRead);
}
MessageBox.Show("Encryption Done!", "Message", MessageBoxButtons.OK, MessageBoxIcon.Information);
}

It's all very well being "confident in your codes", but all good programmers measure.
This is simple, do this:
using System.Diagnostics;
var startTime = DateTime.Now;
//some code here
Debug.WriteLine("This took {0}", DateTime.Now.Subtract(startTime));
Then look at your output window in VS (View->Output).
By wrapping different parts of the method with these two lines you will identify the slow bit.
My suspicions are on the loop where you copy 80MB byte by byte. Try Array.Resize here.

As suggested by weston, do measure your code in order to identify what is being slow. That said, it is probably the file encryption for loop which is slowing you down. If so you can definitely speed up the process by using more than 1 cpu. Look up the parallel constructs in C#: "Parallel.For".
Here is a simple example:
http://www.dotnetcurry.com/ShowArticle.aspx?ID=608