We Keep Coding

Simulating message latency accurately

I have written a simple "latency simulator" which works, but at times, messages are delayed for longer than the time specified. I need help to ensure that messages are delayed for the correct amount of time.
The main problem, I believe, is that I am using Thread.Sleep(x), which is depended on various factors but mainly on the clock interrupt rate, which causes Thread.Sleep() to have a resolution of roughly 15ms. Further, intensive tasks will demand more CPU time and will occasionally result in a delay greater than the one requested. If you are not familiar with the resolution issues of Thread.Sleep, you can read these SO posts: here, here and here.
This is my LatencySimulator:
public class LatencySimulatorResult: EventArgs
{
public int messageNumber { get; set; }
public byte[] message { get; set; }
}
public class LatencySimulator
{
private int messageNumber;
private int latency = 0;
private int processedMessageCount = 0;
public event EventHandler messageReady;
public void Delay(byte[] message, int delay)
{
latency = delay;
var result = new LatencySimulatorResult();
result.message = message;
result.messageNumber = messageNumber;
if (latency == 0)
{
if (messageReady != null)
messageReady(this, result);
}
else
{
ThreadPool.QueueUserWorkItem(ThreadPoolCallback, result);
}
Interlocked.Increment(ref messageNumber);
}
private void ThreadPoolCallback(object threadContext)
{
Thread.Sleep(latency);
var next = (LatencySimulatorResult)threadContext;
var ready = next.messageNumber == processedMessageCount + 1;
while (ready == false)
{
ready = next.messageNumber == processedMessageCount + 1;
}
if (messageReady != null)
messageReady(this, next);
Interlocked.Increment(ref processedMessageCount);
}
}
To use it, you create a new instance and bind to the event handler:
var latencySimulator = new LatencySimulator();
latencySimulator.messageReady += MessageReady;
You then call latencySimulator.Delay(someBytes, someDelay);
When a message has finished being delayed, the event is fired and you can then process the delayed message.
It is important that the order in which messages are added is maintained. I cannot have them coming out the other end of the latency simulator in some random order.
Here is a test program to use the latency simulator and to see how long messages have been delayed for:
private static LatencySimulator latencySimulator;
private static ConcurrentDictionary<int, PendingMessage> pendingMessages;
private static List<long> measurements;
static void Main(string[] args)
{
var results = TestLatencySimulator();
var anomalies = results.Result.Where(x=>x > 32).ToList();
foreach (var result in anomalies)
{
Console.WriteLine(result);
}
Console.ReadLine();
}
static async Task<List<long>> TestLatencySimulator()
{
latencySimulator = new LatencySimulator();
latencySimulator.messageReady += MessageReady;
var numberOfMeasurementsMax = 1000;
pendingMessages = new ConcurrentDictionary<int, PendingMessage>();
measurements = new List<long>();
var sendTask = Task.Factory.StartNew(() =>
{
for (var i = 0; i < numberOfMeasurementsMax; i++)
{
var message = new Message { Id = i };
pendingMessages.TryAdd(i, new PendingMessage() { Id = i });
latencySimulator.Delay(Serialize(message), 30);
Thread.Sleep(50);
}
});
//Spin some tasks up to simulate high CPU usage
Task.Factory.StartNew(() => { FindPrimeNumber(100000); });
Task.Factory.StartNew(() => { FindPrimeNumber(100000); });
Task.Factory.StartNew(() => { FindPrimeNumber(100000); });
sendTask.Wait();
return measurements;
}
static long FindPrimeNumber(int n)
{
int count = 0;
long a = 2;
while (count < n)
{
long b = 2;
int prime = 1;// to check if found a prime
while (b * b <= a)
{
if (a % b == 0)
{
prime = 0;
break;
}
b++;
}
if (prime > 0)
{
count++;
}
a++;
}
return (--a);
}
private static void MessageReady(object sender, EventArgs e)
{
LatencySimulatorResult result = (LatencySimulatorResult)e;
var message = (Message)Deserialize(result.message);
if (pendingMessages.ContainsKey(message.Id) != true) return;
pendingMessages[message.Id].stopwatch.Stop();
measurements.Add(pendingMessages[message.Id].stopwatch.ElapsedMilliseconds);
}
static object Deserialize(byte[] arrBytes)
{
using (var memStream = new MemoryStream())
{
var binForm = new BinaryFormatter();
memStream.Write(arrBytes, 0, arrBytes.Length);
memStream.Seek(0, SeekOrigin.Begin);
var obj = binForm.Deserialize(memStream);
return obj;
}
}
static byte[] Serialize<T>(T obj)
{
BinaryFormatter bf = new BinaryFormatter();
using (var ms = new MemoryStream())
{
bf.Serialize(ms, obj);
return ms.ToArray();
}
}
If you run this code, you will see that about 5% of the messages are delayed for more than the expected 30ms. In fact, some are as high as 60ms. Without any background tasks or high CPU usage, the simulator behaves as expected.
I need them all to be 30ms (or as close to as possible) - I do not want some arbitrary 50-60ms delays.
Can anyone suggest how I can refactor this code so that I can achieve the desired result, but without the use of Thread.Sleep() and with as little CPU overhead as possible?

how to measure execution time in specific part of View [duplicate]

This question already has answers here:
Calculate the execution time of a method
(8 answers)
Closed 5 years ago.
I want to calculate execution time in a specific part of View in MVC asp.net
for example
<p>start time is A</p>
<br />
#for (int i = 0; i < 1000000; i++)
{
if (#i % 100000 == 0)
{
<p>#i</p> <br />
}
}
<p>End time is B</p>
<p>execution time is: C</p>
how to calculate A, B, C in above code ?

What I would recommend you to do here is to create an instance of Stopwatch by Stopwatch.StartNew, execute your method or your code that you want to calculate execution time. Stop the Stopwatch where you think you method should be done with execution, and then get ElapsedMilliseconds like I will do in code below:
#{
Stopwatch stopwatch = Stopwatch.StartNew();
for (int i = 0; i < 1000000; i++)
{
if (i % 100000 == 0)
{
#Html.Raw("<p>" + i.ToString() + "</p> <br />");
}
}
stopwatch.Stop();
#Html.Raw("<p>Elapsed time is: " + stopwatch.ElapsedMilliseconds.ToString() + "</p>");
}
And please, in case you might get an Error while you are trying to execute this code, be sure that you've included System.Diagnostics;
P.S If you don't like stopwatch.ElapsedMilliseconds you might check also for stopwatch.Elapsed to display elapsed time..
As I can see after Alexander's comment that you want start time, end time etc, what I suggest you to do here is to create a custom Stopwatch by inheriting System.Diagnostics.Stopwatchclass and extending it with a couple of DateTime properties.
And I will now show you how you might do it:
public class MyCustomStopWatch : Stopwatch
{
public DateTime? StartAt { get; private set; }
public DateTime? EndAt { get; private set; }
public void Reset()
{
StartAt = null;
EndAt = null;
base.Reset();
}
public void Restart()
{
StartAt = DateTime.Now;
EndAt = null;
base.Restart();
}
//This is what is important to you right now, to get data about that when you code started, and when your code finished with executing
public void Start()
{
StartAt = DateTime.Now;
base.Start();
}
public void Stop()
{
EndAt = DateTime.Now;
base.Stop();
}
}
If you are wondering HOW COULD I USE THAT CODE?
DON'T WORRY, HERE IS AN EXAMPLE BASED ON YOUR QUESTION:
#{
MyCustomStopwatch stopwatch = MyCustomStopwatch();
for (int i = 0; i < 1000000; i++)
{
if (i % 100000 == 0)
{
#Html.Raw("<p>" + i.ToString() + "</p> <br />");
}
}
stopwatch.Stop();
#Html.Raw(String.Format("<p>Stopwatch elapsed: {0}, StartAt: {1}, EndAt: {2} </p>", stopwatch.ElapsedMilliseconds.ToString(), stopwatch.StartAt.Value.ToString(), stopwatch.EndAt.Value.ToString());
}
Whatever I hope that you catch up the point and I hope that this helped you to solve your problem

You can do it like this:
#{
DateTime StartTime = DateTime.Now;
System.Diagnostics.StopWatch Watch = new System.Diagnostics.StopWatch();
#Html.Raw("<p>start time is: " + StartTime.ToString() + "</p>");
Watch.Start();
for (int i = 0; i < 1000000; i++)
{
if (i % 100000 == 0)
{
#Html.Raw("<p>" + i.ToString() + "i</p> <br />");
}
}
Watch.Stop();
DateTime EndTime = DateTime.Now;
#Html.Raw("<p>End time is " + EndTime.ToString() + "</p>");
#Html.Raw("<p>Execution time is: " + Watch.Elapsed.ToString() + "</p>");
}
Normally, I would recommend using a StopWatch to track the elapsed time but since you want the start and end times you might as well captured two dates and display the difference to calculate the time span.

Multithreaded WebRequests being executed in succession

I am building a web scraper in C# that deals with proxies and a large volume of requests. The pages are loaded through a ConnectionManager class that grabs a proxy and retries loading that page with random proxies until the page is correctly loaded.
On average, a single task will take somewhere between 100 and 300 requests, and to speed up the process, I have designed the method to use multithreading to simultaneously download the webpages.
public Review[] getReviewsMultithreaded(int reviewCount)
{
ArrayList reviewList = new ArrayList();
int currentIndex = 0;
int currentPage = 1;
int totalPages = (reviewCount / 10) + 1;
bool threadHasMoreWork = true;
Object pageLock = new Object();
Thread[] threads = new Thread[Program.maxScraperThreads];
for(int i = 0; i < Program.maxScraperThreads; i++)
{
threads[i] = (new Thread(() =>
{
while (threadHasMoreWork)
{
HtmlDocument doc;
lock(pageLock)
{
if (currentPage <= totalPages)
{
string builtString = "http://www.example.com/reviews/" + _ID + "?pageNumber=" + currentPage;
//Log.WriteLine(builtString);
currentPage++;
doc = Program.conManager.loadDocument(builtString);
}
else
{
threadHasMoreWork = false;
continue;
}
}
try
{
//Get info from page and add to list
reviewList.Add(cRev);
}
Log.WriteLine(_asin + " reviews scraped: " + reviewList.Count);
}
catch (Exception ex) { continue; }
}
}));
threads[i].Start();
}
bool threadsAreRunning = true;
while(threadsAreRunning) //this is in a separate thread itself, so as not to interrupt the GUI
{
threadsAreRunning = false;
foreach (Thread t in threads)
if (t.IsAlive)
{
threadsAreRunning = true;
Thread.Sleep(2000);
}
}
//flatten the arraylist to a primitive
return reviewArray;
}
However, I have noticed that the requests are still largely being handled one at a time, and as a result the method isn't much faster than it was before. Is the lock causing problems? Is the fact that the ConnectionManager is instantiated in one object and each thread is calling the loadDocument from the same object?

Ah, nevermind. I noticed the lock included the call to the method that loads the pages, and because of that only one page was loading at a time.

Thread program, 1st and last loop never has a ThreadedState other than 'Running'

I have a programming that is looping x times (10), and using a specified number of threads (2). I'm using a thread array:
Thread[] myThreadArray = new Thread[2];
My loop counter, I believe, starts the first 2 threads just fine, but when it gets to loop 3, which goes back to thread 0 (zero-based), it hangs. The weird thing is, if I throw a MessageBox.Show() in their to check the ThreadState (which shows thread 0 is still running), it will continue on through 9 of the 10 loops. But if no MessageBox.Show() is there, it hangs when starting the 3rd loop.
I'm using .NET 3.5 Framework (I noticed that .NET 4.0 utilizes something called continuations...)
Here's some code examples:
Thread[] threads = new Thread[2];
int threadCounter = 0;
for (int counter = 0; counter < 10; counter++)
{
if (chkUseThreading.Checked)
{
TestRunResult runResult = new TestRunResult(counter + 1);
TestInfo tInfo = new TestInfo(conn, comm, runResult);
if (threads[threadCounter] != null)
{
// If this is here, then it will continue looping....otherwise, it hangs on the 3rd loop
MessageBox.Show(threads[threadCounter].ThreadState.ToString());
while (threads[threadCounter].IsAlive || threads[threadCounter].ThreadState == ThreadState.Running)
Thread.Sleep(1);
threads[threadCounter] = null;
}
// ExecuteTest is a non-static method
threads[threadCounter] = new Thread(new ThreadStart(delegate { ExecuteTest(tInfo); }));
threads[threadCounter].Name = "PerformanceTest" + (counter + 1);
try
{
threads[threadCounter].Start();
if ((threadCounter + 1) == threadCount)
threadCounter = 0;
else
threadCounter++;
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
Application.DoEvents();
}
}
while (true)
{
int threadsFinished = 0;
for (int counter = 0; counter < threadCount; counter++)
{
if (!threads[counter].IsAlive || threads[counter].ThreadState == ThreadState.Stopped)
threadsFinished++;
}
if (threadsFinished == threadCount)
break;
else
Thread.Sleep(1);
}
Obviously the problem is something about how I'm checking to see if thread #1 or #2 is done. The IsAlive always says true, and the ThreadState always has "running" for threads loops 1 and 10.
Where am I going wrong with this?
Update, here's the ExecuteTask() method:
private void ExecuteTest(object tInfo)
{
TestInfo testInfo = tInfo as TestInfo;
Exception error = null;
DateTime endTime;
TimeSpan duration;
DateTime startTime = DateTime.Now;
try
{
if (testInfo.Connection.State != ConnectionState.Open)
{
testInfo.Connection.ConnectionString = connString;
testInfo.Connection.Open();
}
testInfo.Command.ExecuteScalar();
}
catch (Exception ex)
{
error = ex;
failedCounter++;
//if (chkCancelOnError.Checked)
// break;
}
finally
{
endTime = DateTime.Now;
duration = endTime - startTime;
RunTimes.Add(duration);
testInfo.Result.StartTime = startTime;
testInfo.Result.EndTime = endTime;
testInfo.Result.Duration = duration;
testInfo.Result.Error = error;
TestResults.Add(testInfo.Result);
// This part must be threadsafe...
if (lvResults.InvokeRequired)
{
SetTextCallback d = new SetTextCallback(ExecuteTest);
this.Invoke(d, new object[] { tInfo });
}
else
{
lvResults.Items.Add(testInfo.Result.ConvertToListViewItem());
#region Update Results - This wouldn't work in it's own method in the threaded version
const string msPrefix = "ms";
// ShortestRun
TimeSpan shortest = GetShortestRun(RunTimes);
tbShortestRun.Text = shortest.TotalMilliseconds + msPrefix;
// AverageRun
TimeSpan average = GetAverageRun(RunTimes);
tbAverageRun.Text = average.TotalMilliseconds + msPrefix;
// MeanRun
TimeSpan mean = GetMeanRun(RunTimes);
tbMeanRun.Text = mean.TotalMilliseconds + msPrefix;
// LongestRun
TimeSpan longest = GetLongestRun(RunTimes);
tbLongestRun.Text = longest.TotalMilliseconds + msPrefix;
// ErrorCount
int errorCount = GetErrorCount(TestResults);
tbErrorCount.Text = errorCount.ToString();
#endregion
}
testInfo.Command.Dispose();
Application.DoEvents();
}
}

Can you post a snippet of run ()? Doesn't Thread.currentThread().notifyAll() help? May be each thread is waiting for other thread to do something resulting in a deadlock?

Requesting direction for loading my file in chunks (Only needing advice)

I've been working for the last few days on a method to compress 144 million tile representation for my xna game down to a very small size when saved. Having managed to pull that off I now find myself stumped on how to go about getting them back from the file in chunks.
In the file I have.
An integer (it gets compressed to bytes using the 7BitEncodedInt method)
A byte
The compressed integer represents the number of tiles and the byte that follows determines what type the tiles are. This is all well and good and works really well. Most importantly it shrinks the file size down to just 50mb on average.
The problem is that I am currently reading back the entire file.
From the file I'm getting this.
The index value of each tile (just a basic iteration as I grab the tiles)
The type for each tile as a byte value
A byte value representing a texture for that tile (this is hard to explain but its necessary on a per tile basis)
The end result of all this is that I'm managing to save the file and only use about 50mb. But by loading the whole thing back in it expands out to nearly 1.5gigs on the ram. I can't really afford to sacrifice anymore tile info. so I need a way to only load portions of the map based on the player location. The goal is to be around the 100-200mb range
I have been looking at memory mapping the file, using quadtrees, pretty much anything I could find for loading files in chunks. While these options all seem pretty good I'm not sure which is best or if given the situation there may be another even better one. The other problem with all this is that these solutions all seem very involved (especially since this is my first time using them) and while I'm not against devoting myself to some lengthy coding I'd like to know that its gonna do what I need it to before hand.
My question is, given how I have to process the file as I pull it in and the fact that it needs to be done based on the players location what would be the best way to do this ? I'm just looking for some direction here. Code is always welcome but not required.

You want to have fixed length variables in your Tile class and implement something like such:
This is an example of a collection class (People) that can get a value based on index from a collection that was serialised into a file.
Person is the class that is the base of the People collection.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace FileStreamDatabaseTest
{
class Program
{
static void Main(string[] args)
{
People.OpenCollection();
People.Test_WillOverwriteData();
People.CloseCollection();
Console.ReadLine();
}
}
public class Person
{
// define maxium variable sizes for serialisation
protected static int pMaxLength_FirstName = 64;
protected static int pMaxLength_Age = 10;
public static int MaxObjectSize
{
get
{
// return the sum of all the maxlegnth variables to define the entire object size for serialisation
return pMaxLength_FirstName + pMaxLength_Age;
}
}
// define each object that will be serialised as follows
protected string pFirstName;
public string Firstname
{
set
{
// ensure the new value is not over max variable size
if (value.Length > pMaxLength_FirstName)
throw new Exception("the length of the value is to long.");
pFirstName = value;
}
get
{
return pFirstName;
}
}
protected int pAge;
public int Age
{
get
{
return pAge;
}
set
{
pAge = value;
}
}
public byte[] Serialise()
{
// Output string builder
StringBuilder Output = new StringBuilder();
// Append firstname value
Output.Append(Firstname);
// Add extra spaces to end of string until max length is reached
if (Firstname.Length < pMaxLength_FirstName)
for (int i = Firstname.Length; i < pMaxLength_FirstName; i++)
Output.Append(" ");
// Append age value as string
Output.Append(Age.ToString());
// Add extra spaces to end of string until max length is reached
int AgeLength = Age.ToString().Length;
if (AgeLength < pMaxLength_Age)
for (int i = AgeLength; i < pMaxLength_Age; i++)
Output.Append(" ");
// Return the output string as bytes using ascii encoding
return System.Text.Encoding.ASCII.GetBytes(Output.ToString());
}
public void Deserialise(byte[] SerialisedData)
{
string Values = System.Text.Encoding.ASCII.GetString(SerialisedData);
pFirstName = Values.Substring(0, pMaxLength_FirstName).Trim();
pAge = int.Parse(Values.Substring(pMaxLength_FirstName, pMaxLength_Age).Trim());
}
}
public static class People
{
private static string tileDatasource = #"c:\test.dat";
private static System.IO.FileStream FileStream;
public static void OpenCollection()
{
FileStream = new System.IO.FileStream(tileDatasource, System.IO.FileMode.OpenOrCreate, System.IO.FileAccess.ReadWrite, System.IO.FileShare.None);
}
public static void CloseCollection()
{
FileStream.Close();
FileStream.Dispose();
FileStream = null;
}
public static void SaveCollection(Person[] People)
{
FileStream.SetLength(People.Length * Person.MaxObjectSize);
FileStream.Position = 0;
foreach (Person PersonToWrite in People)
{
// call serialise to get bytes
byte[] OutputBytes = PersonToWrite.Serialise();
// write the output buffer
// note: this will always be the same size as each variable should
// append spaces until its max size is reached
FileStream.Write(OutputBytes, 0, OutputBytes.Length);
}
}
public static Person GetValue(int Index)
{
// set the stream position to read the object by multiplying the requested index with the max object size
FileStream.Position = Index * Person.MaxObjectSize;
// read the data
byte[] InputBytes = new byte[Person.MaxObjectSize];
FileStream.Read(InputBytes, 0, Person.MaxObjectSize);
// deserialise
Person PersonToReturn = new Person();
PersonToReturn.Deserialise(InputBytes);
// retun the person
return PersonToReturn;
}
public static void Test_WillOverwriteData()
{
long StartTime;
long EndTime;
TimeSpan TimeTaken;
Console.WriteLine("-------------------------------------------------------------------");
Console.WriteLine("*** Creating 2,000,000 test people... ");
StartTime = DateTime.Now.Ticks;
Person[] People = new Person[2000000];
for (int i = 0; i < 2000000; i++)
{
People[i] = new Person();
People[i].Firstname = "TestName." + i;
People[i].Age = i;
}
EndTime = DateTime.Now.Ticks;
TimeTaken = new TimeSpan(EndTime - StartTime);
Console.WriteLine("-> Completed in " + TimeTaken.TotalSeconds + " seconds");
Console.WriteLine("-------------------------------------------------------------------");
Console.WriteLine("*** Serialising Collection to disk... ");
StartTime = DateTime.Now.Ticks;
SaveCollection(People);
EndTime = DateTime.Now.Ticks;
TimeTaken = new TimeSpan(EndTime - StartTime);
Console.WriteLine("-> Completed in " + TimeTaken.TotalSeconds + " seconds");
Console.WriteLine("-------------------------------------------------------------------");
Console.WriteLine("*** Redundancy Test... ");
StartTime = DateTime.Now.Ticks;
bool Parsed = true;
int FailedCount = 0;
for (int i = 0; i < 2000000; i++)
{
if (GetValue(i).Age != i)
{
Parsed = false;
FailedCount++;
}
}
EndTime = DateTime.Now.Ticks;
TimeTaken = new TimeSpan(EndTime - StartTime);
Console.WriteLine("-> " + (Parsed ? "PARSED" : "FAILED (" + FailedCount + " failed index's"));
Console.WriteLine("-> Completed in " + TimeTaken.TotalSeconds + " seconds");
Console.WriteLine("-------------------------------------------------------------------");
Console.WriteLine("*** Reading 10,000 index's at once... ");
StartTime = DateTime.Now.Ticks;
Person[] ChunkOfPeople = new Person[10000];
for (int i = 0; i < 10000; i++)
ChunkOfPeople[i] = GetValue(i);
EndTime = DateTime.Now.Ticks;
TimeTaken = new TimeSpan(EndTime - StartTime);
Console.WriteLine("-> Completed in " + TimeTaken.TotalSeconds + " seconds");
Console.WriteLine("-------------------------------------------------------------------");
Console.WriteLine("*** Reading 100,000 index's at once... ");
StartTime = DateTime.Now.Ticks;
ChunkOfPeople = new Person[100000];
for (int i = 0; i < 100000; i++)
ChunkOfPeople[i] = GetValue(i);
EndTime = DateTime.Now.Ticks;
TimeTaken = new TimeSpan(EndTime - StartTime);
Console.WriteLine("-> Completed in " + TimeTaken.TotalSeconds + " seconds");
Console.WriteLine("-------------------------------------------------------------------");
Console.WriteLine("*** Reading 1,000,000 index's at once... ");
StartTime = DateTime.Now.Ticks;
ChunkOfPeople = new Person[1000000];
for (int i = 0; i < 1000000; i++)
ChunkOfPeople[i] = GetValue(i);
EndTime = DateTime.Now.Ticks;
TimeTaken = new TimeSpan(EndTime - StartTime);
Console.WriteLine("-> Completed in " + TimeTaken.TotalSeconds + " seconds");
}
}
}

There are a number of options, not all of them may be appropriate for your particular project:
Don't use a single file for all the data. Divide the map in smaller "rooms" and store each one in its own file. Load only the "room" the player starts in and preemptively load neighboring "rooms" and unload old ones.
Reduce the number of tiles you need to store. Use procedural generation to create an area's layout.
If you have a 10x10 room with the floor made of a single tile type then don't store 100 separate tiles but instead use a specific marker that says "this area has a 10x10 floor with this tile". If it's a wall then save the start and end positions and the texture type. If you have a multi-tile doodad in the middle of an open field and it's position is not relevant to the story then position it randomly in the field (and save the seed for the random number generator in the map file so next time it will appear in the same place).