We Keep Coding

Approach to solve a not accurate replay system

Trying to make accurate replay system in unity and c#
Hi all,
Im working on a racing game and I decided to add a replay system to allow "ghost car" too, initially I was recordng data in some events like key pressed but only recording that data in all frames I manage a smooth replay, well its still ok as file is not huge and replay works, but the trouble is there is always a slight variation in time, only like 0.1 seconds or 0.2 at the most, I have a list of keyframes and in each position I record a time to be shown, the trouble I think is that because fps vary then not in all runs the same time marks are shown then the winning frame's time is not always being rendered so the winning frame happens in next update slightly after it should be shown. Im using c# and unity just in case, but I think its independent to this mainly. Thanks a lot about any clue, I have been around this issue for some time now

It sounds like you're doing frame-by-frame replay which, as you've discovered, requires your frames to play back with the same delay as the recording. In a game-render loop, that's not guaranteed to happen.
As you record the car states (position, heading, etc) per frame, you need to also record a timestamp (in this case, accumulating Time.deltaTime from race start should suffice).
When you play it back, find the current timestamp and interpolate (ie, Lerp) the car's state from the recorded bounding frames.
Hints for frame interpolation:
class Snapshot {
public float Timestamp;
public Matrix4x4 Transform; // As an example. Put more data here.
}
private int PrevIndex = 0;
private List<Snapshot> Snapshots = (new List<Snapshot>()).OrderBy(m => m.Timestamp).ToList();
private float GetLerpFactor(float currentTimestamp) {
if ( PrevIndex == Snapshots.Count - 1)
return 0; // Reached end of Snapshots
while (currentTimestamp >= Snapshots[PrevIndex + 1].Timestamp)
PrevIndex++; // move 'frame' forward
var currentDelta = Mathf.Max(0f, currentTimestamp - Snapshots[PrevIndex].Timestamp);
var fullDelta = Snapshots[PrevIndex + 1].Timestamp - Snapshots[PrevIndex].Timestamp;
var lerpFactor = currentDelta / fullDelta;
return lerpFactor;
}

Unless for some reason you need to interact with "ghost" car (like collisions) record final data on position/speed/direction at frequent enough original moments of time and interpolate that to new simulation. I would not record raw inputs but rather resulting changes (like gear shifts) unless you need to measure/show how fast user reacted to something.
If you really want to replay the same inputs you'd have to run two separate simulations at the same time so physics and timing of "real" version don't impact "ghost" one, most likely you'll have to again interpolate output of "ghost" simulation to align with real one (unless you have fixed time steps).

Animation frame rate vs game fps

I'm a beginner programmer just starting out with MonoGame. I was implementing some code for running an animation and came to a problem I don't know how to solve. This is the code I used in my AnimationManager class:
public void Update(GameTime gameTime)
{
timeElapsed += (float)gameTime.ElapsedGameTime.TotalSeconds;
if (timeElapsed > timeToUpdate)
{
timeElapsed -= timeToUpdate;
//Tried to just reset timeElapsed to zero, still doesn't work
//timeElapsed = 0;
if (frameIndex < rectangles.Count -1)
{
frameIndex++;
}
else if (isLooping)
{
frameIndex = 0;
}
}
}
The problem is I animate at 24 frames per second and when the game runs at 60 or 30 fps and on each update code checks is it time to draw a new frame of animation, you get some remainder of frames, because you tried to draw 24 images per second, evenly, with 30 even game frames per second. So there's that remainder of 6 frames. The result of this is that approx 6 frames of animation gets drawn twice, some are skipped a whole animation gets about 25% longer. Is there a way to fix this? Can I maybe move this code to Draw call and then cap just the draw calls at 48 fps for 60 fps game, so each animation frame will be drawn at every second draw call. I don't know how would I go about doing this if it's a reasonable solution at all?
And how will VSync affect all of this at the end. Will it mess up animation frames, even if this problem is solved.
EDIT: forgot to say it's a 2d game, hand drawn animation.

Generally when you're doing this kind of stuff you don't need to worry about VSync or the fact that your game runs at 60 or 30 fps. That's all pretty much irrelevant.
The thing that matters is that you know the desired frames per second of your animation and how long it takes for a single game frame.
From that you can calculate the length of time in a single animation frame. For example, you'd end up with something like this:
var animationFrameTime = 1f / 24f; // 24 frames per second
So far so good.
The next thing you need to do is to start accumulating time somewhere. There's a few different ways to think about this. I like to think about it in terms of how much time is left until the next frame.
var gameFrameTime = (float)gameTime.ElapsedGameTime.TotalSeconds;
timeUntilNextFrame -= gameFrameTime;
Then you need to increment the frameIndex when you've run out of frame time.
if(timeUntilNextFrame <= 0)
{
frameIndex++;
timeUntilNextFrame += animationFrameTime;
}
Notice the important bit here. I've added the animationFrameTime to the timeUntilNextFrame. This way, if there's any left over time from the previous frame, it'll get added to the next frame, keeping the animation smooth.
There's really not much more to it than that. I've left out the rectangles.Count and isLooping bits, but it should be easy enough to add those back in. You might also want to initialize timeUntilNextFrame to animationFrameTime for the first frame.
Just a word of warning with this code, if the game is running slowly for some reason (or perhaps it's paused maybe) the gameFrameTime might end up having a very large value causing some strange behavior. I'm not entirely certain if that can ever happen in MonoGame / XNA but it's worth considering at least.

It turned out this is a well known problem. Basicaly its mathematicaly imposible to jam 24 animation fps in 30 game fps because they are not independent from each other. Interpolation is the only way to fix this problem. It's the same thing that's being done whenever you watch 24 fps movie on a 60 hz monitor.
https://en.wikipedia.org/wiki/Three-two_pull_down

Does anyone know a way to capture webcam video at a constant framerate?

OK, so I have been having a bit of a tough time with webcam capture, and am in need of help to find a way to capture the video at a consistent frame rate.
I have been using Aforge.AVIWriter and Aforge.VideoFileWriter, but to no avail, and have also typed any related phrase I can think of into Google.
I have had a look at the DirectShowLib, but am yet to find it any more accurate.
The video must have a minimum frame rate of 25fps, it is also to be shown in sync with other data which is collected at the same time.
I have also tried an infinite loop:
for (; ; )
{
if (recvid == false)
{
break;
}
if (writer.IsOpen)
{
Bitmap image = (Bitmap)videoSourcePlayer1.GetCurrentVideoFrame();
if (image != null)
{
writer.WriteVideoFrame(image);
}
Thread.Sleep(40);
}
}
Even though this is more accurate for timing, the user can see that the fps changes when they watch the video and view data at the same time.
Any pointers or tips would be greatly appreciated, as I cannot think of a way to go from here.

two main issues that i can see:
writer.write() is it happening in a seperate thread? if not it will take time and hence the timing might not be accurate.
second thread.sleep() says that sleep for at-least 40 ms not exactly 40 ms.. to get better results reduce the wait time to 5 ms and do it in a loop.. use the systems time to actually figure out how long you have slept for and then take a frame capture.
Hope this helps

With most web cameras (except maybe rare exceptions and higher end cameras that offer you fine control over capture process) you don't have enough control over camera frame rate. The camera will be capturing stream of frames at its maximal frame rate for the given mode of operation, esp. capped by resolution and data bandwidth, with possibly lower rate in low level conditions.
No Thread.Sleep is going to help you there because it is way too slow and unresponsive - in order to capture 25 fps the hardware needs to run smoothly without any interruptions and explicit instructions to "capture next frame now" pushing new data on the one end of the queue with you popping captured frames on the other end. You typically have a lag of a few video frames even with decent hardware.

Stuttering when moving a single sprite in an XNA application

Update: I have uploaded a video showing the stutter here: http://intninety.co.uk/xnastutter.mp4 you may have to look closely in the video if you are not viewing it at 1920x1080, but you'll see that there is a rather distinct stutter when moving every 2 seconds or so, I'd recommend viewing it in Windows Media Player rather than your web browser to ensure the video itself isn't choppy and thus preventing you seeing the actual stutter
I'm recently picking up a project I started a while ago, however I am still struggling to solve the problem I left it at!
At the moment I have a very simple application which just has a single sprite on screen and is moved around using the directional keys. The problem is every two seconds or so, the game stutters and the sprite appears to jump backwards and then back forwards very quickly.
The sprite itself is a 55x33 bitmap, so isn't anything large, and the code in use is as follows. Hopefully this is enough to get the ball rolling on some ideas as to what may be the problem, if a video is required to see exactly how the stuttering looks I can put one together and upload it somewhere if need be.
As you'll see in the code it does compensate for time lost between frames by making the movement greater should it happen, however that drop is happening very consistently time wise, which is leading me to believe I'm doing something wrong somewhere.
I've tried on a few different machines but the problem persists across all of them, if anyone has any ideas or can see where it is I'm messing up it'd be greatly appreciated if you could point it out.
Thanks :)
Constructor of the Game Setting up the Graphics Device Manager
graphics = new GraphicsDeviceManager(this);
graphics.IsFullScreen = true;
graphics.SynchronizeWithVerticalRetrace = false;
graphics.PreferredBackBufferWidth = 1920;
graphics.PreferredBackBufferHeight = 1080;
Content.RootDirectory = "Content";
this.IsFixedTimeStep = false;
Code from the Game's Update Method
KeyboardState keyboard = Keyboard.GetState();
GamePadState gamePad = GamePad.GetState(PlayerIndex.One);
if (keyboard.IsKeyDown(Keys.Escape)) {
this.Exit();
}
if ((keyboard.IsKeyDown(Keys.Left)) || (gamePad.DPad.Left == ButtonState.Pressed))
{
this.player.MoveLeft((float)gameTime.ElapsedGameTime.TotalMilliseconds);
} else if ((keyboard.IsKeyDown(Keys.Right)) || (gamePad.DPad.Right == ButtonState.Pressed))
{
this.player.MoveRight((float)gameTime.ElapsedGameTime.TotalMilliseconds);
}
if ((keyboard.IsKeyDown(Keys.Up)) || (gamePad.DPad.Up == ButtonState.Pressed))
{
this.player.MoveUp((float)gameTime.ElapsedGameTime.TotalMilliseconds);
} else if ((keyboard.IsKeyDown(Keys.Down)) || (gamePad.DPad.Down == ButtonState.Pressed))
{
this.player.MoveDown((float)gameTime.ElapsedGameTime.TotalMilliseconds);
}
base.Update(gameTime);
The "Move" Methods seen in the above Update Method
public void MoveLeft(float moveBy)
{
this.position.X -= (moveBy * this.velocity.X);
}
public void MoveRight(float moveBy)
{
this.position.X += (moveBy * this.velocity.X);
}
public void MoveUp(float moveBy)
{
this.position.Y -= (moveBy * this.velocity.Y);
}
public void MoveDown(float moveBy)
{
this.position.Y += (moveBy * this.velocity.Y);
}
The Game's Draw Method
GraphicsDevice.Clear(Color.CornflowerBlue);
spriteBatch.Begin();
spriteBatch.Draw(this.player.Texture, this.player.Position, null, Color.White, this.player.Rotation, this.player.Origin, 1.0f, SpriteEffects.None, 0.0f);
spriteBatch.End();
base.Draw(gameTime);
Edit: forgot to mention, the velocity object used in the Move methods is a Vector2

I've managed to see it occur once for a split second which has led me to what I think is the problem. Since you are using the raw ElapsedGameTime.TotalMilliseconds value as a factor for your movement, all computer lag that your program experiences will be directly applied to the motion. For example, if your computer (OS) does something else for one twentieth of a second, then the elapsed time value will accumulate to a value of ~50 milliseconds, when it is normally about 0.3 milliseconds. This would cause a frame that has 150 times more motion than a normal frame.
To cause this to happen manually, you can do the following:
// define a frame counter
private int mCounter;
...
protected override void Update(GameTime pGameTime)
{
// save the elapsed time value
float time = (float)pGameTime.ElapsedGameTime.TotalMilliseconds;
...
// force a long frame every 2500th frame (change depending on your framerate)
if (mCounter++ > 2500)
{
mCounter = 0;
time = 75; // about 225 times longer frame
}
...
// use the time value in your move calls
if ((keyboard.IsKeyDown(Keys.Left)) || (gamePad.DPad.Left == ButtonState.Pressed))
mPlayer.MoveLeft(time);
To prevent this from happening, (aside from setting IsFixedTimeStep = true;, which would fix it immediately; but assuming you want IsFixedTimeStep to be false), you should use a time value, as above, but cap it. It's up to you to determine the proportion of elapsed time to motion and to determine what is a good amount of time to allow to pass per frame. Ex:
protected override void Update(GameTime pGameTime)
{
// save the elapsed time value
float time = (float)pGameTime.ElapsedGameTime.TotalMilliseconds;
if (time > 1)
time = 1;
...
if ((keyboard.IsKeyDown(Keys.Left)) || (gamePad.DPad.Left == ButtonState.Pressed))
mPlayer.MoveLeft(time);
...
While that will correct the issue for your current program, your frames are only at 0.3ms each since there is not a lot happening. Once there is more to your game, more time will be elapsing per frame, and you will want the cap to be much higher than 1ms.
EDIT: To be clear, this is 'downtime' from your CPU/OS. It's not going to go away*, it's just up to you whether to jump ahead a bunch when it happens (which can cause problems if the elapsed time ever makes it to 2000ms, for example), or to cap these spikes and let them cause lag; either way there is going to be a 'hole' in your motion that you can't fill. This really is normal, and it's not as critical as it seems. Once there is more happening in your game, it will become less and less noticeable. It stands out strongly at the moment particularly because there are only two graphics present and nothing else happening.
*(Actually, you might look for other applications and processes that you can close to keep the CPU from being borrowed by some other program, but since you are using a multi-tasking OS, you are never going to be guaranteed to have the CPU to yourself.)

Update Again
Just had a thought. Have you ever checked that (float)gameTime.ElapsedGameTime.TotalMilliseconds does not result in Infinity or a negative number? The elapsed time becoming negative would explain your sprite jumping back then forward.
Its the keyboard. You can prove this by changing moving left and right to left mouse and right mouse click. Use the keydown and keyup triggers to set a state instead of iskeydown.
Update:
It appears there are a few things that cause "stuttering" in XNA. I thought for sure your issue was the keyboard problem one. However, I can only assume you've tested your code with the GamePad also and it suffers the same.
There is another problem with IsFixedTimeStep but you already have that set to false. A quick google search suggests there are more than a few people with these types of problems without any clear fix.
There is a long discussion at http://forums.create.msdn.com/forums/t/9934.aspx?PageIndex=6
Other things to look at:
Try restricting your frame rate to 60 fps. There isn't any reason to go above that anyways. One power suggests that a 2d app that does nothing could stall the graphics pipeline with a significantly high throughput.
Check to see if IsSlowRunning is ever called. Garbage collection and Chaos Theory could cause this to be set. I

Thread.Sleep(1); is too slow for a Silverlight animation, is there an alternative?

I have this algorithm that are drawing a lot of pixels on a Canvas in an animation. But I have no control of the speed of the animation and it is drawing very fast, so I added a Thread.Sleep(1) but then it is too slow when drawing several thousand pixels.
I have tried a Storyboard approach but that ended up to be a very slow.
So are there an alternative to the Thread.sleep for slowing down my loop?
void DrawGasket(object sender, DoWorkEventArgs e)
{
Random rnd = new Random();
Color color = Colors.White;
while (Counter <= noOfPx)
{
switch (rnd.Next(3))
{
case 0:
m_lastPoint.X = (m_top.X + m_lastPoint.X)/2;
m_lastPoint.Y = (m_top.Y + m_lastPoint.Y)/2;
color = Colors.White;
break;
case 1:
m_lastPoint.X = (m_left.X + m_lastPoint.X)/2;
m_lastPoint.Y = (m_left.Y + m_lastPoint.Y)/2;
color = Colors.Orange;
break;
case 2:
m_lastPoint.X = (m_right.X + m_lastPoint.X)/2;
m_lastPoint.Y = (m_right.Y + m_lastPoint.Y)/2;
color = Colors.Purple;
break;
}
Dispatcher.BeginInvoke(() =>
{
var px = new Rectangle {
Height = m_pxSize,
Width = m_pxSize,
Fill = new SolidColorBrush(color)
};
Canvas.SetTop(px, m_lastPoint.Y);
Canvas.SetLeft(px, m_lastPoint.X);
can.Children.Add(px);
lblCounter.Text = Counter.ToString();
});
Counter++;
Thread.Sleep(1);
}
}

Can't you just sleep every N iterations through the loop rather than every iteration?

Here's the simple method: You want it to draw at a specific time...during each loop have it look to see what time it is. If it's "time" (whatever) rate you determine that to be, then draw/update. Otherwise, just loop. (For example, if you want 10 updates per second, and you just updated, then store the current time in a variable...and then compare against that variable. Until it is a 10th of a second later, don't redraw, just loop.)
This isn't bad, so long as you can do anything else you need to do within this loop. If this loop must complete, however, before anything else happens, you'll just be burning a lot of cpu cycles just waiting.
If you want to get fancy, and do this right, you can have it see how much time has passed, and then update the visual by the correct amount. So, for example, if you want your object to move 30 pixels per second, you could just update where it is precisely, (if .0047 seconds have gone by, for instance, it should move .141 pixels). You store that value. Of course, visually, it won't have moved at all until another round of the loop, but exactly when it moves and how far will be determined by time, rather than the speed of the computer. This way you'll get it moving 30 pixels per second, regardless of the speed of the machine...it will just be jumpier on a slow machine.

The CompositionTarget.Rendering event fires every time Silverlight draws a frame, this is a good place to write any custom drawing code. You can use some other mechanism, like a timer, to keep track of your "game time" and adjust the model appropriately on a separate thread (or at least a separate execution path) from the drawing code. That way when it comes time to draw the UI there are no calculations to perform, you just paint the current state of your model. Basically just move the code in Dispatcher.BeginInvoke to the event handler for CompositionTarget.Rendering and I believe you'll be good.

One option would be to not sleep each time:
if(Counter % 2 == 0) Thread.Sleep(1);

With or without Thread.Sleep(), the render speed is determined by how fast your system is, or by how much time your process gets. -> bad!
You should use the approach that video games are written nowadays. Keep track of the current time, and call BeginInvoke() only x times per second (where x is the frames per second render speed)
In the BeginInvoked function check how much is drawn and how much you still need to draw. Especially don't BeginInvoke for every single Rectangle. Even without a Thread.Sleep() it's always a context switch.