There are about 60 sound samples in my application. I have to play it with strong accuracy in time. 5ms will mater. Sound must play in response to user actions. So I do not know what sound'll play next. I thinked aboud precreating of SoundEffectInstance for all sounds. Does creating of SoundEffectInstance take any time and memory? Is it full copy of SoundEffect in memory or just some pointer information. Can I improve my application if I'll precreate SoundEffectInstances? Is there any other way to play sample as soon as possible in XNA?
A SoundEffect contains the actual waveform data. It uses up a large amount of memory (as much as the waveform data requires).
A SoundEffectInstance is a small object, containing data about playback (position, volume, etc) and a handle to the "voice" that has been allocated to do the playback.
When you use SoundEffect.Play, internally it creates a SoundEffectInstance. It can also pool these instances - so it does not need to recreate them on subsequent calls.
Creating a SoundEffectInstance takes some time and memory. For sound effects in a game running at 30FPS (33ms per frame) the latency involved should be imperceptible.
If you're trying to play back sequenced music by mixing together different sound effects, where even 5ms matters, you need to implement your own audio mixer using DynamicSoundEffectInstance. This will ensure that sounds are synchronised down to the sample.
If you're trying to create some kind of real-time playable "instrument", and you're trying to get your input-to-output latency down to 5ms - then give up now. 5ms is absurdly good and normally requires specialist equipment and software.
Still, if you need to reduce your input-to-output latency as best you can, you can poll input more frequently than once per frame. And you should still probably use DSEI. I am not sure if DSEI has a configurable playback buffer size, but if it does you should make it as small as possible without introducing audio glitching.
When a user chooses a set, you should load those sound effects into a dictionary, so accessing them will be faster later on.
However, you shouldn't load all the sound effects, especially those that you don't need in a certain screen or set. Additionnally, you should unload the sound effects you won't need anymore.
It does take time to load those sound effects, depending on the size and quantity you want to load. I'd recommend you make a basic loading screen for loading/unloading content in your game (most games do it that way). You will be loading the soundeffect (wav format I guess?) in memory, not just a pointer, so that's why sound effects need to be kept short.
An already loaded sound effect will play very fast (the delay is imperceptible). Preloading will increase the application's performance, at the cost of memory usage.
Related
Specifically I'm looking to perform a two channel convolution operation on an audio file at playback. i.e. to add a reverb effect to the file using an impulse response, before it is sent to the sound card for playing.
There is a distinct lack of examples or references to performing this operation in real-time in a C# application.
The NAudio (and maybe CScore) libraries look most promising but the absence of built in convoliution engine seems odd, is this likely because there is not enough call for it or is it more likely that a managed application is not suited to such operations?
Therefore it leads me to ask the posted question Can the real-time multi-channel audio convolution be performed in a C# application?
Yes, no problem you can do this. C# applications however aren't great for audio playback because of possible garbage collection delays. You'll need a fairly large buffer on the sound card and that introduces lag into your signal. From a file that's no issue, from a microphone in real-time it's not so great. I have a system that plays back three music streams simultaneously on three separate sound cards and muxes in additional speech files with ducking all written in C#. I do run the playback threads at higher priority also.
As to why there's no existing library to do this (there probably is), it's a fairly trivial piece of code: just multiplying and adding on a 1D stream of values.
I am having an issue with my vision processing code. I wish there was an actual error that would point me in the right direction and that I could share with all of you. However, there is no error per se. The thing I am getting is unintentional side effects in my video.
I have an IP network camera that is streaming H.264 video on the network. It is set to do this at 20 fps, though I have tried turning it down to 15 fps. Not sure if I want or need to go lower than that.
I am reading the frames in my code just fine. The way I have it integrated now is I get an event that raises every time there is a new frame from the camera, in my case 15 to 20 times per second.
I am trying to process that video frame as quickly as possible but I am doing some OpenCV (or better EmguCV) processing on it along with some object tracking.
Because I am object tracking, i can just ignore or dump frames in between because having them in order/sequential is important.
My processing time fluctuates but can be slower at times but only by a few frames, like 3 to 4 frames. However, when I am slower, I naturally miss frames and this causes the object tracker to drop tracking and the video to appear like it is skipping, both of which I dont want.
Aside from seeing if I can speed up my processing, which I may not be able to do, what is the best way to handle this so I dont skip or drop frames that fire and come in even though I am still processing or processing a little slower?
I am looking in to a concurrent queue to act as a buffer but i am not sure if going down this road is the right approach either.
Thanks!!
I can easily think of a number of situations where it would be useful to change a single pixel in a Texture2D, especially because of the performance hit and inconvenience you get when constantly doing GetData<>(); SetData<>(); every frame or drawing to a RenderTarget2D.
Is there any real reason not to expose setter methods for single pixels? If not, is there a way to modify a single pixel without using the methods above?
Texture data is almost always copied to video memory (VRAM) by the graphics driver when the texture is initialized, for performance reasons. This makes texture fetches by the shaders running on the GPU significantly faster; you would definitely not be happy if every texture cache miss had to fetch the missing data over the PCIe bus!
However, as you've noticed, this makes it difficult and/or slow for the CPU to read or modify the data. Not only is the PCIe bus relatively slow, but VRAM is generally not directly addressable by the CPU; data must usually be transferred using special low-level DMA commands. This is exactly why you see a performance hit when using XNA's GetData<>() and SetData<>(): it's not the function call overhead that's killing you, it's the fact that they have to copy data back and forth to VRAM behind your back.
If you want to modify data in VRAM, the low-level rendering API (e.g. OpenGL or Direct3D 11) gives you three options:
Temporarily "map" the pixel data before your changes (which involves copying it back to main memory) and "unmap" it when your edits are complete (to commit the changes back to VRAM). This is probably what GetData<>() and SetData<>() are doing internally.
Use a function like OpenGL's glTexSubImage2D(), which essentially skips the "map" step and copies the new pixel data directly back to VRAM, overwriting the previous contents.
Instruct the GPU to make the modifications on your behalf, by running a shader that writes to the texture as a render target.
XNA is built on top of Direct3D, so it has to work within these limitations as well. So, no raw pixel data for you!
(As an aside, everything above is also true for GPU buffer data.)
I need capture video and save it to hard drive every xxx minutes. It will be win-service, which is always capturing and saving. There would be several cameras, so I think use processing thread per camera. So if I decide use Thread instead of timer, I should use Thread.Sleep before saving operation. To my mind it's not good practice.
So the question is my design(thread per camera) appropriate for my situation? What about timers?
Thanks, Andrew
Your code may get occasional lockups due to global mutex'es (for example, in driver).
Also, for example, you have open input in one thread, you're flushing it to some stream, and you want to read it from other thread: mutex lock happens. Same with timers. Considering the event-driven machine too hard to implement here, you still can fallback to multiple instances..
You could have a look at the code for the open-source project Media Portal at http://www.team-mediaportal.com/
Look at their sub-project called TvEngine3.
You could force it to graph your cameras as a "Tuner" device, and set it to record via api.
Point is, they've worked out hundreds of DirectShow issues, and there are many.
I would definitively go for one thread per camera as chances are you'd use less than 20-30 cameras. But it is probably not required as reading a camera (usually) is a non-blocking operation.
There are several questions on webcam and video in C# on SO. Try looking at How do I capture video from a webcam?
Note that you may want to pre-allocate large chunks of data to avoid disk fragmentation. Storing multiple streaming videos to disk could really fragment stuff, and then watching/copying/modifying the videos would be really slow.
I have a complex problem, I've been working on it for weeks. My program is an eduactional software which use the webcam for analyzing physical experiments (eg. oscillating movement). I've experienced the folowings:
If the processor is busy, the time
measuring is inaccurate
(ISampleGrabberCB.BufferCB(SampleTime))
If I don't use the time, just count
the samples: 0, 1, 2... it looks
better. I perceive this when I look
at the curve of the movement.
My primary goal is reduce the inaccuracy, what I try to achieve with limitation of the FPS (which cause busy processor).
My WebCam (Intel Classmate PC's built
in webcam) has auto fps and exposure
time. Depending on the illumination
they fluctuate.
IAMStreamConfig.AvgTimePerFrame has no effect.
IAMCameraControl isn't supported by the webcam.
IKsPropertySet: I don't know how to
use this, since I don't have any
support for the webcam. In this
example they can use it for Logitech
webcam: http://social.msdn.microsoft.com/Forums/en/windowsdirectshowdevelopment/thread/47b1317d-87e6-4121-9189-0defe1e2dd44
from the MSDN article on Time and Clocks in DirectShow:
Any object that supports the IReferenceClock interface can serve as a reference clock. A filter with access to a hardware timer can provide a clock (an example is the audio renderer), or the filter graph manager can create one that uses the system time.
I've never attempted to use the IReferenceClock from a filter, but it would be my suspicion that it may not provide a high resolution clock that you need.
This SO post on high resolution timers might be what you need.
IAMStreamConfig.AvgTimePerFrame is for informational purposes, and attempting to adjust it won't have any effect. It's just a value from which you can calculate average frame rate for your video stream.
e.g.
VIDEOINFOHEADER* pVih = (VIDEOINFOHEADER*)m_MediaTypes.VideoType.pbFormat;
if( pVih )
int nFrameRate = (int)( (double) (10000000.0f / pVih->AvgTimePerFrame) );