I have a code that receives a video stream through the webrtc library, which in its function shows them in a PictureBox, my question is .. how to pass that stream from the PictureBoxto a video on my computer?
public unsafe void OnRenderRemote(byte* yuv, uint w, uint h)
{
lock (pictureBoxRemote)
{
if (0 == encoderRemote.EncodeI420toBGR24(yuv, w, h, ref bgrBuffremote, true))
{
if (remoteImg == null)
{
var bufHandle = GCHandle.Alloc(bgrBuffremote, GCHandleType.Pinned);
remoteImg = new Bitmap((int)w, (int)h, (int)w * 3, PixelFormat.Format24bppRgb, bufHandle.AddrOfPinnedObject());
}
}
}
try
{
Invoke(renderRemote, this);
}
catch // don't throw on form exit
{
}
}
This code receives the stream through webrtc and converts it into images that are then shown in a PictureBoxcalling this function .. my question is:
How can I save an array or buffer of remoteImg images so I can write it to a video file on my pc?
Try doing something like this:
FileWriter.Open ("C:\\Users\\assa\\record.avi", (int) w, (int) h, (int) w * 3, VideoCodec.Default, 5000000);
FileWriter.WriteVideoFrame (remoteImg);
but only saves a single capture and not a video, is there any way to save the images of the stream with the OnRenderRemote function (described above) to be able to save them in a video?
OnRenderRemote only updates the PictureBox every time it is called, but I do not know how to save that flow in a video.
Thanks.
First: i do not know how the webrtc works exactly, but i can explain you how you must process the images to save them into a file.
Ok lets start: You currently have only full sized bitmaps of your own that are coming from the lib. That is just fine as long as you do not care about file size and you only want to show the "latest" frame. To store multiple frames into a file that we would call a "video" you need an encoder that processes those frames together.
Complicated things simple: An encoder takes 2 frames, call them Frame A and B and then compresses them in a way that only changes from Frame A to frame B are saved. This saves a lot of storage because in a video we only want to see "changes" aka movments from one frame to another. There are quite a lot of encoders out there but mostly you can see ffmpeg out there, its very popular and there are quite a lot c# wrappers for it so take a look.
Summery: to make 2-x images a "video" you have the process them with an encoder that processes the images in a format that can be played by a video player.
Related
I want to add images on running video using c#.
My Code Is but not working
byte[] mainAudio = System.IO.File.ReadAllBytes(Server.MapPath(image path));//Upload by User
byte[] intreAudio = System.IO.File.ReadAllBytes(Server.MapPath(video path));//File Selected For Interruption
List<byte> int1 = new List<byte>(mainAudio);
int1.AddRange(intreAudio);
byte[] gg = int1.ToArray();
using (FileStream fs =
System.IO.File.Create(Server.MapPath(#"\TempBasicAudio\myfile1.mp3")))
{
if (gg != null)
{
fs.Write(gg, 0, gg.Length);
}
}
Did it ever occor to you that a video file is not justa mindless "array of images" so you can not just add another byte range at the end?
Depending on the video type there is a quite complex structure of management structured you just ignore. Videos are highly complex encoding.
YOu may have to add the images in a specific form WHILE UPDATING THE MANAGEMENT INFORMATION - or you may even have to transcode that (decode all images, then reencode the whole video stream).
Maybe a book about the basics of video processing is in order now? You are like the guy asking why you can not get more horsepower out of your car by running it on rocket fuel - totally ignoring the realities of how cars operate.
common_video.avi
image1.jpg
image2.jpg
i want to insert these two images to the end of the video name common_video.avi programatically in c# so that the image shows for like 5 seconds after the video ends, what's the best way to achieve it? I have looked in to ffmpeg, with and without c# wrappers, but still nothing works. I keep getting errors and exceptions.
here's a piece of code i have tried
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
//using AForge;
//using AForge.Video.VFW;
using AForge.Video.FFMPEG;
using AviFile;
using System.Drawing;
namespace Ffmpeg
{
class Program
{
static void Main(string[] args)
{
var Strings = new string[] { "1.jpg", "2.jpg", "3.jpg" };
//VideoFileWriter W = new VideoFileWriter();
//W.Open("../../Out/new.avi", 1920, 1200, 1, VideoCodec.Raw, 4400);
//foreach (var S in Strings)
//{
// for (int I = 2 * 25; I > 0; I--)
// W.WriteVideoFrame((Bitmap)Image.FromFile(S));
//}
//W.Close();
//load the first image
Bitmap bitmap = (Bitmap)Image.FromFile(Strings[0]);
//create a new AVI file
AviManager aviManager =
new AviManager(#"..\..\out\new.avi", false);
//add a new video stream and one frame to the new file
VideoStream aviStream =
aviManager.AddVideoStream(false, 1, bitmap);
//Bitmap bitmap;
for (int n = 1; n < Strings.Length; n++)
{
if (Strings[n].Trim().Length > 0)
{
bitmap =
(Bitmap)Bitmap.FromFile(Strings[n]);
//for (int I = 2 * 25; I > 0; I--)
aviStream.AddFrame(bitmap);
bitmap.Dispose();
}
}
aviManager.Close();
}
}
}
Ffmpeg throws: "error configuring filters";
Not sure how the FFmpeg C API translates to C#, but adding a video stream in FmMpeg does not let you add more frames at the end of existing video stream. It creates a new video stream, therefore generating a video file with several video streams. Not only this is not what you want, but also not all muxers support muxing two video streams which may be the reason you're getting an error.
What you need could be achieved one of the following way:
Decode the original video, and reencode it. During reencoding you can easily modify the video in any way, and add images anywhere you want. This is the easiest to implement, but you'll lose the video quality due to dual reencoding if you encode into a lossy format.
Find out the exact parameters which were used to encode the original video (resolution, aspect rate, colorspace, frame rate, codec, codec parameters etc), then encode your images using exactly the same codec with the same options. Then you can copy the video stream by reading/writing frames (no reencoding), and append your newly encoded frames to this video. This will avoid reencoding of the original video, but is much trickier to implement and prone to errors, especially if the original video was encoded not by FFmpeg but a different encoder.
There are two ways:
If you don't care about performance, you can read the frames one by one, then decode each frame to raw format(yuv420), and encode the frame again to any codec you want. For your picture, just convert to yuv420, and encode into the file.
If you just want to encode your picture into the avi file: You need to know the video stream is what codec in your avi file, then encode your picture by this codec, and the first picture must be key frame. There is another problem, your codec parameter maybe won't be compatible to the origin video stream, there will be difficult. You have to parse the video stream, and find the parameter details.
I am going through Mp3StreamingDemo from NAudio Source Demo, and I need an explanation (nothing in depth, just a few sentences, to get a general idea) about decompressing the Mp3 frame.
The actual code is:
IMp3FrameDecompressor decompressor = null;
//...
if (decompressor == null)
{
WaveFormat waveFormat = new Mp3WaveFormat(frame.SampleRate, frame.ChannelMode == ChannelMode.Mono ? 1 : 2, frame.FrameLength, frame.BitRate);
//What does AcmMp3FrameDecompressor do?
decompressor = new AcmMp3FrameDecompressor(waveFormat);
this.bufferedWaveProvider = new BufferedWaveProvider(decompressor.OutputFormat);
}
int decompressed = decompressor.DecompressFrame(frame, buffer, 0);
I do have some knowledge about MP3, how does it look, about frames, etc. I just don't understand the process of mp3 frame decompression? Specifically:
what for is AcmMp3FrameDecompressor class used? What does DecompressFrame method do?
I can see the code from the class, but to understand it in depth I think I'll need much more knowledge about audio itself. And at the moment, as I said, I would appreciate just a description in general.
Thank you for your time and help.
AcmMp3FrameDecompressor decompresses an MP3 frame to PCM using the ACM codec on your computer. All desktop versions of Windows since Windows XP come with one, but there are some cases you might encounter where one is unavailable. NAudio also supplies a DMO based MP3 frame decoder, which can be used on Windows Vista and newer.
EDIT: I keep getting OutOfMemoryException was unhandled,
I think it's how I am saving the image to isolated storage ,I think this is where I can solve my problem how do I reduce the size of the image before I save it? (added code where I save Image)
I am opening images from Isolated storage sometimes over 100 images and I want to loop over them images but I get a OutOfMemory Exception when there is around 100 to 150 images loaded in to a storyboard. How can I handle this exception, I have already brought down the resolution of the images. How can I handle this exception and stop my app from crashing?
I get the exception at this line here
image.SetSource(isStoreTwo.OpenFile(projectFolder + "\\MyImage" + i + ".jpg", FileMode.Open, FileAccess.Read));//images from isolated storage
here's my code
private void OnLoaded(object sender, RoutedEventArgs e)
{
IsolatedStorageFile isStoreTwo = IsolatedStorageFile.GetUserStoreForApplication();
try
{
storyboard = new Storyboard
{
//RepeatBehavior = RepeatBehavior.Forever
};
var animation = new ObjectAnimationUsingKeyFrames();
Storyboard.SetTarget(animation, projectImage);
Storyboard.SetTargetProperty(animation, new PropertyPath("Source"));
storyboard.Children.Add(animation);
for (int i = 1; i <= savedCounter; i++)
{
BitmapImage image = new BitmapImage();
image.SetSource(isStoreTwo.OpenFile(projectFolder + "\\MyImage" + i + ".jpg", FileMode.Open, FileAccess.Read));//images from isolated storage
var keyframe = new DiscreteObjectKeyFrame
{
KeyTime = KeyTime.FromTimeSpan(TimeSpan.FromMilliseconds(100 * i)),
Value = image
};
animation.KeyFrames.Add(keyframe);
}
}
catch (OutOfMemoryException exc)
{
//throw;
}
Resources.Add("ProjectStoryBoard", storyboard);
storyboard.Begin();
}
EDIT This is how I am saving the image to Isolated storage, I think this is where I can solve my problem, How do I reduce the size of the image when saving it to isolated storage?
void cam_CaptureImageAvailable(object sender, Microsoft.Devices.ContentReadyEventArgs e)
{
string fileName = folderName+"\\MyImage" + savedCounter + ".jpg";
try
{
// Save picture to the library camera roll.
//library.SavePictureToCameraRoll(fileName, e.ImageStream);
// Set the position of the stream back to start
e.ImageStream.Seek(0, SeekOrigin.Begin);
// Save picture as JPEG to isolated storage.
using (IsolatedStorageFile isStore = IsolatedStorageFile.GetUserStoreForApplication())
{
using (IsolatedStorageFileStream targetStream = isStore.OpenFile(fileName, FileMode.Create, FileAccess.Write))
{
// Initialize the buffer for 4KB disk pages.
byte[] readBuffer = new byte[4096];
int bytesRead = -1;
// Copy the image to isolated storage.
while ((bytesRead = e.ImageStream.Read(readBuffer, 0, readBuffer.Length)) > 0)
{
targetStream.Write(readBuffer, 0, bytesRead);
}
}
}
}
finally
{
// Close image stream
e.ImageStream.Close();
}
}
I would appreciate if you could help me thanks.
It doesn't matter how large your images are on disk because when you load them into memory they're going to be uncompressed. The memory required for the image will be approximately (stride * height). stride is width * bitsPerPixel)/8, and then rounded up to the next multiple of 4 bytes. So an image that's 1024x768 and 24 bits per pixel will take up about 2.25 MB.
You should figure out how large your images are, uncompressed, and use that number to determine the memory requirements.
You are getting the OutOfMemory Exception because you are storing all the images in memory at the same time in order to create your StoryBoard. I don't think you will be able to overcome the uncompressed bitmap size that the images require to be displayed on screen.
So to get past this we must think about your goal rather than trying to fix the error. If your goal is to show a new image in sequence every X ms then you have a few options.
Keep using StoryBoards but chain them using the OnCompleted event. This way you don't have to create them all at once but can just generate the next few. It might not be fast enough though if you're changing images every 100ms.
Use CompositionTarget.Rendering as mentioned in my answer here. This would probably take the least amount of memory if you just preload the next one (as opposed to having them all preloaded as your current solution does). You'd need to manually check the elapsed time though.
Rethink what you're doing. If you state what you are going after people might have more alternatives.
To answer the edit at the top of your post, try ImageResizer. There's a NuGet package, and a HanselBlog episode on it. Obviously , this is Asp.Net based, but I'm sure you could butcher it to work in your scenario.
Tackling these kind of problems at design layer usually works better.
Making application smart about the running environment via some configurations makes your application more robust. For example you can define some variables like image size, image count, image quality... based on available memory and set these variables at run-time in your App. So your application always works; fast on high memory machines and slow on low memory ones; but never crash. (Don't believe working in managed environment means no worry about the environment... Design always matters)
Also there are some known design patterns like Lazy Loading you can benefit from.
I don't know about windows phone in particular, but in .net winforms, you need to use a separate thread when doing a long-running task. Are you using a BackgroundWorker or equivalent? The finalizer thread can become blocked, which will prevent the resources for the images from being disposed. Using a separate thread from the UI thread will allow will allow the Dispose method to be run automatically.
Ok, an image (1024x768) has at least a memsize of 3 mb (argb)
Don't know how ObjectAnimationUsingKeyFrames works internal. Maybe you can force the gc by destroying the instances of BitmapImage (and KeyFrames) without loss of its data in the animation.
(not possible, see comments!)
Based on one of your comments, you are building a Time Lapse app. Commercial time-lapse apps for WP7 compress the images to video, not stills. e.g. Time Lapse Pro
The whole point of video playback is to reduce similar, or time-related, images to highly compressed stream that do not require massive amounts of memory to play back.
If you can add the ability to encode to video, in your app, you will avoid the problem of trying to emulate a video player (using 100s of single full-resolution frames as a flick-book).
Processing the images into video server-side may be another option (but not as friendly as in-camera).
after a whole day of testing I came up with this code, which captures current screen using DirectX (SlimDX) and saves it into a file:
Device d;
public DxScreenCapture()
{
PresentParameters present_params = new PresentParameters();
present_params.Windowed = true;
present_params.SwapEffect = SwapEffect.Discard;
d = new Device(new Direct3D(), 0, DeviceType.Hardware, IntPtr.Zero, CreateFlags.SoftwareVertexProcessing, present_params);
}
public Surface CaptureScreen()
{
Surface s = Surface.CreateOffscreenPlain(d, Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height, Format.A8R8G8B8, Pool.Scratch);
d.GetFrontBufferData(0, s);
return s;
}
Then I do the following:
DxScreenCapture sc = new DxScreenCapture();
..code here
private void button1_Click(object sender, EventArgs e)
{
Stopwatch stopwatch = new Stopwatch();
// Begin timing
stopwatch.Start();
Surface s = sc.CaptureScreen();
Surface.ToFile(s, #"c:\temp\test.png", ImageFileFormat.Png);
s.Dispose();
stopwatch.Stop();
textBox1.Text = ("Elapsed:" + stopwatch.Elapsed.TotalMilliseconds);
}
The results are:
0. when I don't save surface: avg. elapsed time: 80-90ms
1. when I also save Surface to BMP file: format: ImageFileFormat.Bmp , avg. elapsed time: 120ms, file size: 7mb
2. when I also save Surface to PNG file: format: ImageFileFormat.Png , avg. elapsed time: 800ms, file size: 300kb
The questions are:
1. Is it possible to optimise current image capture? According to this article - Directx screen capture should be faster than GDI. For me, GDI usually takes 20ms to get a "Bitmap", whereas it takes 80ms to get "Surfare" using DX (both without saving).
http://www.codeproject.com/Articles/274461/Very-fast-screen-capture-using-DirectX-in-Csharp
2a. How to save Surface to PNG image format faster? When I save surface to 7mb BMP file it takes almost 6 times less time, than when I save the same surface to 300kb PNG file..
2b. Is it possible to save Surface directly to Bitmap so I don't have to create temporary files?
So I don't have to do following: Surface -> image file; image file open -> bitmap;, but instead: Surface -> bitmap
that's all for now. I'll gladly accept any tips, thanks!
Edit:
Just solved 2b by doing:
Bitmap bitmap = new Bitmap(SlimDX.Direct3D9.Surface.ToStream(s, SlimDX.Direct3D9.ImageFileFormat.Bmp));
Edit2:
Surface.ToFile(s, #"C:\temp\test.bmp", ImageFileFormat.Bmp);
Bitmap bitmap = new Bitmap(#"C:\temp\test.bmp");
is faster than:
Bitmap bitmap = new Bitmap(SlimDX.Direct3D9.Surface.ToStream(s, SlimDX.Direct3D9.ImageFileFormat.Bmp));
by 100 ms!!! Yeah, I couldn't believe my eyes too ;) I don't like the idea of temporary file creation, but a 50% performance increase (100-200ms instead of 200-300+) is a very good thing.
If you don't want to use SlimDX library you can also try
public Bitmap GimmeBitmap(Surface s)
{
GraphicsStream gs = SurfaceLoader.SaveToStream(ImageFileFormat.Bmp, s);
return new Bitmap(gs);
}
and try the same for .png - I did not test performance but it have to be faster than using disc temporary file :)
and as for 1st question - try to only once create surface and then on every screenshot only put into it device's buffer data and create the bitmap
d.GetFrontBufferData(0, s);
return new Bitmap(SurfaceLoader.SaveToStream(ImageFileFormat.Bmp, s));
this should save you some time :)
If performance really is an issue, you should consider writing your code in C++ instead. Therefor you dont need an external library but can directly access the backend-buffer of your video card via Windows-API + DirectX.
Accessing the backend(-video)-buffer is a lot faster than reading from the frontend-buffer.
To optimize performance (which also awnsers your question 1) use multithreading (see TPL or threading depending on your needs).
Here is an inside of how to do it in C++ CodeProject examples in C++.
From my personal experience, DirectX was by far the fastest.
These steps
1. reading backend-buffer into a bitmap to process the data
2. spawning new thread to repeat step 1 while previous thread is still busy
take about 10-40ms (together) - implemented in C++ (on NVIDIA GeForce GTX 970M) and depending on the current workload of the hardware
Possible middle course
If you want to stick with C# but also need the performance, writing a C++-dll for .NET (see .NET Programming with C++/CLI (Visual C++)) which reads the video buffer and returns the data to your C#-Code will do the trick.