We Keep Coding

Converting Texture2D into a video

I've did a lot of research, but I can't find a suitable solution that works with Unity3d/c#. I'm using a Fove-HMD and would like to record/make a video of the integrated camera. So far I managed every update to take a snapshot of the camera, but I can't find a way to merge this snapshots into a video. Does someone know a way of converting them? Or can someone point me in the right direction, in which I could continue my research?
public class FoveCamera : SingletonBase<FoveCamera>{
private bool camAvailable;
private WebCamTexture foveCamera;
private List<Texture2D> snapshots;
void Start ()
{
//-------------just checking if webcam is available
WebCamDevice[] devices = WebCamTexture.devices;
if (devices.Length == 0)
{
Debug.LogError("FoveCamera could not be found.");
camAvailable = false;
return;
}
foreach (WebCamDevice device in devices)
{
if (device.name.Equals("FOVE Eyes"))
foveCamera = new WebCamTexture(device.name);//screen.width and screen.height
}
if (foveCamera == null)
{
Debug.LogError("FoveCamera could not be found.");
return;
}
//-------------camera found, start with the video
foveCamera.Play();
camAvailable = true;
}
void Update () {
if (!camAvailable)
{
return;
}
//loading snap from camera
Texture2D snap = new Texture2D(foveCamera.width,foveCamera.height);
snap.SetPixels(foveCamera.GetPixels());
snapshots.Add(snap);
}
}
The code works so far. The first part of the Start-Method is just for finding and enabling the camera. In the Update-Method I'm taking every update a snapshot of the video.
After I "stop" the Update-Method, I would like to convert the gathered Texture2D object into a video.
Thanks in advance

Create MediaEncoder
using UnityEditor; // VideoBitrateMode
using UnityEditor.Media; // MediaEncoder
var vidAttr = new VideoTrackAttributes
{
bitRateMode = VideoBitrateMode.Medium,
frameRate = new MediaRational(25),
width = 320,
height = 240,
includeAlpha = false
};
var audAttr = new AudioTrackAttributes
{
sampleRate = new MediaRational(48000),
channelCount = 2
};
var enc = new MediaEncoder("sample.mp4", vidAttr, audAttr);
Convert each snapshot to Texture2D
Call consequently AddFrame to add each snapshot to MediaEncoder
enc.AddFrame(tex);
Once done call Dispose to close the file
enc.Dispose();

I see two methods here, one is fast to implement, dirty and not for all platforms, second one harder but pretty. Both rely on FFMPEG.
1) Save every frame into image file (snap.EncodeToPNG()) and then call FFMPEG to create video from images (FFmpeg create video from images) - slow due to many disk operations.
2) Use FFMPEG via wrapper implemented in AForge and supply its VideoFileWriter class with images that you have.
Image sequence to video stream?
Problem here is it uses System.Bitmap, so in order to convert Texture2D to Bitmap you can use: How to create bitmap from byte array?
So you end up with something like:
Bitmap bmp;
Texture2D snap;
using (var ms = new MemoryStream(snap.EncodeToPNG()))
{
bmp = new Bitmap(ms);
}
vFWriter.WriteVideoFrame(bmp);
Both methods are not the fastest ones though, so if performance is an issue here you might want to operate on lower level data like DirectX or OpenGL textures.

Access violation/Program not responding when Matching template from Camera capture. (C# EmguCV)

Here is the code i'm currently using in C# and EmguCV included:
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;
using Emgu.CV;
using Emgu.CV.Structure;
using Emgu.Util;
namespace CameraCapture
{
public partial class CameraCapture : Form
{
//declaring global variables
private Capture capture; //takes images from camera as image frames
private bool captureInProgress; // checks if capture is executing
public CameraCapture()
{
InitializeComponent();
}
private void ProcessFrame(object sender, EventArgs arg)
{
Image<Bgr, Byte> ImageFrame = capture.QueryFrame();
Image<Bgr, Byte> template = new Image<Bgr, byte>(#"D:\yugiCards\kuriboh.jpg");
Image<Bgr, Byte> imageToShow = ImageFrame.Copy();
using (Image<Gray, float> result = imageToShow.MatchTemplate(template, Emgu.CV.CvEnum.TM_TYPE.CV_TM_CCOEFF_NORMED))
{
double[] minValues, maxValues;
Point[] minLocations, maxLocations;
result.MinMax(out minValues, out maxValues, out minLocations, out maxLocations);
// You can try different values of the threshold. I guess somewhere between 0.75 and 0.95 would be good.
if (maxValues[0] > 0.9)
{
// This is a match. Do something with it, for example draw a rectangle around it.
Rectangle match = new Rectangle(maxLocations[0], template.Size);
imageToShow.Draw(match, new Bgr(Color.Red), 3);
}
}
CamImageBox.Image = imageToShow;
//ImageFrame.Save(#"E:\MyPic.jpg"); //saves to location
}
private void CameraOutput_Load(object sender, EventArgs e)
{
}
private void btnStart_Click(object sender, EventArgs e)
{
#region if capture is not created, create it now
if (capture == null)
{
try
{
capture = new Capture();
}
catch (NullReferenceException excpt)
{
MessageBox.Show(excpt.Message);
}
}
#endregion
if (capture != null)
{
if (captureInProgress)
{ //if camera is getting frames then stop the capture and set button Text
// "Start" for resuming capture
btnStart.Text = "Start!"; //
Application.Idle -= ProcessFrame;
}
else
{
//if camera is NOT getting frames then start the capture and set button
// Text to "Stop" for pausing capture
btnStart.Text = "Stop";
Application.Idle += ProcessFrame;
}
captureInProgress = !captureInProgress;
}
}
private void ReleaseData()
{
if (capture != null)
capture.Dispose();
}
}
}
I am trying to find a template in the camera capture, however, when I run the program and start camera capture my camera LED lights up and then the program becomes not responding when it's trying to capture. Maybe it has to do with the placement of the match template function or the types of variables used but I'm not really sure since I'm not that experienced so I wanted some input on the problem with my code.
I do realize the imageToShow variable isn't needed but I decided to leave it be until I can get the thing working then I can mess around with it more myself.
Code for detecting the template was found here
emgu finding image a in image b
It was mainly for detecting template between 2 static images though, but I edited the source to be from the webcam.
The webcam is working normally when I remove the using match template segment from the code.
Any input would be appreciated, thanks and sorry if it is an obvious error I'm still new to this kind of thing.
EDIT: Forgot to mention it gives this error when debugging The program
'[6164] CameraCapture.vshost.exe' has exited with code -1073741819 (0xc0000005) 'Access violation'.

Solved, after hours of trying to mess around with the code it turned out the template being used just wasn't a good one.
After looking more into what the author of the code said, he mentioned you might want grey around your template, thought he meant the Bgr to Gray that's why I was frustrated with the code. Turned out it meant you literally needed grey around your template.

Using a background worker to efficiently write to a GUI thread

I am using a BackgroundWorker to pull video from a camera and write it to a PictureBox on my WinForms form. In the BW thread I simply pull a frame from the camera, put it into the PictureBox, sleep, and continue:
while (CaptureThreadRunning)
{
Thread.Sleep(5);
Image tmp = Camera.GetFrame(500);
pbCameraFeed.Image = tmp;
//this.BeginInvoke(new MethodInvoker(delegate { pbCameraFeed.Image = Camera.GetFrame(500); }));
}
The issue is that eventually adjusting or moving the form around my screen will throw the exception System.InvalidOperationException with the message Additional information: Object is currently in use elsewhere. on the line pbCameraFeed.Image = tmp;
I assume that the library is trying to paint something to do with the PictureBox at the same time as my while loop is, so I switched to the this.BeginInvoke implementation that is commented out above. Unfortunately that cuts my framerate significantly. I am running this code on a very slow Mini PC which may be contributing to the issue.
What I really want is a way to update my GUI with the image reliably that doesn't drop my framerate by nearly half. Are there other standard ways to do this? A BW thread seemed perfect for this application, but am I missing something?
Thanks

If I were you I would definitely check out the AForge.NET Framework. No need to reinvent the wheel ;)
http://www.aforgenet.com/framework/samples/video.html
AForge.NET is an open source C# framework designed for developers and
researchers in the fields of Computer Vision and Artificial
Intelligence - image processing, neural networks, genetic algorithms,
fuzzy logic, machine learning, robotics, etc.
The framework is comprised by the set of libraries and sample
applications, which demonstrate their features:
AForge.Imaging - library with image processing routines and filters;
AForge.Vision - computer vision library;
AForge.Video - set of libraries for video processing;
...

I would recommend not to use a PictureBox, and instead directly draw to a UserControl surface. This can be easily done by addig code to the Paint and Invalidate events of a UserControl.
This example below, creates a user control which has a BitMap property that it's drawed to its surface every time the control is invalidated.
So, for example, to randomly render JPG images from folder D:\MyPictures, you can do the following:
using System.Windows.Forms;
using System.Drawing;
void Main()
{
var pictures = Directory.GetFiles(#"D:\MyPictures", "*.jpg");
var rnd = new Random();
var form = new Form();
var control = new MyImageControl() { Dock = DockStyle.Fill };
form.Controls.Add(control);
var timer = new System.Windows.Forms.Timer();
timer.Interval = 50;
timer.Tick += (sender, args) =>
{
if (control.BitMap != null)
control.BitMap.Dispose();
control.BitMap = new Bitmap(pictures[rnd.Next(0, pictures.Length)]);
control.Invalidate();
};
timer.Enabled = true;
form.ShowDialog();
}
public class MyImageControl : UserControl // or PictureBox
{
public Bitmap BitMap { get; set; }
public MyImageControl()
{
this.Paint += Graph_Paint;
this.Resize += Graph_Resize;
}
private void Graph_Paint(object sender, PaintEventArgs e)
{
if (this.BitMap != null)
{
lock (this.BitMap)
{
Graphics g = e.Graphics;
g.DrawImage(this.BitMap, this.ClientRectangle);
}
}
}
private void Graph_Resize(object sender, System.EventArgs e)
{
this.Invalidate();
}
}
I think this can be easily changed to render the camera images instead of the picture files.
The code was tested on LinqPad

Emgucv return null Image for QueryFrame for IP Camera [duplicate]

I am working on one application where I want to use IP camera for displaying video streaming and and some other major operations on image captured by the IP Camera.
Libraries used in Camera capture
For Camera Capture : Emgu.CV Library
Below is the code which I am using in C#.
Variable Declaration
private Capture capture; //takes images from camera as image frames
private Emgu.CV.UI.ImageBox img; // Dynamic Picture Controls
private int nCam; // no of cameras
Code for Processing Image
private void ProcessFrame(object sender, EventArgs arg)
{
try
{
// Live Streaming Display
Image<Bgr, Byte> ImageFrame = capture.QueryFrame();
// If Ip camera try to reinitialize the IP camera
if(ImageFrame == null)
{
capture.Dispose();
capture = new Capture(URL);
ImageFrame = capture.QueryFrame();
}
ImageFrame = ImageFrame.Resize(img.Width, img.Height, Emgu.CV.CvEnum.INTER.CV_INTER_LINEAR);
img.Image = ImageFrame;
// Here I am doing some other operations like
// 1. Save Image captured from the IP Camera
// 2. Detect faces in Image
// 3. Draw Face markers on Image
// 4. Some database based on result of Face Detection
// 4. Delete image File
// continue Looping for other Ip Cameras
}
catch (NullReferenceException e)
{
}
}
Now, The Problem is after some time the QueryFrame() provide null value and camera Stop streaming.
Can any one tell me why this is happening?
How I can resolve this problem?
If any more information is needed Please Let me know.
Thanks in Advance.

Sorry about the delay but I have provide an example that works with several public IP cameras. It will need the EMGU reference replacing with your current version and the target build directory should be set to "EMGU Version\bin" alternatively extract it to the examples folder.
http://sourceforge.net/projects/emguexample/files/Capture/CameraCapture%20Public%20IP.zip/download
Rather than using the older QueryFrame() method it uses the RetrieveBgrFrame() method. It has worked reasonably well and I have had no null exceptions. However if you do replace the ProcessFrame() method with something like this
You should not be attempting to do any operations if the frame returned is Image is a nullable field and should not have a problem if _capture.RetrieveBgrFrame(); returns null if there is a problem then there is a bigger issue.
private void ProcessFrame(object sender, EventArgs arg)
{
//If you want to access the image data the use the following method call
//Image<Bgr, Byte> frame = new Image<Bgr,byte>(_capture.RetrieveBgrFrame().ToBitmap());
if (RetrieveBgrFrame.Checked)
{
Image<Bgr, Byte> frame = _capture.RetrieveBgrFrame();
//because we are using an autosize picturebox we need to do a thread safe update
if(frame!=null)
{
DisplayImage(frame.ToBitmap());
Image<Bgr, Byte> ImageFrame = frame.Resize(img.Width, img.Height, Emgu.CV.CvEnum.INTER.CV_INTER_LINEAR);
// Here I am doing some other operations like
// 1. Save Image captured from the IP Camera
// 2. Detect faces in Image
// 3. Draw Face markers on Image
// 4. Some database based on result of Face Detection
// 4. Delete image File
// continue Looping for other Ip Cameras
}
//else do nothing as we have no image
}
else if (RetrieveGrayFrame.Checked)
{
Image<Gray, Byte> frame = _capture.RetrieveGrayFrame();
//because we are using an autosize picturebox we need to do a thread safe update
if (frame != null) DisplayImage(frame.ToBitmap());
}
}
On a separate note your comment 'continue Looping for other Ip Cameras' may cause several issues. You should have a new Capture constructor for each camera camera you are using. How many camera are you using? and what public ip camera are you using so I can attempt to replicate the issue? The reason for the separate constructor is that ip cameras take a while to negotiate connections with and constantly Disposing of the original construct and replacing it will play havoc with the garbage collector and introduce no end if timing issues.
Cheers
Chris
[EDIT]
If your camera is returning null frames after a timeout period then I would check to see if there is an issue with the setup or maybe your connection is so slow it disconnects you to reduce lag to others there are various causes but this is not a code problem. You can use c# alone to acquire the data to a bitmap and then pass this to an Image type variable. There is a great article here:
http://www.codeproject.com/Articles/15537/Camera-Vision-video-surveillance-on-C
I've adapted this so you can use a HttpWebRequest as a final check to see if the stream is alive although there are still null exceptions that will be produced here:
using System.Net;
using System.IO;
string url;
private void ProcessFrame(object sender, EventArgs arg)
{
//***If you want to access the image data the use the following method call***/
//Image<Bgr, Byte> frame = new Image<Bgr,byte>(_capture.RetrieveBgrFrame().ToBitmap());
if (RetrieveBgrFrame.Checked)
{
Image<Bgr, Byte> frame = _capture.RetrieveBgrFrame();
//because we are using an autosize picturebox we need to do a thread safe update
if (frame != null)
{
DisplayImage(frame.ToBitmap());
}
else
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
// get response
WebResponse resp = req.GetResponse();
//get stream
Stream stream = resp.GetResponseStream();
if (!stream.CanRead)
{
//try reconnecting the camera
captureButtonClick(null, null); //pause
_capture.Dispose();//get rid
captureButtonClick(null, null); //reconnect
}
}
}
else if (RetrieveGrayFrame.Checked)
{
Image<Gray, Byte> frame = _capture.RetrieveGrayFrame();
//because we are using an autosize picturebox we need to do a thread safe update
if (frame != null) DisplayImage(frame.ToBitmap());
}
}
private void captureButtonClick(object sender, EventArgs e)
{
url = Camera_Selection.SelectedItem.ToString(); //add this
... the rest of the code
}
To display multiple webcams you would create a class to handle the Capture construct and processframe event. Ideally you would raise an purpose built event call that would include a camera identifier as the frameready event call does not call this. I have to make things easier created a form with as a MDI parent and opened an object to manage the capture variables and frame ready event. The Alpha version is available here:
http://sourceforge.net/projects/emguexample/files/Capture/CameraCapture%20Public%20IP%20Multipl%20Display%20Alpha.zip/download

fileDialogue Bitmaps

I would like to do some image processing in C# and need to align two images before applying a filter to them. I will attempt to do this by scanning the images at a fixed point in a small rectangular section, which I believe makes it necessary to use the Bitmap class.
This section has a large amount of white pixels so I would like to take an average pixel value in this area to find the shift in the y-axis, as there is a large white horizontal bar going across the images.
The x-axis will be the same in both images. I would like to setup a few test images with different shift values, from small to large, positive and negative, so I can search for the minimum value .
This will require a scroll bar on the images to move them in small amounts.
I am totally new to C#, and low level programmer. I have been trying to get the image in pictureBox1 with the following code.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing.Imaging;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
namespace imageAlign
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private void button1_Click(object sender, EventArgs e)
{
Bitmap myImage = (Bitmap)pictureBox1.Image;
OpenFileDialog ofd = new OpenFileDialog();
if (ofd.ShowDialog() == System.Windows.Forms.DialogResult.OK) ;
{
pictureBox1.Image = Image.FromFile();
// this.pictureBox1.Image = myImage;
}
}
}
}
I have left the Image.FromFile(); with nothing passed as I wish to choose the images when I click the button on the form. Currently, I only have one button and picture box.

You need to use the FileName property of the OpenFileDialog:
if (ofd.ShowDialog() == DialogResult.OK && ofd.FileName != "")
{
pictureBox1.Image = Image.FromFile(ofd.FileName);
}
From the link:
The file name includes both the file path and the extension. If no files are selected, this method returns an empty string ("").