We Keep Coding

Converting Texture2D into a video

I've did a lot of research, but I can't find a suitable solution that works with Unity3d/c#. I'm using a Fove-HMD and would like to record/make a video of the integrated camera. So far I managed every update to take a snapshot of the camera, but I can't find a way to merge this snapshots into a video. Does someone know a way of converting them? Or can someone point me in the right direction, in which I could continue my research?
public class FoveCamera : SingletonBase<FoveCamera>{
private bool camAvailable;
private WebCamTexture foveCamera;
private List<Texture2D> snapshots;
void Start ()
{
//-------------just checking if webcam is available
WebCamDevice[] devices = WebCamTexture.devices;
if (devices.Length == 0)
{
Debug.LogError("FoveCamera could not be found.");
camAvailable = false;
return;
}
foreach (WebCamDevice device in devices)
{
if (device.name.Equals("FOVE Eyes"))
foveCamera = new WebCamTexture(device.name);//screen.width and screen.height
}
if (foveCamera == null)
{
Debug.LogError("FoveCamera could not be found.");
return;
}
//-------------camera found, start with the video
foveCamera.Play();
camAvailable = true;
}
void Update () {
if (!camAvailable)
{
return;
}
//loading snap from camera
Texture2D snap = new Texture2D(foveCamera.width,foveCamera.height);
snap.SetPixels(foveCamera.GetPixels());
snapshots.Add(snap);
}
}
The code works so far. The first part of the Start-Method is just for finding and enabling the camera. In the Update-Method I'm taking every update a snapshot of the video.
After I "stop" the Update-Method, I would like to convert the gathered Texture2D object into a video.
Thanks in advance

Create MediaEncoder
using UnityEditor; // VideoBitrateMode
using UnityEditor.Media; // MediaEncoder
var vidAttr = new VideoTrackAttributes
{
bitRateMode = VideoBitrateMode.Medium,
frameRate = new MediaRational(25),
width = 320,
height = 240,
includeAlpha = false
};
var audAttr = new AudioTrackAttributes
{
sampleRate = new MediaRational(48000),
channelCount = 2
};
var enc = new MediaEncoder("sample.mp4", vidAttr, audAttr);
Convert each snapshot to Texture2D
Call consequently AddFrame to add each snapshot to MediaEncoder
enc.AddFrame(tex);
Once done call Dispose to close the file
enc.Dispose();

I see two methods here, one is fast to implement, dirty and not for all platforms, second one harder but pretty. Both rely on FFMPEG.
1) Save every frame into image file (snap.EncodeToPNG()) and then call FFMPEG to create video from images (FFmpeg create video from images) - slow due to many disk operations.
2) Use FFMPEG via wrapper implemented in AForge and supply its VideoFileWriter class with images that you have.
Image sequence to video stream?
Problem here is it uses System.Bitmap, so in order to convert Texture2D to Bitmap you can use: How to create bitmap from byte array?
So you end up with something like:
Bitmap bmp;
Texture2D snap;
using (var ms = new MemoryStream(snap.EncodeToPNG()))
{
bmp = new Bitmap(ms);
}
vFWriter.WriteVideoFrame(bmp);
Both methods are not the fastest ones though, so if performance is an issue here you might want to operate on lower level data like DirectX or OpenGL textures.

How to adjust jpeg quality with Magick.Net

I am trying to set the image quality of two images appended to one another to 10% and resize the images to 40x40.
using (var images = new MagickImageCollection {designFile, swatchFile})
{
MagickImage sprite = images.AppendHorizontally();
sprite.Format = MagickFormat.Jpeg;
sprite.SetOption(MagickFormat.Jpeg, "quality", "10%");
sprite.SetOption(MagickFormat.Jpeg, "size", "40x40"); ;
sprite.Write(spriteFile);
}
Unfortunately the SetOption and Format calls don't seem to be affecting the file that is written to sprite.Write()?

The method SetOption is the same as -define in ImageMagick. And this method will be renamed to SetDefine in the next release. The following resizes your image to 40x40 and uses a quality of 10%.
using (MagickImage sprite = images.AppendHorizontally())
{
sprite.Format = MagickFormat.Jpeg;
sprite.Quality = 10;
sprite.Resize(40, 40);
sprite.Write(spriteFile);
}
If you need more help feel free to post another question here: https://magick.codeplex.com/discussions

Application fails visualizing more than 600 Images

I wrote a WPF app that should swap (fast) between a large set of images (600+, 190Kb average size), but I'm finding some difficulties.
private int appendImages(Canvas c, int start, int end)
{
int tot = 0;
for (int i = start; i < end; i++)
{
BitmapImage bi = new BitmapImage();
bi.BeginInit();
//bi.CacheOption = BitmapCacheOption.OnLoad;
bi.UriSource = new Uri(appFolder+#"/"+imgFolder+"/"+filename(i)+".jpg");
bi.EndInit();
Image img = new Image
{
Width = imgWidth,
Height = imgHeight,
Source = bi,
Name = name(i),
Visibility = i == startImg ? Visibility.Visible : Visibility.Hidden
};
c.Children.Add(img);
tot++;
}
}
Apparently the inizialization is fine, but if I try to swap the images like this:
private void changeImageTo(int n)
{
Image img = findImage(n);
Image old = findImage(prevImg);
if (img != null)
{
img.Visibility = Visibility.Visible;
if (old != null && old != img)
old.Visibility = Visibility.Hidden;
prevImg = n;
}
}
..then the app shows the first 200/300 images (depending on the sources I use), and the others are just empty/blank (i can see the canvas underneath).
I suspect it's a memory issue, but I'm not really sure what causes it.
By the way, if I uncomment the commented line (BitmapCacheOption.OnLoad) sometimes I get a vshost error when launching the app.
Any help would be MUCH appreciated, since I couldn't find anything useful browsing around.
Thanks in advance!

It looks like you're loading all the images at once, and putting them into WinForms/WPF controls. That is a very bad idea with that many images, as each one takes resources even if it's not shown.
Rough back of the envelope calculation, assuming 640x480 images, 24bpp being the native GDI+ format, shows a bit over 2gb for loading all the images at once, and that would, of course, increase exponentially with image size.
What I would do instead, is have only one Image. Move the actual image loading code into your changeImageTo function, build the file name based on n, and set the loaded image to the Image there.

Image resize optimization causes memory usage to sky rocket

I made a function that takes an original image and resizes it to 3 different zoom scales -> 16x, 10x, and 4x. For a better understanding, continue reading this paragraph. Let's say the original image is 1000x1000. I declare that at 1x zoom it's dimensions will be 50x50. That means 4x zoom will be 200x200, 10x zoom will be 500x500 and 16x zoom will be 800x800. So my function needs to resize the original 1000x1000 down to 800x800, then down to 500x500, then down to 200x200. Please note I have done this successfully and my question is regarding memory usage.
Below I have two methods of doing so. Both methods work, but one causes a HUGE memory usage bloat using approximately 3x/4x more memory than the other... I like the 2nd method better, because it loads significantly faster than the first method because it's not resizing each of the 3 images from the original image, instead it's resizing them from the previously resized image.
Notes: I'm using Xcode Instruments to measure memory usage. The ImageResizer Class contains a function called "Resize" which resizes the Image.
Method 1.)
public List<UIImage> InitImageList_BFObjects ( UIImage image, SizeF frameSize )
{
List<UIImage> listOfImages = new List<UIImage>();
for ( int i = 0; i < 3; i++ )
{
if ( i == 0 )
zoomScale = 16f;
else if ( i == 1 )
zoomScale = 10f;
else// if ( i == 2 )
zoomScale = 4f;
Resizer = new ImageResizer(image);
Resizer.Resize(frameSize.Width * zoomScale, frameSize.Height * zoomScale);
UIImage resizedImage = Resizer.ModifiedImage;
listOfImages.Insert(0, resizedImage);
}
return listOfImages;
}
Method 1 works and uses very little memory usage. I ran this with a group of about 20 images. My app had about 14mb of memory usage after this loaded (using Xcodes Instruments to examine memory usage)
Method 2.)
public List<UIImage> InitImageList_BFObjects ( UIImage image, SizeF frameSize )
{
List<UIImage> listOfImages = new List<UIImage>();
for ( int i = 0; i < 3; i++ )
{
if ( i == 0 )
zoomScale = 16f;
else if ( i == 1 )
zoomScale = 10f;
else// if ( i == 2 )
zoomScale = 4f;
if ( listOfImages.Count == 0 )
{
Resizer = new ImageResizer(image);
Resizer.Resize(frameSize.Width * zoomScale, frameSize.Height * zoomScale);
UIImage resizedImage = Resizer.ModifiedImage;
listOfImages.Insert(0, resizedImage);
}
else
{
// THIS LINE CONTAINS THE MAIN DIFFERENCE BETWEEN METHOD 1 AND METHOD 2
// Notice how it resizes from the most recent image from listOfImages rather than the original image
Resizer = new ImageResizer(listOfImages[0]);
Resizer.Resize(frameSize.Width * zoomScale, frameSize.Height * zoomScale);
UIImage resizedImage = Resizer.ModifiedImage;
listOfImages.Insert(0, resizedImage);
}
}
return listOfImages;
}
Method 2 works but the memory usage sky rockets! I ran this with the same group of about 20 images. My app had over 60mb of memory usage after this loaded (using Xcodes Instruments to examine memory usage) Why is the memory usage so high? What is it about Method 2 that causes the memory to sky rocket? It's almost as if a variable is not getting cleaned up properly
* Additional Information, ImageResizer Class **
I cut out the non-needed functions from my ImageResizer Class and renamed it "ImageResizer_Abridged". I even switched over to using this class to make sure I didn't accidentally cut out anything needed.
public class ImageResizer_Abridged
{
UIImage originalImage = null;
UIImage modifiedImage = null;
public ImageResizer_Abridged ( UIImage image )
{
this.originalImage = image;
this.modifiedImage = image;
}
/// <summary>
/// strech resize
/// </summary>
public void Resize( float width, float height )
{
UIGraphics.BeginImageContext( new SizeF( width, height ) );
//
modifiedImage.Draw( new RectangleF( 0,0, width, height ) );
modifiedImage = UIGraphics.GetImageFromCurrentImageContext();
//
UIGraphics.EndImageContext();
}
public UIImage OriginalImage
{
get
{
return this.originalImage;
}
}
public UIImage ModifiedImage
{
get
{
return this.modifiedImage;
}
}
}
I created a simplified test project showing this problem *
Here is a dropbox link to the project: https://www.dropbox.com/s/4w7d87nn0aafph9/TestMemory.zip
Here is Method 1's Xcode Instruments screen shot as evidence (9 mb memory usage):
http://i88.photobucket.com/albums/k194/lampshade9909/AllImagesResizedFromOriginalImage_zps585228c6.jpg
Here is Method 2's Xcode Instruments screens hot as evidence (55 mb memory usage):
http://i88.photobucket.com/albums/k194/lampshade9909/SignificantIncreaseInMemoryUsage_zps19034bad.jpg
Below is the code block needed to run the test project
// Initialize My List of Images
ListOfImages = new List<UIImage>();
for ( int i = 0; i < 30; i++ )
{
// Create a UIImage Containing my original Image
UIImage originalImage = UIImage.FromFile ("b2Bomber.png");
float newWidth = 100f;
float newHeight = 40f;
float zoomScale;
float resizedWidth, resizedHeight;
UIImage resizedImage1;
UIImage resizedImage2;
// Basically, I want to take the originalImage Image and resize it twice.
// Method 1.) Resize the originalImage and save it as ResizedImage1. Resize the originalImage and save it as ResizedImage2. We're finished!
// Method 2.) Resize the originalImage and save it as ResizedImage1. Resize ResizedImage1 and save it as ResizedImage2. We're finished!
// The pro to Method 1 is that we get the best possible quaility on all resized images. The con is, this takes a long time if we're doing dozens of very large images
// The pro to Method 2 is that it's faster than Method 1. This is why I want to use Method 2, it's speed. But it has a HUGE con, it's memory usage.
// Please run this project on an iPad connected to XCodes Instruments to monitor memory usage and see what I mean
zoomScale = 10f;
resizedWidth = newWidth*zoomScale;
resizedHeight = newHeight*zoomScale;
UIGraphics.BeginImageContext( new SizeF( resizedWidth, resizedHeight ) );
originalImage.Draw( new RectangleF( 0, 0, resizedWidth, resizedHeight ) );
resizedImage1 = UIGraphics.GetImageFromCurrentImageContext();
UIGraphics.EndImageContext();
zoomScale = 4f;
resizedWidth = newWidth*zoomScale;
resizedHeight = newHeight*zoomScale;
UIGraphics.BeginImageContext( new SizeF( resizedWidth, resizedHeight ) );
// Run this project on an iPad and examine the memory usage in XCode's Instruments.
// The Real Memory Usage will be aroud 9 MB.
// Uncomment this "originalImage.Draw" line to see this happening, make sure to comment out the "resizedImage1.Draw" line
// originalImage.Draw( new RectangleF( 0, 0, resizedWidth, resizedHeight ) );
// Run this project on an iPad and examine the memory usage in XCode's Instruments.
// The Real Memory Usage will be aroud 55 MB!!
// My question is, why does the memory sky rocket when doing this, and how can I prevent the memory from sky rocketing??
// My App requires me to resize around a hundred images and I want to be able to resize an already resized image (like in this example) without the memory usage sky rocketing like this...
// Uncomment this "resizedImage1.Draw" line to see this happening, make sure to comment out the "originalImage.Draw" line
resizedImage1.Draw( new RectangleF( 0, 0, resizedWidth, resizedHeight ) );
resizedImage2 = UIGraphics.GetImageFromCurrentImageContext();
UIGraphics.EndImageContext();
// Add my resized images to the list of Images
ListOfImages.Add (resizedImage1);
ListOfImages.Add (resizedImage2);
}

I'm not sure about your Resize code but I have seen Scale do strange thing. It's not really strange, once you dig into it, but it's definitively not obvious.
Creating an UIImage can be very cheap, memory wise, as long as its backing CGImage is not created. IOW iOS might not immediately allocate a new CGImage backing image that match the new size. That allocation will be differed until the CGImage is needed.
In such case it's possible for some code (like your method 1) to require almost no additional memory when scaling up. However your 2nd method is using a scaled-up image (and will need to allocate the backing CGImage to do so) so it end up requiring the memory earlier.
How can you check for this ?
Compare your resizedImage.Size with the resizedImage.CGImage.Size. If they don't match then you're likely hitting caching.
Notes
I say might because the caching logic is unknown (undocumented). I know that this can differ from running on the simulator and devices - and it also vary between iOS versions;
Caching is a good thing - but it can be surprising :-) I just wish this was documented.

Have you checked whether Resizer implements the Dispose() method? I don't see you disposing it anywhere.
I believe your new code line is implementing the Zoom on the entire image, hence the increased memory usage.
Resizer is zooming the ENTIRE image at the new zoomed in scale, so that an incoming 4MB image is zoomed to 8MB, 16MB and 32MB, consuming your memory.

UIImage implements IDisposable, so something will have to Dispose of it eventually. The Resize method appears to "lose" the reference to modifiedImage, so I'm going to Dispose() it. Hopefully the caller is doing the same to all images in the list returned by InitImageList_BFObjects when it's done with them. Or that class implements IDisposable and it's kicked up the line of who has to deal with it. But rest assured, these images being created need to be Dispose()d somewhere sometime.
public class ImageResizer_Abridged
{
private readonly UIImage originalImage;
private UIImage modifiedImage;
public ImageResizer_Abridged(UIImage image)
{
this.originalImage = image;
this.modifiedImage = image;
}
/// <summary>
/// stretch resize
/// </summary>
public void Resize(float width, float height)
{
UIGraphics.BeginImageContext(new SizeF(width, height));
//
var oldImage = this.modifiedImage;
this.modifiedImage.Draw(new RectangleF(0, 0, width, height));
this.modifiedImage = UIGraphics.GetImageFromCurrentImageContext();
oldImage.Dispose();
//
UIGraphics.EndImageContext();
}
public UIImage OriginalImage
{
get
{
return this.originalImage;
}
}
public UIImage ModifiedImage
{
get
{
return this.modifiedImage;
}
}
}

Processing large bitmap images in WPF

I need to process (change brightness, contrast etc) very large high-quality bitmaps (often over 10MPx) several times per second and need to update it on screen every time ( on Image control in WPF). Currently I'm using AForge.NET library for unmanaged image processing, but there are some problems I cannot solve. First of all, one operation takes ~300ms (without updating the screen) which is not acceptable for me. Here's sample code:
UnmanagedImage _img;
BrightnessCorrection _brightness = new BrightnessCorrection();
void Load()
{
_img = UnmanagedImage.FromManagedImage((Bitmap)Bitmap.FromFile("image.jpg"));
}
void ChangeBrightness(int val) // this method is invoked by changing Slider value - several times per second
{
_brightness.AdjustValue = val;
_brightness.ApplyInPlace(_img); // it takes ~300ms for image 22MPx, no screen update - just change brightness "in background"
}
I have no experience in image processing, but I think it cannot be much faster since it is very high resolution. Am I right?
Another problem - how to efficiently update the screen? At the moment I have the following (ofc very bad) solution:
void ChangeBrightness(int val)
{
_brightness.AdjustValue = val;
_brightness.ApplyInPlace(_img);
using (MemoryStream ms = new MemoryStream())
{
using (Bitmap b = _img.ToManagedImage())
{
b.Save(ms, ImageFormat.Bmp);
ms.Seek(0, SeekOrigin.Begin);
var bmp = new BitmapImage();
bmp.BeginInit();
bmp.StreamSource = ms;
bmp.CacheOption = BitmapCacheOption.OnLoad;
bmp.EndInit();
MyImageControl.Source = new WriteableBitmap(bmp); // !!!
}
}
}
As you can see, every time new WriteableBitmap is created (you can imagine what is happenin). Instead of these "usings" I tried that way:
WriteableBitmapSource.Lock(); // this object (of type WriteableBitmap) is just MVVM ViewModel's property which is binded to MyImageControl.Source
WriteableBitmapSource.Source.WritePixels(new Int32Rect(0, 0, _img.Width, _img.Height), _img.ImageData, _img.Stride * _img.Height * 3, _img.Stride, 0, 0); // image's PixelFormat is 24bppRgb
... but WritePixels method throws "Value does not fall within the expected range." Any ideas why?
Any help will be much appreciated :)
P.S.
Is AForge.NET a good choice at all? Maybe there is better image processing lib?
sorry for my english ;P

~300ms for image 22MPx is about 20 ns per pixel. That should be about right.
You need to consider CPU cost and memory access cost.
If you want to improve this further, consider:
1) Use multiple threads, each responsible for a section of the bitmap.
2) Write your own implementation, using SIMD instructions.
3) Do not pre-process the bitmap, transform bitmap scanline when they're needed.