I've did a lot of research, but I can't find a suitable solution that works with Unity3d/c#. I'm using a Fove-HMD and would like to record/make a video of the integrated camera. So far I managed every update to take a snapshot of the camera, but I can't find a way to merge this snapshots into a video. Does someone know a way of converting them? Or can someone point me in the right direction, in which I could continue my research?
public class FoveCamera : SingletonBase<FoveCamera>{
private bool camAvailable;
private WebCamTexture foveCamera;
private List<Texture2D> snapshots;
void Start ()
{
//-------------just checking if webcam is available
WebCamDevice[] devices = WebCamTexture.devices;
if (devices.Length == 0)
{
Debug.LogError("FoveCamera could not be found.");
camAvailable = false;
return;
}
foreach (WebCamDevice device in devices)
{
if (device.name.Equals("FOVE Eyes"))
foveCamera = new WebCamTexture(device.name);//screen.width and screen.height
}
if (foveCamera == null)
{
Debug.LogError("FoveCamera could not be found.");
return;
}
//-------------camera found, start with the video
foveCamera.Play();
camAvailable = true;
}
void Update () {
if (!camAvailable)
{
return;
}
//loading snap from camera
Texture2D snap = new Texture2D(foveCamera.width,foveCamera.height);
snap.SetPixels(foveCamera.GetPixels());
snapshots.Add(snap);
}
}
The code works so far. The first part of the Start-Method is just for finding and enabling the camera. In the Update-Method I'm taking every update a snapshot of the video.
After I "stop" the Update-Method, I would like to convert the gathered Texture2D object into a video.
Thanks in advance
Create MediaEncoder
using UnityEditor; // VideoBitrateMode
using UnityEditor.Media; // MediaEncoder
var vidAttr = new VideoTrackAttributes
{
bitRateMode = VideoBitrateMode.Medium,
frameRate = new MediaRational(25),
width = 320,
height = 240,
includeAlpha = false
};
var audAttr = new AudioTrackAttributes
{
sampleRate = new MediaRational(48000),
channelCount = 2
};
var enc = new MediaEncoder("sample.mp4", vidAttr, audAttr);
Convert each snapshot to Texture2D
Call consequently AddFrame to add each snapshot to MediaEncoder
enc.AddFrame(tex);
Once done call Dispose to close the file
enc.Dispose();
I see two methods here, one is fast to implement, dirty and not for all platforms, second one harder but pretty. Both rely on FFMPEG.
1) Save every frame into image file (snap.EncodeToPNG()) and then call FFMPEG to create video from images (FFmpeg create video from images) - slow due to many disk operations.
2) Use FFMPEG via wrapper implemented in AForge and supply its VideoFileWriter class with images that you have.
Image sequence to video stream?
Problem here is it uses System.Bitmap, so in order to convert Texture2D to Bitmap you can use: How to create bitmap from byte array?
So you end up with something like:
Bitmap bmp;
Texture2D snap;
using (var ms = new MemoryStream(snap.EncodeToPNG()))
{
bmp = new Bitmap(ms);
}
vFWriter.WriteVideoFrame(bmp);
Both methods are not the fastest ones though, so if performance is an issue here you might want to operate on lower level data like DirectX or OpenGL textures.
Related
I'm trying to get an image in skiasharp that's left rotated by 90 degrees to be centered and fit perfectly on the canvas. I've tried 2 ways. My own custom way, and another one that seems like a popular solution but maybe I'm not understanding how it works correctly?
My own way.
Here is the code:
SKSurface surf = e.Surface;
SKCanvas canvas = surf.Canvas;
SKSize size = canvasView.CanvasSize;
canvas.Clear();
SKRect rect = SKRect.Create(0.0f, 0.0f, size.Height, size.Width);
canvas.RotateDegrees(85);
canvas.DrawBitmap(m_bm, rect);
"m_bm" is a bitmap that was retrieved in a separate function. That function is:
// Let user take a picture.
var result = await MediaPicker.CapturePhotoAsync(new MediaPickerOptions
{
Title = "Take a picture"
});
// Save stream.
var stream = await result.OpenReadAsync();
// Create the bitmap.
m_bm = SKBitmap.Decode(stream);
// Set to true because the image will be prepared soon.
m_displayedImage = true;
I only put 85 instead of 90 because I wanted to visually see it getting closer but when I do that, it goes off screen. I'm coming from a game programming background so this is normally solved with getting the width of whatever we're working with (like the player in the game) and adding that to the x position, and boom. But with Xamarin, didn't work. That's my own attempt. Then I hit the internet of course to find help, and a different implementation was given to me.
Popular solution.
See here for this popular solution and it's the FIRST answer to this users question. The code I used is SLIGHTLY different because I didn't see the point in returning an image in that users function. Here it is:
// Save stream.
var stream = await result.OpenReadAsync();
using (var bitmap = SKBitmap.Decode(stream))
{
var rotated = new SKBitmap(bitmap.Height, bitmap.Width);
using (var surface = new SKCanvas(rotated))
{
surface.Clear();
surface.Translate(rotated.Height, rotated.Width);
surface.RotateDegrees(90);
surface.DrawBitmap(bitmap, 0, 0);
}
}
I'm drawing the bitmap with the canvas and I thought that would work because testing it in other code samples it did exactly that, so I definitely am not rotating properly or something?
The link that #Cheesebaron gave me in the reply to the original post ended up working out. But a new issue arises but I'll google that myself. Here's my own code:
namespace BugApp
{
public partial class MainPage : ContentPage
{
// Save bitmaps for later use.
static SKBitmap m_bm;
static SKBitmap m_editedBm;
// Boolean for displaying the image captured with the camera.
bool m_displayedImage;
public MainPage()
{
// Set to explicit default values to always be in control of the assignments.
m_bm = null;
m_editedBm = null;
// No picture has been taken yet.
m_displayedImage = false;
InitializeComponent();
}
// Assigned to the button in the xaml page.
private async void SnapPicture(Object sender, System.EventArgs e)
{
// Let user take a picture.
var result = await MediaPicker.CapturePhotoAsync(new MediaPickerOptions
{
Title = "Take a picture"
});
// Save stream.
var stream = await result.OpenReadAsync();
// Create the bitmap.
m_bm = SKBitmap.Decode(stream);
// Get the rotated image.
m_editedBm = Rotate();
// Set to true because the image will be prepared soon.
m_displayedImage = true;
}
public static SKBitmap Rotate()
{
using (var bitmap = m_bm)
{
var rotated = new SKBitmap(bitmap.Height, bitmap.Width);
using (var surface = new SKCanvas(rotated))
{
surface.Translate(bitmap.Width, 0);
surface.RotateDegrees(90);
surface.DrawBitmap(bitmap, 0, 0);
}
return rotated;
}
}
private void OnCanvasViewPaintSurface(object sender, SKPaintSurfaceEventArgs e)
{
if(m_bm != null && m_displayedImage == true)
{
e.Surface.Canvas.Clear();
// Draw in a new rect space.
e.Surface.Canvas.DrawBitmap(m_editedBm, new SKRect(0.0f, 0.0f, 300.0f, 300.0f));
// ---Testing.
// e.Surface.Canvas.DrawBitmap(m_editedBm, new SKRect(112, 238, 184, 310), new SKRect(0, 0, 9, 9));
// Avoid having this function launch again for now.
m_displayedImage = false;
}
}
}
}
The main portion of code that matters is the rotate function which was this one here: Link.
Thanks to everyone that replied.
I am trying to create an application that generates a bitmap image every frame based on user actions and have it display that image to the screen. I would like the application to also be able to update that image in unity in real time as soon as the user makes another action.
I have created an application that does this and it works. However, it is veryyyy slow. My Update() method is attached below.
My idea was:
Capture user data (mouse location).
Convert that data into a special signal format that another program recognizes.
Have that program return a bitmap image.
Use that bitmap as a texture and update the existing texture with the new image.
Code:
UnityEngine.Texture2D oneTexture;
Bitmap currentBitmap;
private int frameCount = 0;
void Update()
{
// Show mouse position in unity environment
double xValue = Input.mousePosition.x;
double yValue = Screen.height - Input.mousePosition.y;
myPoints = "" + xValue + "," + yValue + Environment.NewLine;
// Show heatmap being recorded.
signals = Program.ConvertStringToSignalsList(myPoints);
currentBitmap = Program.CreateMouseHeatmap(Screen.width, Screen.height, signals);
// Update old heatmap texture.
UpdateTextureFromBitmap();
ri.texture = oneTexture;
ri.rectTransform.sizeDelta = new Vector2(Screen.width, Screen.height);
frameCount++;
// Write points to Database.
StartCoroutine(WriteToDB(xValue, yValue)); // <<<<< Comment out when playback.
}
private void UpdateTextureFromBitmap()
{
// Convert Bitmap object into byte array instead of creating actual
// .bmp image file each frame.
byte[] imageBytes = ImageToBytes(currentBitmap);
BMPLoader loader = new BMPLoader();
BMPImage img = loader.LoadBMP(imageBytes);
// Only initialize the Texture once.
if (frameCount == 0)
{
oneTexture = img.ToTexture2D();
}
else
{
Color32[] imageData = img.imageData;
oneTexture.SetPixels32(imageData);
oneTexture.Apply();
}
}
I was wondering if someone could help me improve the rate at which the image updates to the screen? I know that it is possible to make this program much faster but I am so new to unity and C# that I don't know how to make that happen. Also if there is a completely different way that I should be going about doing this then I am open to that too. Any help would be appreciated. Thanks!
Also, below is a screenshot of the Profiler showing the breakdown of CPU Usage. Currently it looks like every frame is taking about 500ms.
I'm trying to create a Screenshot of all Screens on my PC. In the past I've been using the GDI Method, but due to performance issues I'm trying the DirectX way.
I can take a Screenshot of a single Screen without issues, with a code like this:
using Microsoft.DirectX;
using Microsoft.DirectX.Direct3D;
using System.Windows.Forms;
using System.Drawing;
class Capture : Form
{
private Device device;
private Surface surface;
public Capture()
{
PresentParameters p = new PresentParameters();
p.Windowed = true;
p.SwapEffect = SwapEffect.Discard;
device = new Device(0, DeviceType.Hardware, this, CreateFlags.HardwareVertexProcessing, p);
surface = device.CreateOffscreenPlainSurface(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height, Format.A8B8G8R8, Pool.Scratch);
}
public Bitmap Frame()
{
GraphicsStream gs = SurfaceLoader.SaveToStream(ImageFileFormat.Jpg, surface);
return new Bitmap(gs);
}
}
(Lets ignore deleting the Bitmap from memory for this question)
With that Code I can take a Screenshot of my Primary Screen. Changing the first parameter of the Device constructor to a different number corresponds to a different Screen. If I have 3 Screens and I pass 2 as a parameter, I get a Screenshot of my third Screen.
The issue I have is how to handle capturing all Screens. I came up with the following:
class CaptureScreen : Form
{
private int index;
private Screen screen;
private Device device;
private Surface surface;
public Rectangle ScreenBounds { get { return screen.Bounds; } }
public Device Device { get { return device; } }
public CaptureScreen(int index, Screen screen, PresentParameters p)
{
this.screen = screen; this.index = index;
device = new Device(index, DeviceType.Hardware, this, CreateFlags.HardwareVertexProcessing, p);
surface = device.CreateOffscreenPlainSurface(screen.Bounds.Width, screen.Bounds.Height, Format.A8R8G8B8, Pool.Scratch);
}
public Bitmap Frame()
{
device.GetFrontBufferData(0, surface);
GraphicsStream gs = SurfaceLoader.SaveToStream(ImageFileFormat.Jpg, surface);
return new Bitmap(gs);
}
}
class CaptureDirectX : Form
{
private CaptureScreen[] screens;
private int width = 0;
private int height = 0;
public CaptureDirectX()
{
PresentParameters p = new PresentParameters();
p.Windowed = true;
p.SwapEffect = SwapEffect.Discard;
screens = new CaptureScreen[Screen.AllScreens.Length];
for (int i = 0; i < Screen.AllScreens.Length; i++)
{
screens[i] = new CaptureScreen(i, Screen.AllScreens[i], p);
//reset previous devices
if (i > 0)
{
for(int j = 0; j < i; j++)
{
screens[j].Device.Reset(p);
}
}
width += Screen.AllScreens[i].Bounds.Width;
if (Screen.AllScreens[i].Bounds.Height > height)
{
height = Screen.AllScreens[i].Bounds.Height;
}
}
}
public Bitmap Frame()
{
Bitmap result = new Bitmap(width, height);
using (var g = Graphics.FromImage(result))
{
for (int i = 0; i < screens.Length; i++)
{
Bitmap frame = screens[i].Frame();
g.DrawImage(frame, screens[i].Bounds);
}
}
return result;
}
}
As you can see, I iterate though the available Screens and create multiple devices and surfaces in a seperate Class. But calling Frame() of the CaptureDirectX class throws the following error:
An unhandled exception of type 'Microsoft.DirectX.Direct3D.InvalidCallException' occurred in Microsoft.DirectX.Direct3D.dll
At the line
device.GetFrontBufferData(0, surface);
I've been researching this a bit but didn't have a whole lot of success. I'm not really sure what the issue is.
I've found a link that offers a solution that's talking about resetting the Device Objects. But as you can see in my code above, I've been trying to reset all previously created Device objects, sadly without success.
So my questions are:
Is what I'm trying to achieve even possible through this method (i.e. GetFrontBufferData) ?
What am I doing wrong? What am I missing?
Do you see any performance issues when capturing the Screen at a high rate, like say 30 fps? (Capturing a single screen with a target of 30fps gave me a rate of about 25 - 30fps, compared with the GDI methology which sinks to like 15fps sometimes)
FYI it's a WPF application, i.e. .NET 4.5
Edit: I should mention that I'm aware of IDXGI_DesktopDuplication but sadly it doesn't fit my requirements. As far as I know, that API is only available from Windows 8 onwards, but I'm trying to get a solution that works from Windows 7 onwards because of my clients.
Well, in the end the solution was something completely different. The System.Windows.Forms.Screen Class doesn't play nicely with the DirectX Classes. Why? Because the indexes don't match up. The first object in AllScreens does not necessarly have to be index 0 in the Device instatiation.
Now usually this isn't a problem, except when you have a "strange" monitor setup like mine. On the desk I have 3 screens, one vertical (1200,1920), one horizontal (1920, 1200) and another horizontal laptop screen (1920, 1080).
What happened in my case: The first object in AllScreens was the vertical monitor on the left. I try to create a device for index 0, 1200 width and 1920 height. Index 0 corresponds to my main monitor though, i.e. the horizontal monitor in the middle. So I'm essentially going out of the screen bounds with my instatiation. The instatiation doesn't throw an exception and at some point later I try to read the front buffer data. Bam, Exception because I'm trying to take a 1200x1920 screenshot of a monitor that's 1920x1200.
Sadly, even after I got this working, the performance was no good. A single frame of all 3 monitors takes about 300 to 500ms. Even with a single monitor, the execution time was something like 100ms. Not good enough for my usecase.
Didn't get the Backbuffer to work either, it just produces black images.
I went back to the GDI method and enhanced it by only updating specific chunks of the bitmap on each Frame() call. You want to capture a 1920x1200 region, which gets cut into 480x300 Rectangles.
I am writing an application in Unity which will be required to capture an image from a camera every frame (at ~60fps), and send the resultant data to another service running locally.
The issue is, I am aware that capturing the rendered data from the camera can cause massive frame rate drops (as explained in this article) when using the GetPixels() method. The article explains that "GetPixels() blocks for ReadPixels() to complete" and "ReadPixels() blocks while flushing the GPU" which is why the GPU and CPU have to sync up, resulting in a lag.
I have produced a sample project with a script attached which simply outputs frames to a file as a PNG to replicate the functionality of the program I wish to create. I have done my best to implement what is described in the article, namely allowing the GPU to render a frame, then wait a few frames before calling GetPixels() so as not to cause the GPU and CPU to forcefully sync up. However, I really haven't made any progress with it. The project still plays at about 10-15fps.
How can I achieve a realtime capture of 60 frames per second in Unity?
using System;
using System.Collections;
using System.IO;
using UnityEngine;
namespace Assets
{
public class MyClass: MonoBehaviour
{
private const float reportInterval = 0.5f;
private int screenshotCount = 0;
private const float maxElapsedSecond = 20;
private string screenshotsDirectory = "UnityHeadlessRenderingScreenshots";
public Camera camOV;
public RenderTexture currentRT;
private int frameCount = 0;
private Texture2D resultantImage;
public void Start()
{
camOV.forceIntoRenderTexture = true;
if (Directory.Exists(screenshotsDirectory))
{
Directory.Delete(screenshotsDirectory, true);
}
if (!Application.isEditor)
{
Directory.CreateDirectory(screenshotsDirectory);
camOV.targetTexture = currentRT;
}
}
// Update is called once per frame
public void Update()
{
//Taking Screenshots
frameCount += 1;
if (frameCount == 1)
{
TakeScreenShot();
}
else if (frameCount == 3)
{
ReadPixelsOut("SS_"+screenshotCount+".png");
}
if (frameCount >= 3)
{
frameCount = 0;
}
}
public void TakeScreenShot()
{
screenshotCount += 1;
RenderTexture.active = camOV.targetTexture;
camOV.Render();
resultantImage = new Texture2D(camOV.targetTexture.width, camOV.targetTexture.height, TextureFormat.RGB24, false);
resultantImage.ReadPixels(new Rect(0, 0, camOV.targetTexture.width, camOV.targetTexture.height), 0, 0);
resultantImage.Apply();
}
private void ReadPixelsOut(string filename)
{
if (resultantImage != null)
{
resultantImage.GetPixels();
RenderTexture.active = currentRT;
byte[] bytes = resultantImage.EncodeToPNG();
// save on disk
var path = screenshotsDirectory + "/" + filename;
File.WriteAllBytes(path, bytes);
Destroy(resultantImage);
}
}
}
}
The article implies that it is possible, but I haven't managed to get it to work.
Many thanks in advance for your help.
I am not sure if OP still need the answer. But in case someone in the future getting the same problem, Let me share what i found.
https://github.com/unity3d-jp/FrameCapturer
This is a plugin designed for rendering animation video in Unity editor. But it can also work in standalone. In my case, i take some part of it, and make my app stream Motion Jpeg. I did it with 30fps, never tried 60fps
I really cannot get my head around this, so I hope that someone can give me a little hand ^^
I'm trying to detect motion in C# via my webcam.
So far I've tried multiple libraries (AForge Lib), but failed because I did not understand how to use it.
At first I just wanted to compare the pixels from the current frame with the last one, but that turned out to work like utter s**t :I
Right now, my webcam runs an event "webcam_ImageCaptured" every time the picture from the webcam, which is like 5-10 fps.
But I cannot find a simple way to get the difference from the two images, or at least something that works decent.
Has anybody got an idea on how I could do this rather simple (as possible as that is)?
Getting motion detection to work using the libraries you mention is trivial. Following is an AForge (version 2.2.4) example. It works on a video file but you can easily adapt it to the webcam event.
Johannes' is right but I think playing around with these libraries eases the way to understanding basic image processing.
My application processes 720p video at 120FPS on a very fast machine with SSDs and around 50FPS on my development laptop.
public static void Main()
{
float motionLevel = 0F;
System.Drawing.Bitmap bitmap = null;
AForge.Vision.Motion.MotionDetector motionDetector = null;
AForge.Video.FFMPEG.VideoFileReader reader = new AForge.Video.FFMPEG.VideoFileReader();
motionDetector = GetDefaultMotionDetector();
reader.Open(#"C:\Temp.wmv");
while (true)
{
bitmap = reader.ReadVideoFrame();
if (bitmap == null) break;
// motionLevel will indicate the amount of motion as a percentage.
motionLevel = motionDetector.ProcessFrame(bitmap);
// You can also access the detected motion blobs as follows:
// ((AForge.Vision.Motion.BlobCountingObjectsProcessing) motionDetector.Processor).ObjectRectangles [i]...
}
reader.Close();
}
// Play around with this function to tweak results.
public static AForge.Vision.Motion.MotionDetector GetDefaultMotionDetector ()
{
AForge.Vision.Motion.IMotionDetector detector = null;
AForge.Vision.Motion.IMotionProcessing processor = null;
AForge.Vision.Motion.MotionDetector motionDetector = null;
//detector = new AForge.Vision.Motion.TwoFramesDifferenceDetector()
//{
// DifferenceThreshold = 15,
// SuppressNoise = true
//};
//detector = new AForge.Vision.Motion.CustomFrameDifferenceDetector()
//{
// DifferenceThreshold = 15,
// KeepObjectsEdges = true,
// SuppressNoise = true
//};
detector = new AForge.Vision.Motion.SimpleBackgroundModelingDetector()
{
DifferenceThreshold = 10,
FramesPerBackgroundUpdate = 10,
KeepObjectsEdges = true,
MillisecondsPerBackgroundUpdate = 0,
SuppressNoise = true
};
//processor = new AForge.Vision.Motion.GridMotionAreaProcessing()
//{
// HighlightColor = System.Drawing.Color.Red,
// HighlightMotionGrid = true,
// GridWidth = 100,
// GridHeight = 100,
// MotionAmountToHighlight = 100F
//};
processor = new AForge.Vision.Motion.BlobCountingObjectsProcessing()
{
HighlightColor = System.Drawing.Color.Red,
HighlightMotionRegions = true,
MinObjectsHeight = 10,
MinObjectsWidth = 10
};
motionDetector = new AForge.Vision.Motion.MotionDetector(detector, processor);
return (motionDetector);
}
Motion detection is a complex matter, and it requires a lot of computing power.
Try to limit what you want to detect first. With increasing complexity: Do your want to detect whether there is motion or not? Do you want to detect how much motion? Do you want to detect which areas of the image are actually moving?
I assume you just want to know when something changed:
subtract adjacent frames from each other
calc the sum of all squares of all pixel differences
divide by number of pixels
watch the number for your webcam stream. It will have a certain ground noise and will significantly go up when something moves.
try to limit to a certain color channel only, this may improve things