We Keep Coding

Possible Memory leak in FFMPEGAutogen Example

I'm using the https://github.com/Ruslan-B/FFmpeg.AutoGen library to connect to an rtsp camera stream and save the frames as png images.
Testing with the FFMPEG.Autogen's example project present in the GitHub repository with no changes I've noticed that the memory usage seems to constantly grow in infinitely. I've made sure to dispose of all bitmaps, pointers etc after use but am unable to pinpoint the source of the issue.
It seems to be coming from their VideoStreamDecoder.TryDecodeNextFrame method as shown below:
public bool TryDecodeNextFrame(out AVFrame frame)
{
ffmpeg.av_frame_unref(_pFrame);
ffmpeg.av_frame_unref(_receivedFrame);
int error;
do
{
try
{
do
{
error = ffmpeg.av_read_frame(_pFormatContext, _pPacket);
if (error == ffmpeg.AVERROR_EOF)
{
frame = *_pFrame;
return false;
}
else if(error < 0)
{
}
error.ThrowExceptionIfError();
} while (_pPacket->stream_index != _streamIndex);
ffmpeg.avcodec_send_packet(_pCodecContext, _pPacket).ThrowExceptionIfError();
}
finally
{
ffmpeg.av_packet_unref(_pPacket);
}
error = ffmpeg.avcodec_receive_frame(_pCodecContext, _pFrame);
} while (error == ffmpeg.AVERROR(ffmpeg.EAGAIN));
error.ThrowExceptionIfError();
if (_pCodecContext->hw_device_ctx != null)
{
ffmpeg.av_hwframe_transfer_data(_receivedFrame, _pFrame, 0).ThrowExceptionIfError();
frame = *_receivedFrame;
}
else
{
frame = *_pFrame;
}
return true;
}

Take several photos with PhotoCapture Unity C#

I got a script that should be able to take more than one photo. I am using PhotoCapture and get an error making it unable to capture a second photo. I get the error "Value cannot be null" on the photoCaptureObject.StartPhotoModeAsync(cameraParameters, result => row, but I don't understand why this is.
I have commented away the photoCaptureObject = null; row so that the photoCaptureObject should not be null. The row if (photoCaptureObject == null) return; also proofs that the photoCaptureObjectis not null.
PhotoCapture photoCaptureObject = null;
Texture2D targetTexture = null;
public string path = "";
CameraParameters cameraParameters = new CameraParameters();
private void Awake()
{
var cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
targetTexture = new Texture2D(cameraResolution.width, cameraResolution.height);
// Create a PhotoCapture object
PhotoCapture.CreateAsync(false, captureObject =>
{
photoCaptureObject = captureObject;
cameraParameters.hologramOpacity = 0.0f;
cameraParameters.cameraResolutionWidth = cameraResolution.width;
cameraParameters.cameraResolutionHeight = cameraResolution.height;
cameraParameters.pixelFormat = CapturePixelFormat.BGRA32;
});
}
private void Update()
{
// if not initialized yet don't take input
if (photoCaptureObject == null) return;
if (Input.GetKeyDown(KeyCode.K) || Input.GetKeyDown("k"))
{
Debug.Log("k was pressed");
VuforiaBehaviour.Instance.gameObject.SetActive(false);
// Activate the camera
photoCaptureObject.StartPhotoModeAsync(cameraParameters, result =>
{
if (result.success)
{
// Take a picture
photoCaptureObject.TakePhotoAsync(OnCapturedPhotoToMemory);
}
else
{
Debug.LogError("Couldn't start photo mode!", this);
}
});
}
}
There is some code here in between that change the photo taken and so on, but I don't think that that code is part of the problem.
private void OnStoppedPhotoMode(PhotoCapture.PhotoCaptureResult result)
{
// Shutdown the photo capture resource
VuforiaBehaviour.Instance.gameObject.SetActive(true);
photoCaptureObject.Dispose();
//photoCaptureObject = null;
Debug.Log("Photomode stopped");
}
So what else could be null? Is it the StartPhotoModeAsync somehow? How can I fix this issue?
Thanks!

Okay so I understand now thanks to Henriks comment.
Unity specifically says this about StartPhotoModeAsync:
Only one PhotoCapture instance can start the photo mode at any given
time
I have focused more on the sentence after saying that one should always use PhotoCapture.StopPhotoModeAsyncbecause having PhotoCaptureMode on takes more power so I never thought about that the instance wouldn't start again after stopping it.
Now I only have TakePhotoAsyncin the update on key press and do not stop the PhotoMode due to that the app that I am making should always be able to capture photos.

Understand Kinect Facetracker.track() method

I'm Using Kinect v1 with SDK 1.8. I need to pass my data to faceTracker.Track() method.
But in constructor of track() is asking Skeleton parameter at the end.
But cannot take directly skeleton data to skeleton object. It provides as an data array.
How can I pass data to this faceTracker.Track() method as the track method requested?
The following is the way I tried.
using (SkeletonFrame skeletonFrame = allFramesReadyEventArgs.OpenSkeletonFrame())
{
if (skeletonFrame != null && this.skeletonData != null) // check that a frame is available
{
skeletonData = new Skeleton [newSensor.SkeletonStream.FrameSkeletonArrayLength];
skeletonFrame.CopySkeletonDataTo(this.skeletonData); // get the skeletal information in this frame
this.skeletonData = new Skeleton[this.newSensor.SkeletonStream.FrameSkeletonArrayLength];
}
}
track method
if (faceTracker == null)
{
try
{
faceTracker = new FaceTracker(sensorChooser.Kinect);
}
catch (InvalidOperationException)
{
//write something
}
if (faceTracker != null)
{
FaceTrackFrame frame = faceTracker.Track(
sensorChooser.Kinect.ColorStream.Format,
colorImageData,
sensorChooser.Kinect.DepthStream.Format,
colorPixels,
skeletonData);
}
}
i defined Skeleton [] skeletonData ;as this
and follwing is the track method given in Kinect SDK
public FaceTrackFrame Track(
ColorImageFormat colorImageFormat,
byte[] colorImage,
DepthImageFormat depthImageFormat,
short[] depthImage,
Skeleton skeletonOfInterest)
{
return this.Track(colorImageFormat, colorImage, depthImageFormat, depthImage, skeletonOfInterest, Rect.Empty);
}

foreach (Skeleton skeleton in this.skeletonData)
{
if (skeleton.TrackingState == SkeletonTrackingState.Tracked
|| skeleton.TrackingState == SkeletonTrackingState.PositionOnly)
{
// We want keep a record of any skeleton, tracked or untracked.
if (!this.trackedSkeletons.ContainsKey(skeleton.TrackingId))
{
this.trackedSkeletons.Add(skeleton.TrackingId, new SkeletonFaceTracker());
}
// Give each tracker the upated frame.
SkeletonFaceTracker skeletonFaceTracker;
if (this.trackedSkeletons.TryGetValue(skeleton.TrackingId, out skeletonFaceTracker))
{
skeletonFaceTracker.OnFrameReady(this.Kinect, colorImageFormat, colorImage, depthImageFormat, depthImage, skeleton);
skeletonFaceTracker.LastTrackedFrame = skeletonFrame.FrameNumber;
}
}
}
I solved it as above with the help of kinect toolkit 1.8 the project FaceTrackingBasics-WPF application.

Compare two joints position with Kinect

I want to make a Kinect program thanks to, when your left hand X position cross your right shoulder X position, it plays the following slide on PowerPoint. I've already done the program part which draws the skeleton stream, and it's work correctly. For the second part, I've tried this :
private void NextSlide(Skeleton skeleton)
{
Joint leftHand = skeleton.Joints[JointType.HandLeft];
Joint spine = skeleton.Joints[JointType.Spine];
Joint shoulderRight = skeleton.Joints[JointType.ShoulderRight];
bool check = false;
if (leftHand.Position.X == shoulderRight.Position.X && check == false)
{
check = true;
System.Windows.Forms.SendKeys.Send("{RIGHT}");
}
if (leftHand.Position.X == spine.Position.X && check == true)
{
check = false
}
}
Can anyone explain me what is the problem in my code ?
Thanks,
Edit :
I've also tried this :
private void NextSlide(Skeleton skeleton)
{
Joint leftHand = skeleton.Joints[JointType.HandLeft];
Joint spine = skeleton.Joints[JointType.Spine];
Joint shoulderRight = skeleton.Joints[JointType.ShoulderRight];
double lefthandposition = (int)leftHand.Position.X;
double shoulderrightposition = (int)shoulderRight.Position.X;
double spineposition = (int)spine.Position.X;
bool turn = false;
double right = lefthandposition - shoulderrightposition;
bool finish = false;
double ok = lefthandposition - spineposition;
if (right < 0)
{
turn = true;
}
if (ok > 0)
{
finish = true;
}
bool check = false;
if (turn == true && check == false && finish == false)
{
System.Windows.Forms.SendKeys.Send("{RIGHT}");
check = true;
turn = false;
}
if (finish == true && check == true && turn == false)
{
check = false;
finish = false;
}
}
But it doesn't work too :/

The trouble you are running into is that you using == as a check. You will rarely, if ever, run into a situation where two joints are exactly the same value. You need to be using >= or <= for your checks, depending on your needs.
For example, check the JointType.LeftHand >= JointType.RightShoulder, and that JointType.LeftHand <= JointType.Spine, for your position checks.
Another solution could be to use an existing gesture library to accomplish what you want. Here are two of them:
http://kinecttoolbox.codeplex.com/
https://github.com/EvilClosetMonkey/Fizbin.Kinect.Gestures
The Fizbin library's GitHub page explains how to set up and execute on existing gestures: Executing on Gestures. Included in the existing gesture set is a "Swipe Right" and a "Swipe Left" gesture, which execute similar to what you are wanting to do.
The GitHub pages goes into more detail, but a quick rundown would start with setting up the gesture library and a callback:
gestureController = new GestureController();
gestureController.GestureRecognized += OnGestureRecognized;
You'd then initialize the gesture(s) you want to use:
IRelativeGestureSegment[] swipeleftSegments = new IRelativeGestureSegment[3];
swipeleftSegments[0] = new SwipeLeftSegment1();
swipeleftSegments[1] = new SwipeLeftSegment2();
swipeleftSegments[2] = new SwipeLeftSegment3();
gestureController.AddGesture("SwipeLeft", swipeleftSegments);
You can modify the existing gesture components or write your own. The several examples will demonstrate how you can compare joint positions. For example, the first segment in the "SwipeLeft" gesture follows:
public class SwipeLeftSegment1 : IRelativeGestureSegment
{
/// <summary>
/// Checks the gesture.
/// </summary>
/// <param name="skeleton">The skeleton.</param>
/// <returns>GesturePartResult based on if the gesture part has been completed</returns>
public GesturePartResult CheckGesture(Skeleton skeleton)
{
// right hand in front of right shoulder
if (skeleton.Joints[JointType.HandRight].Position.Z < skeleton.Joints[JointType.ElbowRight].Position.Z && skeleton.Joints[JointType.HandLeft].Position.Y < skeleton.Joints[JointType.ShoulderCenter].Position.Y)
{
// right hand below shoulder height but above hip height
if (skeleton.Joints[JointType.HandRight].Position.Y < skeleton.Joints[JointType.Head].Position.Y && skeleton.Joints[JointType.HandRight].Position.Y > skeleton.Joints[JointType.HipCenter].Position.Y)
{
// right hand right of right shoulder
if (skeleton.Joints[JointType.HandRight].Position.X > skeleton.Joints[JointType.ShoulderRight].Position.X)
{
return GesturePartResult.Succeed;
}
return GesturePartResult.Pausing;
}
return GesturePartResult.Fail;
}
return GesturePartResult.Fail;
}
}
Notice it goes through a series of joint checks and returns one of three results:
Succeed: The gesture has completed in full (fired when all the 'if' statements succeed)
Pausing: The gesture is partially complete (fired when you want to indicate the gesture is "in motion" but not complete)
Fail: The gesture is not being executed at all
When you have the gesture you want initialized you just need to send the 'skeleton' data to the gesture library:
private void OnSkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
using (SkeletonFrame frame = e.OpenSkeletonFrame())
{
if (frame == null)
return;
// resize the skeletons array if needed
if (skeletons.Length != frame.SkeletonArrayLength)
skeletons = new Skeleton[frame.SkeletonArrayLength];
// get the skeleton data
frame.CopySkeletonDataTo(skeletons);
foreach (var skeleton in skeletons)
{
// skip the skeleton if it is not being tracked
if (skeleton.TrackingState != SkeletonTrackingState.Tracked)
continue;
// update the gesture controller
gestureController.UpdateAllGestures(skeleton);
}
}
}
Finally, execute on the gesture:
private void OnGestureRecognized(object sender, GestureEventArgs e)
{
switch (e.GestureName)
{
case "SwipeLeft":
// do what you want to do
break;
default:
break;
}
}

Kinect's SkeletonStream to feedback user undetected C#

I am trying to get my Kinect's SkeletonStream to feedback a data that tells me that nobody is detected. I can get feed if my skeleton is detected but I am unable to get any notification if there is nobody there. Is there a way to get the kinect to tell me if no skeletons are picked up?
private void kinect_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
using (SkeletonFrame frame = e.OpenSkeletonFrame())
{
if (frame == null)
{
return;
}
frame.GetSkeletons(ref allSkeletons);
if (allSkeletons.All(s => s.TrackingState == SkeletonTrackingState.NotTracked))
return;
foreach (var skeleton in allSkeletons)
{
if (skeleton.TrackingState != SkeletonTrackingState.Tracked)
{
continue;
}
if (skeleton.TrackingState == SkeletonTrackingState.NotTracked)
{
}
foreach (Joint joint in skeleton.Joints)
{
if (joint.TrackingState != JointTrackingState.Tracked)
continue;
if (joint.JointType == JointType.HipCenter)
{
hipCenter = joint.Position;
AdvanceFunction();
}
}
sdm.Draw(frame.GetSkeletons(), false);
}
}
}

There is nothing that will simply tell you that no skeletons are currently being tracked. You will need to look through the skeleton frames to determine if there are any users.
Your foreach loop steps through all the skeletons...
foreach (var skeleton in allSkeletons)
{
if (skeleton.TrackingState != SkeletonTrackingState.Tracked)
{
continue;
}
....
}
In the first if statement -- if the current skeleton is not being actively tracked then the loop will move on to the next skeleton. You'll want to add a flag if any skeletons have been found. For example, you could do something like...
bool hasActivePlayer = false;
foreach (var skeleton in allSkeletons)
{
if (skeleton.TrackingState != SkeletonTrackingState.Tracked)
{
continue;
}
hasActivePlayer = true;
....
}
if (hasActivePlayer == false)
{
// you aren't tracking anyone, deal with it
}
You may also be interested in checking SkeletonTrackingState.PositionOnly. In this case the Kinect knows someone is there, but it is not actively tracking their skeleton. You can update the ckeck in the foreach loop if you want to look for it as well.