We Keep Coding

GstSharp raw video pausing after gst_pad_push() = not-linked

I've been working on a small Microsoft Teams extension that takes an incoming Audio/Video feed from each participant and records the data into an mp4 file for each participant. In all the MS Teams documentation they focus on getting access to the raw data itself and gloss over any kind of persistence of it into a usable video container. The video comes in as an NV12 encoded byte array and the audio is wav format. I've been trying to use Gstreamer to take the raw video data, push it to a videoconvert then on to a filesink to try and save the data as an h264 encoded mp4. I've pulled together a mix and match of code examples and think I'm close but missing something in the process. My pipeline creates successfully but when pushing buffers into the appsrc I get the following in my console:
pausing after gst_pad_push() = not-linked
error: Internal data stream error.
error: streaming stopped, reason not-linked (-1)
The code I use to create my pipeline, pads and sinks etc is as follows:
private Gst.App.AppSrc CreatePipeline(Guid identifier, string identity, int width, int height, string directory)
{
Directory.CreateDirectory($"{directory}\\temporary\\{identifier}");
var pipeline = new Gst.Pipeline($"pipeline_{identity}");
var VideoQueue = Gst.ElementFactory.Make("queue", $"video_queue_{identity}");
var VideoConvert = Gst.ElementFactory.Make("videoconvert", $"video_convert_{identity}");
var VideoSink = Gst.ElementFactory.Make("autovideosink", $"video_sink_{identity}");
var AppQueue = Gst.ElementFactory.Make("queue", $"app_queue_{identity}");
var AppSink = new Gst.App.AppSink($"app_sink_{identity}");
var AppSource = new Gst.App.AppSrc($"app_src_{identity}");
var Tee = Gst.ElementFactory.Make("tee", $"tee_{identity}");
var FileSink = Gst.ElementFactory.Make("filesink", $"file_sink_{identity}");
AppSource.Caps = Gst.Caps.FromString($"video/x-raw,format=NV12,width={width},height={height},framerate={this.fixedFps}");
AppSource.IsLive = true;
AppSink.EmitSignals = true;
AppSink.Caps = Gst.Caps.FromString($"video/x-raw,format=NV12,width={width},height={height},framerate={this.fixedFps}");
AppSink.NewSample += NewSample;
Console.WriteLine("Setting Filesink location");
FileSink["location"] = $"{directory}\\temporary\\{identifier}\\{identity}.mp4";
pipeline.Add(AppSource, Tee, VideoQueue, VideoConvert, VideoSink, AppQueue, AppSink, FileSink);
var teeVideoPad = Tee.GetRequestPad("src_%u");
var queueVideoPad = VideoQueue.GetStaticPad("sink");
var teeAppPad = Tee.GetRequestPad("src_%u");
var queueAppPad = AppQueue.GetStaticPad("sink");
if ((teeVideoPad.Link(queueVideoPad) != Gst.PadLinkReturn.Ok) ||
(teeAppPad.Link(queueAppPad) != Gst.PadLinkReturn.Ok))
{
Console.WriteLine("Tee could not be linked.");
throw new Exception("Tee could not be linked.");
}
AppSource.PadAdded += new Gst.PadAddedHandler(this.OnVideoPadAdded);
var bus = pipeline.Bus;
bus.AddSignalWatch();
bus.Connect("message::error", HandleError);
pipeline.SetState(Gst.State.Playing);
return AppSource;
}
And this gets called every time a new participant joins the call. Then each time a new video frame is sent to the call from a participant the following code is executed:
private bool NV12toMP4(byte[] array, string identity, int width, int height, long timestamp, int length)
{
var buffer = new Gst.Buffer(null, (ulong)length, Gst.AllocationParams.Zero)
{
Pts = (ulong)timestamp,
Dts = (ulong)timestamp
};
buffer.Fill(0, array);
var ret = this.participantSources[identity].PushBuffer(buffer);
buffer.Dispose();
if (ret != Gst.FlowReturn.Ok)
{
return false;
}
return true;
}
I was expecting my PadAdded method to get called on the AppSrc when it first detects the type of input but this never gets triggered so I'm not sure where to look next on this.

Unity: CopyTexture to External texture2D

I need to expose a Unity texture/Rendertexture to some native plugin, which requires the "D3D11_RESOURCE_MISC_SHARED" flag on the texture.
textures created by unity doesn't have this flag, so, I created it from the plugin side, and then created a reference texture within unity using CreateExternalTexture, and copied the contents to this native texture using Graphics.CopyTexture.
the 2 textures have the same dimension, same size, same format, and same mipCount(0)
the problem is, when I show it in unity (for debugging purpose), I can see nothing and no error occurs.
btw, if I copy by ReadPixel, an error occures :
ReadPixels called on undefined image 0 (valid values are 0 - -1
if I create the texture using unity api, CopyTexture succeeds and result can be seen. but then, I lose the "D3D11_RESOURCE_MISC_SHARED" flag.
so, maybe the texture I created is not valid?
my code:
D3D11_TEXTURE2D_DESC desc = { 0 };
desc.Width = width;
desc.Height = height;
desc.MipLevels = 0;
desc.ArraySize = 1;
desc.Format = DXGI_FORMAT_R8G8B8A8_UNORM;//这里,格式能不能调整?比如,A8是需要的吗?
desc.SampleDesc.Count = 1;
desc.Usage = D3D11_USAGE_DEFAULT;
desc.BindFlags = D3D11_BIND_SHADER_RESOURCE;//普通资源
//desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ;//应该不需要cpu访问的,不是read也不是write
desc.MiscFlags = D3D11_RESOURCE_MISC_SHARED;//for "OpenSharedHandle" d3d11 api
HRESULT hr = E_FAIL;
if (FAILED(hr = pDevice->CreateTexture2D(&desc, nullptr, &g_unityEquirectTexture)))
{
Log(" Create Shared Texture Failed!");
return NULL;
}
Log("CreateSharedTexture success");
//return g_unityEquirectTexture;
Unity CopyTexture Code:
if (output == null)
{
Debug.Log($"limit = {QualitySettings.masterTextureLimit}");
//output = new Texture2D(equirect.width, equirect.height, TextureFormat.RGBA32,false);//uncomment this line and then copyTexture below succeeds
IntPtr externalTextureData = CGDKInterOp.cgdk_c_CreateExternalTexture(equirectLeft.GetNativeTexturePtr(), equirectLeft.width * 2, equirectLeft.height);
if (externalTextureData != IntPtr.Zero)
{
output = Texture2D.CreateExternalTexture(equirectLeft.width * 2, equirectLeft.height, TextureFormat.RGBA32, false, true, externalTextureData);
}
}
if (output == null)
{
Debug.LogError("create texture from external failed!");
return;
}
//RenderTexture.active = equirect;
//output.ReadPixels(new Rect(0, 0, equirect.width, equirect.height), 0, 0);
//RenderTexture.active = null;
Graphics.CopyTexture(equirect, output);

OK, solved by myself.
the problem is: Miplevel == 0.this causes d3d11 creating a texture with 0B memory allocated!
change Miplevel to 1 solved the problem
note:
In unity inspector, we can see the memory allocated by textures. I found that my texture has 0B memory from the inspector and then, I search using this clue and found the solution

There might be a better solution for this but the last time I needed to copy over texture data from a native side to managed (Unity), I did it through marshalling of the data.
You essentially just need to expose a method in the native plugin to pass you the texture data as an array and then have a method in the C# code to fetch it data (by calling the method), releasing the pointer to native memory when you're done. You can find information about marshalling and interop in Microsoft's documentation (e.g. https://learn.microsoft.com/en-us/dotnet/framework/interop/marshalling-different-types-of-arrays).
If your texture is always guaranteed to be of the same size and format, it's easier - but if you need to know some additional parameters so that you know how the Texture should be represented in managed-land, you can always pass yourself the additional data through the same method.

Guide to use Yolo with gpu c#

I've been trying to find a tutorial or something on how to make yolo c# use gpu instead of cpu, I always find that it says that it works on both cpu and gpu but no one ever says how to use the gpu since it always uses cpu for me. Here's my code with yolo v5 c#. It doesn't really matter for me if it uses yolo v5 just that it uses gpu. Tutorial I found that tutorial but i can't even find the download for Nvidia cuDNN v7.6.3 for CUDA 10.1. It feels very unclear on how to use it with gpu please help me :D
var image = pictureBox1.Image;
var scorer = new YoloScorer<YoloCocoP5Model>("Assets/Weights/yolov5n.onnx");
List<YoloPrediction> predictions = scorer.Predict(image);
var graphics = Graphics.FromImage(image);
foreach (var prediction in predictions) // iterate predictions to draw results
{
using (MemoryStream ms = new MemoryStream())
{
pictureBox1.Image.Save(ms, ImageFormat.Png);
prediction.Label.Color = Color.FromArgb(255, 255, 0, 0);
double score = Math.Round(prediction.Score, 2);
graphics.DrawRectangles(new Pen(prediction.Label.Color, 1),
new[] { prediction.Rectangle });
var (x, y) = (prediction.Rectangle.X - 3, prediction.Rectangle.Y - 23);
graphics.DrawString($"{prediction.Label.Name} ({score})",
new Font("Consolas", 16, GraphicsUnit.Pixel), new SolidBrush(prediction.Label.Color),
new PointF(x, y));
pictureBox1.Image = image;
}
}

Before you are going to use scorer you need an option set.
//https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html
bool initResult = false;
var cudaProviderOptions = new Microsoft.ML.OnnxRuntime.OrtCUDAProviderOptions(); // Dispose this finally
var providerOptionsDict = new Dictionary<string, string>();
providerOptionsDict["device_id"] = "0";
providerOptionsDict["gpu_mem_limit"] = "2147483648";
providerOptionsDict["arena_extend_strategy"] = "kSameAsRequested";
/*
cudnn_conv_algo_search
The type of search done for cuDNN convolution algorithms.
Value Description
EXHAUSTIVE (0) expensive exhaustive benchmarking using cudnnFindConvolutionForwardAlgorithmEx
HEURISTIC (1) lightweight heuristic based search using cudnnGetConvolutionForwardAlgorithm_v7
DEFAULT (2) default algorithm using CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_PRECOMP_GEMM
Default value: EXHAUSTIVE
*/
providerOptionsDict["cudnn_conv_algo_search"] = "DEFAULT";
/*
do_copy_in_default_stream
Whether to do copies in the default stream or use separate streams. The recommended setting is true. If false, there are race conditions and possibly better performance.
Default value: true
*/
providerOptionsDict["do_copy_in_default_stream"] = "1";
/*
cudnn_conv_use_max_workspace
Check tuning performance for convolution heavy models for details on what this flag does. This flag is only supported from the V2 version of the provider options struct when used using the C API. The V2 provider options struct can be created using this and updated using this. Please take a look at the sample below for an example.
Default value: 0
*/
providerOptionsDict["cudnn_conv_use_max_workspace"] = "1";
/*
cudnn_conv1d_pad_to_nc1d
Check convolution input padding in the CUDA EP for details on what this flag does. This flag is only supported from the V2 version of the provider options struct when used using the C API. The V2 provider options struct can be created using this and updated using this. Please take a look at the sample below for an example.
Default value: 0
*/
providerOptionsDict["cudnn_conv1d_pad_to_nc1d"] = "1";
cudaProviderOptions.UpdateOptions(providerOptionsDict);
options = SessionOptions.MakeSessionOptionWithCudaProvider(cudaProviderOptions); // Dispose this finally
if (options != null)
{
// check yolo model file is accesible
if (File.Exists(yoloModelFile))
{
scorer = new YoloScorer<YoloCocoP5Model>(yoloModelFile, options);
initResult = true;
}
else
{
DebugMessage("Yolo model ONNX file (" + yoloModelFile + ") is missing!\r\n", 2);
}
}
else
DebugMessage("Yolo instance initializing error! Session options are empty!\r\n", 2);
}

How to use function D3DXSaveSurfaceToFile() in C#

I use directx 9 to build model.
I want to save it to bmp file. I found the function D3DXSaveSurfaceToFile()
But I do not know how to use it in C#.
How can I use it?

Unfortunately, there is no such function in c#.
Try the following code instead:
try
{
// initialize D3D device
PresentParameters presentParams = new PresentParameters();
presentParams.Windowed = true;
presentParams.SwapEffect = SwapEffect.Discard;
Device myDevice = new
Device(0,DeviceType.Hardware,this,CreateFlags.SoftwareVertexProcessing,presentParams);
// create a surface the size of screen,
// format had to be A8R8G8B8, as the GetFrontBufferData returns
// only memory pool types allowed are Pool.Scratch or Pool.System memory
Surface mySurface =
myDevice.CreateOffscreenPlainSurface(SystemInformation.PrimaryMonitorSize.Width,SystemInformation.PrimaryMonitorSize.Height,Format.A8R8G8B8,Pool.SystemMemory);
//Get the front buffer.
myDevice.GetFrontBufferData(0,mySurface);
//saves surface to file
SurfaceLoader.Save("surface.bmp",ImageFileFormat.Bmp,mySurface);
}
catch
{
//whatever
}

Taking input from a joystick with C# .NET

I searched around on Google for this, but the only things I came up with were outdated and did not work.
Does anyone have any information on how to get joystick data using C# .NET?

Since this was the top hit I got on google while researching joystick / gamepad input in C#, I thought I should post a response for others to see.
The easiest way I found was to use SharpDX and DirectInput. You can install it via NuGet (SharpDX.DirectInput)
After that, it's simply a matter of calling a few methods:
Sample code from SharpDX
static void Main()
{
// Initialize DirectInput
var directInput = new DirectInput();
// Find a Joystick Guid
var joystickGuid = Guid.Empty;
foreach (var deviceInstance in directInput.GetDevices(DeviceType.Gamepad,
DeviceEnumerationFlags.AllDevices))
joystickGuid = deviceInstance.InstanceGuid;
// If Gamepad not found, look for a Joystick
if (joystickGuid == Guid.Empty)
foreach (var deviceInstance in directInput.GetDevices(DeviceType.Joystick,
DeviceEnumerationFlags.AllDevices))
joystickGuid = deviceInstance.InstanceGuid;
// If Joystick not found, throws an error
if (joystickGuid == Guid.Empty)
{
Console.WriteLine("No joystick/Gamepad found.");
Console.ReadKey();
Environment.Exit(1);
}
// Instantiate the joystick
var joystick = new Joystick(directInput, joystickGuid);
Console.WriteLine("Found Joystick/Gamepad with GUID: {0}", joystickGuid);
// Query all suported ForceFeedback effects
var allEffects = joystick.GetEffects();
foreach (var effectInfo in allEffects)
Console.WriteLine("Effect available {0}", effectInfo.Name);
// Set BufferSize in order to use buffered data.
joystick.Properties.BufferSize = 128;
// Acquire the joystick
joystick.Acquire();
// Poll events from joystick
while (true)
{
joystick.Poll();
var datas = joystick.GetBufferedData();
foreach (var state in datas)
Console.WriteLine(state);
}
}
I hope this helps.
I even got this to work with a DualShock3 and the MotioninJoy drivers.

One: use SlimDX.
Two: it looks something like this (where GamepadDevice is my own wrapper, and the code is slimmed down to just the relevant parts).
Find the joystick / pad GUIDs:
public virtual IList<GamepadDevice> Available()
{
IList<GamepadDevice> result = new List<GamepadDevice>();
DirectInput dinput = new DirectInput();
foreach (DeviceInstance di in dinput.GetDevices(DeviceClass.GameController, DeviceEnumerationFlags.AttachedOnly))
{
GamepadDevice dev = new GamepadDevice();
dev.Guid = di.InstanceGuid;
dev.Name = di.InstanceName;
result.Add(dev);
}
return result;
}
Once the user has selected from the list, acquire the gamepad:
private void acquire(System.Windows.Forms.Form parent)
{
DirectInput dinput = new DirectInput();
pad = new Joystick(dinput, this.Device.Guid);
foreach (DeviceObjectInstance doi in pad.GetObjects(ObjectDeviceType.Axis))
{
pad.GetObjectPropertiesById((int)doi.ObjectType).SetRange(-5000, 5000);
}
pad.Properties.AxisMode = DeviceAxisMode.Absolute;
pad.SetCooperativeLevel(parent, (CooperativeLevel.Nonexclusive | CooperativeLevel.Background));
pad.Acquire();
}
Polling the pad looks like this:
JoystickState state = new JoystickState();
if (pad.Poll().IsFailure)
{
result.Disconnect = true;
return result;
}
if (pad.GetCurrentState(ref state).IsFailure)
{
result.Disconnect = true;
return result;
}
result.X = state.X / 5000.0f;
result.Y = state.Y / 5000.0f;
int ispressed = 0;
bool[] buttons = state.GetButtons();

The bad news is that Microsoft seems to stop supporting their NET libraries for DirectX and focus on XNA instead. I don't work in GameDev so I don't need to use XNA but you may try it if you developing computer games. The good news is that there are other approaches. One is SlimDX the new framework to help you to wok with DirectX from C#. The other way is to directly add references of "Microsoft.DirectX.dll" and "Microsoft.DirectX.DirectInput.dll" to your project. you can find them "..\Windows\Microsoft.NET\DirectX for Managed Code\". if you you are going to use last approach here is a link to codeproject where you can read how to work with a joystick.
EDIT:
If your application is based on NET version newer then 2.0 the application may hang on. To fix this problem change config file and add this:
<startup useLegacyV2RuntimeActivationPolicy="true">

Google led me here and while not a requirement of this question, OpenTK is a good option for Windows and Linux (under mono) support.
From the OpenTK docs, this code works on Raspberry Pi + Raspbian + Mono 3.12.0:
for (int i = 0; i < 4; i++)
{
var state = Joystick.GetState(i);
if (state.IsConnected)
{
float x = state.GetAxis(JoystickAxis.Axis0);
float y = state.GetAxis(JoystickAxis.Axis1);
// Print the current state of the joystick
Console.WriteLine(state);
}
}

This question is old, but it seems to be active even to this day, so I'm posting anyway.
If you need to get input from XInput only, take a look at XInputium. This is a .NET library that is specialized in XInput controllers. It is pretty straightforward, and has many code samples you can look at.