We Keep Coding

Unity: CopyTexture to External texture2D

I need to expose a Unity texture/Rendertexture to some native plugin, which requires the "D3D11_RESOURCE_MISC_SHARED" flag on the texture.
textures created by unity doesn't have this flag, so, I created it from the plugin side, and then created a reference texture within unity using CreateExternalTexture, and copied the contents to this native texture using Graphics.CopyTexture.
the 2 textures have the same dimension, same size, same format, and same mipCount(0)
the problem is, when I show it in unity (for debugging purpose), I can see nothing and no error occurs.
btw, if I copy by ReadPixel, an error occures :
ReadPixels called on undefined image 0 (valid values are 0 - -1
if I create the texture using unity api, CopyTexture succeeds and result can be seen. but then, I lose the "D3D11_RESOURCE_MISC_SHARED" flag.
so, maybe the texture I created is not valid?
my code:
D3D11_TEXTURE2D_DESC desc = { 0 };
desc.Width = width;
desc.Height = height;
desc.MipLevels = 0;
desc.ArraySize = 1;
desc.Format = DXGI_FORMAT_R8G8B8A8_UNORM;//这里,格式能不能调整?比如,A8是需要的吗?
desc.SampleDesc.Count = 1;
desc.Usage = D3D11_USAGE_DEFAULT;
desc.BindFlags = D3D11_BIND_SHADER_RESOURCE;//普通资源
//desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ;//应该不需要cpu访问的,不是read也不是write
desc.MiscFlags = D3D11_RESOURCE_MISC_SHARED;//for "OpenSharedHandle" d3d11 api
HRESULT hr = E_FAIL;
if (FAILED(hr = pDevice->CreateTexture2D(&desc, nullptr, &g_unityEquirectTexture)))
{
Log(" Create Shared Texture Failed!");
return NULL;
}
Log("CreateSharedTexture success");
//return g_unityEquirectTexture;
Unity CopyTexture Code:
if (output == null)
{
Debug.Log($"limit = {QualitySettings.masterTextureLimit}");
//output = new Texture2D(equirect.width, equirect.height, TextureFormat.RGBA32,false);//uncomment this line and then copyTexture below succeeds
IntPtr externalTextureData = CGDKInterOp.cgdk_c_CreateExternalTexture(equirectLeft.GetNativeTexturePtr(), equirectLeft.width * 2, equirectLeft.height);
if (externalTextureData != IntPtr.Zero)
{
output = Texture2D.CreateExternalTexture(equirectLeft.width * 2, equirectLeft.height, TextureFormat.RGBA32, false, true, externalTextureData);
}
}
if (output == null)
{
Debug.LogError("create texture from external failed!");
return;
}
//RenderTexture.active = equirect;
//output.ReadPixels(new Rect(0, 0, equirect.width, equirect.height), 0, 0);
//RenderTexture.active = null;
Graphics.CopyTexture(equirect, output);

OK, solved by myself.
the problem is: Miplevel == 0.this causes d3d11 creating a texture with 0B memory allocated!
change Miplevel to 1 solved the problem
note:
In unity inspector, we can see the memory allocated by textures. I found that my texture has 0B memory from the inspector and then, I search using this clue and found the solution

There might be a better solution for this but the last time I needed to copy over texture data from a native side to managed (Unity), I did it through marshalling of the data.
You essentially just need to expose a method in the native plugin to pass you the texture data as an array and then have a method in the C# code to fetch it data (by calling the method), releasing the pointer to native memory when you're done. You can find information about marshalling and interop in Microsoft's documentation (e.g. https://learn.microsoft.com/en-us/dotnet/framework/interop/marshalling-different-types-of-arrays).
If your texture is always guaranteed to be of the same size and format, it's easier - but if you need to know some additional parameters so that you know how the Texture should be represented in managed-land, you can always pass yourself the additional data through the same method.

Scalar instead of vector type for bitmap input with ml.net and TensorFlow model

I started to get familiar with ml.Net, especially in context of image classification with a pretrained TensorFlow model.
The existing example (https://github.com/dotnet/machinelearning-samples/tree/master/samples/csharp/getting-started/DeepLearning_ImageClassification_TensorFlow) for that task is working fine. I also managed to adapt it to use an image from memory instead of loading it in pipeline.
The next step is to consume an internal TensorFlow model and that's where the question begins.
The model can have several images as input instead of one and I don't get the input data structure running.
The model request as input type:
Vector<Single, 275, 384, 3>
The description for the input is:
<Number of images, image height, image width, image channels (RGB)>.
In my data structure I tried to use an array of Bitmaps (instead of single Bitmap like in modified example mentioned above). With mapping for input column and definition of image size.
public class ImageData
{
[ColumnName("serving_default_input_4"), ImageType(275, 384)]
public Bitmap[] Input { get; set; }
}
public class ImagePrediction : ImageData
{
[ColumnName("StatefulPartitionedCall")]
public float[] Scores { get; set; }
}
With this I get an exception when I try to create empty data for fitting (in line: var data = mlContext.Data.LoadFromEnumerable(new List());) that a scalar is expected as data type.
//Pipeline
IEstimator<ITransformer> pipeline = mlContext.Transforms.ResizeImages(outputColumnName: "serving_default_input_4", imageWidth: 384, imageHeight: 275, inputColumnName: "serving_default_input_4")
.Append(mlContext.Transforms.ExtractPixels("serving_default_input_4", "serving_default_input_4"))
.Append(mlContext.Model.LoadTensorFlowModel(_testTensorFlowModel)
.ScoreTensorFlowModel(outputColumnNames: new[] { "StatefulPartitionedCall" }, inputColumnNames: new[] { "serving_default_input_4" }, addBatchDimensionInput: true));
//Fitting
var data = mlContext.Data.LoadFromEnumerable(new List<ImagePrediction>()); // This line crashes
ITransformer model = pipeline.Fit(data);
The Exception I receive is:
System.ArgumentOutOfRangeException
HResult=0x80131502
Message = Column 'serving_default_input_4' is supposed to be scalar, but type of associated field 'Input' is vector
Source= Microsoft.ML.Data
I guess my problem is related to how the image list in ImageData is defined, but I don't see it... My research about how to declare a scalar data type was not really helpful so far.
Maybe somebody has an idea or hint.
Thanks.
Some system information:
Microsoft.ML 1.5.1
Microsoft.ML.ImageAnalytics 1.5.1
Microsoft.ML.TensorFlow 1.5.1
SciSharp.TensorFlow.Redist 2.3.0
Visual Studio 2019
Windows 10 64 bit

Getting an "InvalidCastException" when passing a Rectangle ValueType in c#

I am using the StitchEngine.dll from the Microsofot Image Composite Engine (ICE) to attempt to stitch images together in c#. I am trying to export the resulting image using the following method:
StartExporting(string, System.ValueType, Microsoft.Research.ICE.Stitching.OutputOptions, bool)
The description of this method gives:
public bool StartExporting(string *filename*, System.ValueType *cropRectangle*, float *scale*, Microsoft.Research.ICE.Stitching.OutputOptions *outputOptions*, bool *showCompletion*.
When I attempt to call the method, I am getting an InvalidCastException, stating that the Specified cast is not valid.
I am doing the following:
Microsoft.Research.ICE.Stitching.OutputOptions outOpt = new Microsoft.Research.ICE.Stitching.OutputOptions(Microsoft.Research.ICE.Stitching.ExportFormat.JPEG, 75, false, false, false);
var valTyp = new Rectangle(0, 0, 500, 1280);
StartExporting(myFilename, valTyp, outOpt, false);
Does anyone have any idea how I could correct this?

As Hans Passant has shown me, the correct type is System.Windows.Int32Rect. This requires adding WindowsBase as a reference.

Wrong values passed as parameter to C library using SWIG

Following my three previous posts, I can now pass a managed array of struct to my wrapped method. Here is an extract from the files:
// packer.i
typedef struct {
int width; // input
int height; // input
frame_t view; // output
frame_t dest; // output
} image_t;
CSHARP_ARRAYS(image_t, image_t)
%apply image_t INOUT[] { image_t *images }
int pack(image_t *images, int nb_images, parameters_t params);
Which generates a function with this signature:
// packer_cs.cs
public static int pack(image_t[] images, int nb_images, parameters_t arg2)
Which I call like this:
// Program.cs
var files = Directory.GetFiles("./images");
var images = new image_t[files.Length];
for (var f = 0; f < files.Length; f++)
{
using (var imgInfo = Image.FromFile(files[f]))
{
var imgStruct = new image_t()
{
width = imgInfo.Width,
height = imgInfo.Height,
dest = new frame_t(),
view = new frame_t()
};
images[f] = imgStruct;
}
}
var result = packer_cs.pack(images, images.Length, new parameters_t());
All is well and done, but when I run the pack() method, I have a protected memory access problem (System.AccessViolationException). Thankfully I have access to the source code of the C library, and Visual Studio automagically opens it for debugging and stepping through as soon as I enable unmanaged code debugging.
So, if I add a breakpoint at the start of the pack() function, and I use a watch to check images[x], I can see that the width and height values have nothing to do with what is provided (sometimes it's even 0 or negative). What's going on ? If I inspect my managed array on the C# side, the values are correctly stored and retrieved. Why doesn't C get the right values ? The other parameters (nb_images and params) don't have any problem.
Thank you !

Do the following:
Check that images[f].width and height have the values you expect
If yes, then check the SWIG-generated code to verify that those fields are properly copied.
If you can't spot any problem by looking at the code, you should break on packer_cs.pack and use the debugger to look at the wrapper code that copies the C# array to the C++ array, see what is wrong.
It is probably something in the typemaps that is incorrect. Once you know what that is, you can copy the typemaps code from SWIG source (the csharp_array.i file) to a new typemap in your .i and fix as required.

LUA - Unexpected symbol near '・

I'm trying to load a Lua script into my project at runtime. However, whenever I try to do so, I receive the following error when loading the file:
Items\food.lua:1: unexpected symbol near '・
The file itself is simple enough, but I can't see the cause of this:
config = {
itemtype = 2,
name = "Food",
basevalue = 1,
inventorytexture = 0,
red = 255,
green = 255,
blue = 255,
inventorywidth = 1,
inventoryheight = 1
}
Here is the code that I am using to run this in C#, in case this has anything to do with it:
// In main initialisation
public void ItemSetup()
{
itemTemplates.Add(new ItemTemplate("Items\\food.lua"));
}
// constructor for ItemTemplate class
Lua LuaLoader = new Lua();
public ItemTemplate(string filename)
{
LuaLoader.DoFile(filename); // here is where the error comes up
}
I have tried changing the settings for the file itself, however none of these appear to help. Here are the settings that I have for this file:
Build Action: None
Copy to Output Directory: Copy of newer
Custom Tool: <blank>
Custom Tool Namespace: <blank>
File Name: food.lua
Full Path: [...]\Items\food.lua

Looks like it could be a byte order mark. See: How do I remove the BOM character from my xml file and XML - Data At Root Level is Invalid
I've dealt with it in some XML files before, so I wouldn't be surprised if this is the same issue. As pointed out in some of the linked questions, there are utilities available to remove/avoid them easily as part of your workflow.