We Keep Coding

How does one convert an array to an openCV Mat object in C#?

Context:
We are working on a Machine Learning project using a pre-trained model which was further trained by a contractor using a Python codebase. There are two models for this Machine Learning process: one for identification and another for classification, and each has its own specific requirements for frame size thanks to them both being pre-trained using Keras-Retinanet.
The goal here is to take a grayscale image (stored as a byte array), apply a color palette, then resize it. The initial way in which the model was trained was using Python's OpenCV library with their default interpolation with their cv2.resize() function. As such we need to replicate that behavior in order to get consistent results (as far as I am aware).
Thus far we have been completely unable to load any format of an array into a Mat object, and the error that comes back is obfuscated, so it's very much unclear what the problem is.
Before this is just flagged as duplicate, I have already tried applying the following solutions:
Convert RGB array to Mat (OpenCv)
https://reposhub.com/dotnet/graphics/SciSharp-SharpCV.html
At the moment we are trying to use the OpenCVSharp4 library. As far as I have been able to find, in order to replicate the resize behavior then we need to store the data in a Mat object, or rebuild their interpolation on our own, which seems like an even bigger waste of time.
public static float[,] ConvertToBGR(byte[] grayscale, Color[] palette)
{
// initilize 3 layer array
float[,] bgrImage = new float[grayscale.Length, 3];
// loop through the grayscale image and get the color for each height value in the grayscale image
for (int i = 0; i < grayscale.Length; i++)
{
bgrImage[i, 0] = palette[grayscale[i]].B;
bgrImage[i, 1] = palette[grayscale[i]].G;
bgrImage[i, 2] = palette[grayscale[i]].R;
}
return bgrImage;
}
Mat matrix = new Mat(totalRows, totalColumns, MatType.CV_32FC3, ConvertToBGR(rawImageData, colorPallet));
This is an example of one of the many different attempts to get this to load into a matrix object. We have also tried a number of other array formats:
/*
float[,,]: [3colorLayers, height, width]
float[,]: [3colorLayers, greyscaleLength]
{
{B, G, R},
{B, G, R},
...
}
float[]: [3x greyscaleLength]
{B, G, R, B, G, R, B, G, R,...}
Even went so far as to just try applying the flat grayscale to a matrix which also failed (rawImageData is a byte[]):
Mat matrix = new Mat(totalRows, totalColumns, MatType.CV_8UC1, rawImageData);
*/
How can I convert an array to a matrix? Is there a way to use NumSharp? Is there a better solution?

I did not add the OpenCvSharp4.runtime.win package to the project, so none of the native behavior was actually available. This whole thing works just fine if you properly include both packages, which is literally spelled out in the description for OpenCvSharp4 "OpenCV wrapper for .NET. Since this package includes only core managed libraries, another package of native bindings for your OS is required (OpenCvSharp4.runtime.*)."

How to create a bitmap with 16 bits per pixel on MacOS?

I want to create a bitmap that has 16 bits per pixel, but when I specify any of the pixel formats:
PixelFormat.Format16bppArgb1555, PixelFormat.Format16bppGrayScale, PixelFormat.Format16bppRgb555, PixelFormat.Format16bppRgb565
They all throw a System.NotImplementedException with the error message "Not Implemented". I can't allocate 16 bits per pixel manually because Bitmap.SetPixel() only allows you to set a byte to alpha, red, green, or blue (so I need to use marshalling which requires a particular pixel format). Rather than setting a byte for each color value, I'm trying to distribute 16 bits between red, green, and blue. For example 5 bits for red, 5 bits for green, and 5 bits for blue.
Below is an example of one of the 16 bits per pixel pixel formats that throws a System.NotImplementedException:
Bitmap bitmap = new Bitmap(width, height, PixelFormat.Format16bppArgb1555);
My system is MacOS.

System.Drawing does not support every pixel format on all platforms. Once I collected the differences (see the Restrictions of Possible Pixel Formats on Different Platforms part). I think MacOS uses the same package as the Linux version.
As Bitmap encoders cannot save images preserving the Format16bppArgb1555 pixel format anyway I suggest you to create a Bitmap instance using the closest compatible format (for example, Format32bppArgb instead of Format16bppArgb1555), and if you really want to transform the colors into the ARGB1555 color space, you can use the Quantize method with PredefinedColorsQuantizer.Argb1555 using the same library I linked for the comparison. You can download it from NuGet.
Disclaimer: I dit not test it on MacOS but it works in Linux (Ubuntu).
Rather than setting a byte for each color value, I'm trying to distribute 16 bits between red, green, and blue. For example 5 bits for red, 5 bits for green, and 5 bits for blue.
Edit: I'm already working on the next version, which will allow using a managed bitmap data with any PixelFormat on any platform. See the public BitmapDataFactory.CreateBitmapData method (not available in a public release yet but you can already play with it if you check out the Development branch).
Update:
I have image bytes (i.e. bytes representing only an image, no header), and I'm testing to see if the format of the image the image bytes decode to is 16 bits per pixel.
Now I see why you need to set pixels in a 16-bit raw format. As in Linux/MacOS the System.Drawing.Bitmap has a limited support the easiest solution if you create managed bitmap data as I mentioned above.
Note: I published a pre-release, which makes the BitmapDataFactory.CreateBitmapData method available without checking out my development branch. And now you can turn your raw bytes of any PixelFormat to a Bitmap like this:
(of course, if pfGuess is not the actual format the result can be messy)
public Bitmap ToBitmap(byte[] rawData, PixelFormat pfGuess, Size size)
{
// the result bitmap will be ARGB32, which works on every platform
Bitmap result = new Bitmap(size.Width, size.Height);
// we create an accessor for the result so we can quickly copy the translated pixels
using var bitmapDataDst = result.GetWritableBitmapData();
// For the managed source any pixel format will work.
// Please note though that unlike System.Drawing.Bitmap images,
// managed bitmap data never has padding bytes at the end of each line.
// So if your rawData has padding (eg. when size has odd width and pf is 16bpp),
// then maybe the best if you include the padding in width.
// See https://learn.microsoft.com/en-us/windows/win32/medfound/image-stride for details
using var bitmapDataSrc = BitmapDataFactory.CreateBitmapData(size, pfGuess);
// Checking if rawData is long enough. RowSize is the width in bytes.
if (rawData.Length < bitmapDataSrc.RowSize * size.Height)
throw new ArgumentException("rawData is too short");
int currentByte = 0;
// iterating through rows
for (int y = 0; y < size.Height; y++)
{
var rowSrc = bitmapDataSrc[y]; // has the specified pixel format
var rowDst = bitmapDataDst[y]; // has ARGB32 pixel format
// writing raw pixels
for (int x = 0; x < bitmapDataSrc.RowSize; x++)
rowSrc.WriteRaw<byte>(x, rawData[currentByte++]);
// copying the pixels interpreted as pfGuess to result
for (int x = 0; x < size.Width; x++)
rowDst[x] = rowSrc[x];
}
// The result bitmap is now populated from the arbitrarily interpreted raw data.
// As it has ARGB32 format you can save it on any platform without any issues.
return result;
}

C# Converting 32bpp image to 8bpp

I'm trying to convert a 32bpp screenshot image to an 8bpp (or 4bpp, or 1bpp) format using C#. I've already looked at several stackoverflow answers on similar subjects and most suggest variations using the following code:
public static Bitmap Convert(Bitmap oldbmp)
{
Bitmap newbmp = new Bitmap(oldbmp.Width, oldbmp.Height, PixelFormat.Format8bppIndexed);
Graphics gr = Graphics.FromImage(newbmp);
gr.PageUnit = GraphicsUnit.Pixel;
gr.DrawImageUnscaled(oldbmp, 0, 0);
return newbmp;
}
However, when this executes, I get a the exception: A graphics object cannot be created from an image that has an indexed pixel format. I understand that 8, 4 and 1bpp images have colour table mappings rather than the actual colour pixels themselves (as in 32 or 16bpp images) so I assume I'm missing some conversion step somewhere, but I'm fairly new to C# (coming from a C++ background) and would prefer to be able do this using native C# calls rather than resorting to PInvoking BitBlt and GetDIBits etc. Anybody able to help me solve this? Thanks.
EDIT: I should point out that I need this to be backwardly compatible to .NET framework 2.0

GDI+ in general has very poor support for indexed pixel formats. There is no simple way to convert an image with 65536 or 16 million colors into one that only has 2, 16 or 256. Colors have to be removed from the source image and that is a lossy conversion that can have very poor results. There are multiple algorithms available to accomplish this, none of them are perfect for every kind of image. This is a job for a graphics editor.
There is one trick I found. GDI+ has an image encoder for GIF files. That's a graphics format that has only 256 colors, the encoder must limit the number of colors. It uses a dithering algorithm that's suitable for photos. It does have a knack for generating a grid pattern, you'll be less than thrilled when it does. Use it like this:
public static Image Convert(Bitmap oldbmp) {
using (var ms = new MemoryStream()) {
oldbmp.Save(ms, ImageFormat.Gif);
ms.Position = 0;
return Image.FromStream(ms);
}
}
The returned image has a 8bpp pixel format with the Palette entries calculated by the encoder. You can cast it to Bitmap if necessary. By far the best thing to do is to simply not bother with indexed formats. They date from the stone age of computing back when memory was severely constrained. Or use a professional graphics editor.

AForge library is doing it perfectly using Grayscale.
var bmp8bpp = Grayscale.CommonAlgorithms.BT709.Apply(bmp);
This class is the base class for image grayscaling [...]
The filter accepts 24, 32, 48 and 64 bpp color images and produces 8
(if source is 24 or 32 bpp image) or 16 (if source is 48 or 64 bpp
image) bpp grayscale image.

Negative stride signifies the image is bottom-up (inverted). Just use the absolute of the stride if you dont care. I know that works for 24bpp images, unaware if it works for others.

You can use System.Windows.Media.Imaging in PresentationCore Assembly take a look at here for more information

How to load PNGs and export to TGA keeping alpha in C#?

I'm writing a tool to automate some of our asset making for a game. What I want to do is take a folder of PNG files, combine them into a texture atlas and then export the atlas as a TGA and the UV coords to XML.
I'm not sure which method I should use to load the PNG files in C# as there seem to be several. What is the recommended method to load images in C# that gives access to the colour/alpha data so I can extract it to the TGA?
I also already have TGA creation code in C++ which I plan to move to C# but I'm wondering if there is anything already available in .Net to create/save TGAs?
Thanks for reading.

Loading a PNG file into a .Net Bitmap is easy:
Bitmap bmp = (Bitmap)Bitmap.FromFile("c:\wherever\whatever.png");
// yes, the (Bitmap) cast is necessary. Don't ask me why.
Once you have the Bitmap loaded, you can access all of its info (including alpha channel info) most efficiently using the Bitmap's LockBits method (there are many LockBits code samples on StackOverflow).
Update: here's a code sample that shows how to use LockBits to access the Bitmap's data pixel-by-pixel:
System.Drawing.Imaging.BitmapData data =
bmp.LockBits(new Rectangle(0, 0, bmp.Width, bmp.Height),
System.Drawing.Imaging.ImageLockMode.ReadOnly, bmp.PixelFormat);
unsafe
{
// important to use the BitmapData object's Width and Height
// properties instead of the Bitmap's.
for (int x = 0; x < data.Width; x++)
{
int columnOffset = x * 4;
for (int y = 0; y < data.Height; y++)
{
byte* row = (byte*)data.Scan0 + (y * data.Stride);
byte B = row[columnOffset];
byte G = row[columnOffset + 1];
byte R = row[columnOffset + 2];
byte alpha = row[columnOffset + 3];
}
}
}
bmp.UnlockBits(data);
You need to set the "Allow unsafe code" compiler option for your project to use this code. You could also use the GetPixel(x, y) method of the Bitmap, but this is amazingly slow.

I dont have a code sample, but i can give you a guideline
Load PNG using Image.FromFile(). .NET supports loading PNG
Open a file handle to your targa. Create an XmlDocument. .NET doesnt support targa, so you have to manually write it.
Lock the bitmap to get at the pixels. GetPixel() is very slow. I think the method is named LockBits(). You get a pointer to the surface to read the pixels
Write to targa. Targa is a container format, so any bitmap should fit.
Save the UV as Xml.
Targa format
Do you want to use a palette ? Since your making a game, i would recommend you compute a palette for your bitmaps and put that into the targa to reduce the file size.
Oh , before i forget, .NET doesnt use RGBA, instead, the pixels are BGRA. Dont ask me why, but its like that.

Is there any way to do Image Quantization safely and with no Marshalling?

I'm currently using Brendan Tompkins ImageQuantization dll.
http://codebetter.com/blogs/brendan.tompkins/archive/2007/06/14/gif-image-color-quantizer-now-with-safe-goodness.aspx
But it doesn't run in medium trust in asp.net.
Does anyone know of a Image Quantization library that does run in medium trust?
Update
I don't care if the solution is slow. I just need something that works.

You should be able to replace the code using Marshal with explicit reading of the underlying stream via something like BinaryReader. This may be slower since you must read the stream entirely into your managed memory or seek into it rather than relying on the copy already in unmanaged memory being quickly accessible but is fundamentally your only option.
You simply cannot go spelunking into unmanaged memory from a medium trust context, even if only performing read operations.
Having looked at the linked code there's a reason you're not allowed to do this sort of thing. For starters he's ignoring the 64/32bit aspect of the IntPtr!
The underlying BitMapData class he's using is utterly predicated on having unfettered read access to arbitrary memory, this is never happening under medium trust.
A significant rewrite of his base functionality will be required to either use BitMap's directly (with the slow GetPixel calls) or read the data directly via conventional stream apis, dropping it into an array(s) and then parse it out yourself. Neither of these are likely to be pleasant. The former will be much slower (I would expect order of magnitude due to the high overhead per pixel read), the later less slow (though still slower) but has much more associated effort in terms of rewriting the low level parsing of the image data.
Here's a rough guide to what you need to change based on the current code:
from Quantizer.cs
public Bitmap Quantize(Image source)
{
// Get the size of the source image
int height = source.Height;
int width = source.Width;
// And construct a rectangle from these dimensions
Rectangle bounds = new Rectangle(0, 0, width, height);
// First off take a 32bpp copy of the image
Bitmap copy = new Bitmap(width, height, PixelFormat.Format32bppArgb);
// And construct an 8bpp version
Bitmap output = new Bitmap(width, height, PixelFormat.Format8bppIndexed);
// Now lock the bitmap into memory
using (Graphics g = Graphics.FromImage(copy))
{
g.PageUnit = GraphicsUnit.Pixel;
// Draw the source image onto the copy bitmap,
// which will effect a widening as appropriate.
g.DrawImage(source, bounds);
}
//!! BEGIN CHANGES - no locking here
//!! simply use copy not a pointer to it
//!! you could also simply write directly to a buffer then make the final immage in one go but I don't bother here
// Call the FirstPass function if not a single pass algorithm.
// For something like an octree quantizer, this will run through
// all image pixels, build a data structure, and create a palette.
if (!_singlePass)
FirstPass(copy, width, height);
// Then set the color palette on the output bitmap. I'm passing in the current palette
// as there's no way to construct a new, empty palette.
output.Palette = GetPalette(output.Palette);
// Then call the second pass which actually does the conversion
SecondPass(copy, output, width, height, bounds);
//!! END CHANGES
// Last but not least, return the output bitmap
return output;
}
//!! Completely changed, note that I assume all the code is changed to just use Color rather than Color32
protected virtual void FirstPass(Bitmap source, int width, int height)
{
// Loop through each row
for (int row = 0; row < height; row++)
{
// And loop through each column
for (int col = 0; col < width; col++)
{
InitialQuantizePixel(source.GetPixel(col, row));
} // Now I have the pixel, call the FirstPassQuantize function...
}
}
you would need to do roughly the same in the other functions.
This removes any need for Color32, the Bitmap class will deal with all that for you.
Bitmap.SetPixel() will deal with the second pass. Note that this is the easiest way to port things but absolutely not the fastest way to do it within a medium trust environment.