Is it possible to do Unsafe Image Processing in Python?
As with C# I encountered a hard wall with my pixel processing in Python as getPixel method from Image is simply running too slow.
Is it possible to get a direct access to the image in memory like LockBits in c# does? It will make my program run much faster.
Thanks,
Mark
There is nothing "unsafe" about this.
Once you understand how Python works, it becomes patent that calling a method to retrieve information on each pixel is going to be slow.
First of all, although you give no information on it, I assume you are using "Pillow" - the Python Image Library (PIL) which is the most well known library for image manipulation using Python. As it is a third party package, nothing obliges you to use that. (PIL does have a getpixel method on images, but not a getPixel one)
One straightforward way to have all data in a manipulable way is to create a bytearray object of the image data - given an image in a img variable you can do:
data = bytearray(img.tobytes())
And that is it, you have linear access to all data on your image. To get to an specific Pixel in there, you need to get the image width , heigth and bytes-per-pixel. The later one is not a direct Image attribute in PIL, so you have to compute it given the Image's mode. The most common image types are RGB, RGBA and L.
So, if you want to write out a rectangle at "x, y, width, size" at an image, you can do this:
def rectangle(img, x,y, width, height):
data = bytearray(img.tobytes())
blank_data = (255,) * witdh * bpp
bpp = 3 if data.mode == 'RGB' else 4 if data.mode == 'RGBA' else 1
stride = img.width * bpp
for i in range(y, y + height):
data[i * stride + x * bpp: i * stride + (x + width) * bpp)] = blank_data
return Image.frombytes(img.mode, (img.width, img.height), bytes(data))
That is not much used, but for simple manipulation. People needing to apply filters and other more complicated algorithms in images with Python usually access the image using numpy - python high-performance data manipulation package, wich is tightly coupled with a lot of other packages that have things specific for images - usually installed as scipy.
So, to have the image as an ndarray, which already does all of the above coordinate -> bytes conversion for you, you can use:
import scipy.misc
data = scipy.misc.imread(<filename>)
(Check the docs at https://docs.scipy.org/doc/scipy/reference/)
Related
I need to mask image from memory buffers (rectangle area filled with black). So I naively re-use the Bitmap class with ImageFormat.MemoryBmp for my API. This works quite well on my local machine:
public static void MaskBitmap(Bitmap input, Rectangle maskRegion)
{
var bytesPerPixel = Image.GetPixelFormatSize(input.PixelFormat) / 8;
var row = new byte[maskRegion.Width * bytesPerPixel];
var maskData = input.LockBits(maskRegion, ImageLockMode.WriteOnly, input.PixelFormat);
for (var i = 0; i < maskData.Height; ++i)
{
Marshal.Copy(row, 0, maskData.Scan0 + (i * maskData.Stride), row.Length);
}
input.UnlockBits(maskData);
}
However when deploying to production it turns out that the following throw a NotImplementedException:
var image16 = new Bitmap(512, 512, PixelFormat.Format16bppGrayScale);
I eventually tracked it down to here:
https://github.com/mono/libgdiplus/blob/6.0.4/src/bitmap.c#L848
So my question is: is there any existing class in c# that I can re-use to hold images of pixelformat type:
PixelFormat.Format8bppIndexed:
PixelFormat.Format16bppGrayScale:
PixelFormat.Format24bppRgb:
I know GDI+ does not support saving/displaying 16bits image, I simply need a memory structure with image-style access.
Just for reference, I tried the following hack:
var image = new Bitmap(512,512,PixelFormat.Format24bppRgb);
image.Flags = ImageFlags.ColorSpaceGray;
But Flags is read-only.
As you could see, the GDI+ Bitmap does not support the 16bpp grayscale pixel format on Linux at all, and actually its support is quite limited on Windows, too. Once I collected the limitations for both platforms, see the table under the Restrictions of Possible Pixel Formats on Different Platforms section here.
I need to mask image from memory buffers
To use completely managed in-memory representation of a bitmap both on Linux and Windows, you can use this library (disclaimer: written by me). You can create a 16bpp grayscale bitmap data by the BitmapDataFactory.CreateBitmapData method, which returns an IReadWriteBitmapData that allows a lot of managed operations (see the link that enlists the usable extension methods). You can even convert it to an actual Bitmap by the ToBitmap extension, but on Linux this converts the result to a Bitmap with 24bpp RGB pixel format.
Example:
var image16 = BitmapDataFactory.CreateBitmapData(new Size(512, 512), PixelFormat.Format16bppGrayScale);
var row = image16.FirstRow;
do
{
for (int x = 0; x < image16.Width; x++)
{
// row[x] uses a Color32 structure (8 bit per color) but raw access
// enables you to use the full 16-bit color range:
row.WriteRaw<ushort>(x, 32768); // 50% gray
}
} while (row.MoveNextRow());
As for the 8bpp indexed and 24bpp RGB formats, these are supported by the native Bitmap also on Linux, but please note that starting with version .NET 6 System.Drawing will be supported only on Windows by default. Microsoft recommends using other libraries instead, but you can still enable the Unix support by adding "System.Drawing.EnableUnixSupport": true to runtimeconfig.json. Or, if you decide to use my library I mentioned above, just call DrawingModule.Initialize() before anything else, which enables the Unix support without editing any config files.
I think the Wpf bitmaps should support 16-bit grayscale.
However, most systems I have worked on that uses 16-bit grayscale images uses a custom data type. Something like:
public class My16BitImage{
public ushort[] Data {get;}
public int Width {get;}
public int Height {get;}
}
Note that for displaying the image you most probably need to convert it to an 8-bit image anyway, and you probably need to scale the values to make the max/min values map to the largest/smallest 8 bit values.
I'm using the Kinect 2 for Windows and the C# version of the SDK. If needed writing a separate C++ lib or using C#'s unsafe regions for better performance is definitely an option
I'm trying to downsample the input of the Kinect's Color Camera as 1920x1080 pixels # 30 fps is a bit much. But I cannot find a built in function to reduce the resolution (very odd, am I missing something?)
My next idea was to store the data in a large byte[] and then selectively sample from that byte[] directly into another byte[] to reduce the amount of data.
int ratio = full.Length / smallBuffer.Length;
int bpp = (int)frameDescription.BytesPerPixel;
for (int i = 0; i < small.Length; i += bpp)
{
Array.Copy(full, i * ratio, small, i, bpp);
}
However, this method gives me a very funny result. The image has the correct width and height but the image is repeated along the horizontal axis multiple times. (Twice if I use half the original resoltion, thrice if I use a third, etc...).
How can I correctly downsample (subsample is actually a better description) the video?
My final solution was letting the encoder (x264VFW in my case do the downsampling, the real bottleneck turned out to be the copying of the array which was solved by giving the encoder a pointer to where the array was in managed memory (using a GCHandle).
I've read many C# tutorials on using lockbits to manipulate images, but I just don't know how to apply this info into PowerShell.
This is the problem:
$image1's height is 2950 pixels. $image2's height is 50 px taller, 3000 pixels. I need to fit $image2 into $image1 and I can skip $image2's first 49 px lines. So in pseudocode:
For(y=0... For(x=0.... { image1(x,y) = image2(x,y+50) } ....))
The PowerShell script below works, but does not work very fast:
$rect = new-object Drawing.Rectangle 0, 0, $image1.width, $image1.height
$image1drawing.drawimage($image2,
$rect,
0, 50, $image2.width, ($image2.height - 50),
$graphicalUnit)
The pages I've found, such as this (Not able to successfully use lockbits) or this (https://web.archive.org/web/20121203144033/http://www.bobpowell.net/lockingbits.htm) are in "plain English" but how to convert this concept into PowerShell?
You are using the correct approach. Using DrawImage will be faster that copying the pixels one by one.
Some suggestions to make it faster:
Try using Image.Clone to copy a rectangle from the original image, this will result in the smallest number of objects you need to create.
Make sure you use the same PixelFormat as the original image (faster copying). There is a PixelFormat attribute in Image.
Most important: accessing Width and Height takes a long time so save them to a local variable for reuse. If you know them before hand that's a good way to speed-up things.
Don't expect miracles at a width to height ratio of 4:3 each image is 3932 * 2950 * 3(assuming 24 bit RGB) = 33Mb per image. That's a lot of data, you may easily be be trying to copy a few gigs depending on how many images you have.
You are better off writing a simple cmdlet and using it in your PowerShell script.
BTW in case you are still interested:
Using lockbits in C# (as per your examples) relies on unsafe context and using pointers. I don't believe PowerShell has access to unsafe context.
You can manipulate unmanaged data without using an unsafe context by using the Marshall class; specifically the Read and Write methods (and you might be able to speed things up with the Copy method at expense of memory).
PowerShell was not meant as a replacement for .Net generic languages, that's what CmdLets are for.
I am receiving an image from an external system in the form of a sequence of BGR values followed by an empty byte. The sequence looks sort of like...
[B,G,R,0,B,G,R,0,...,B,G,R,0] where each BGR0 is a single pixel in an image.
I need this in a .NET Bitmap so I can perform some manipulations on it and have come up with the following function to do so:
private Bitmap fillBitmap(byte[] data, int width, int height)
{
Bitmap map = new Bitmap(width, height);
for (int i = 0; i < data.Length; i += 4)
{
int y = ((i / 4) / width);
int x = ((i / 4) - (y * width));
int b = data[i];
int g = data[i + 1];
int r = data[i + 2];
Color pixel = Color.FromArgb(r, g, b);
map.SetPixel(x, y, pixel);
}
return map;
}
This would normally be ok except that most of my images are 1920x1200... so I have a loop that's iterating over 2 million times. Even then that wouldn't be so bad as 2 million iterations shouldn't be very taxing on any modern processor.
But for some reason, this loop can take upwards of 5-15 seconds to run on a pretty beefy Xeon on my server. It would be trivial to parallelize the loop but I suspect there is an even better way of going about this. Any help would be greatly appreciated!
The description of the Bitmap.LockBits Method says,
You can change the color of an image with the SetPixel method, although the LockBits method offers better performance for large-scale changes.
An alternative, I'd guess, might be to use the Bitmap(Stream) Constructor, after you create a Stream which matches the file format of a bitmap.
Check some FastBitmap implementation that will help you set the array from which you will be able to generate the image.
A call to GetPixel and SetPixel makes an interop call to native functions. Each time you call one of the functions the Bitmap image is locked the corresponding pixel/bytes is modified and then the image is finally unlocked. You can imagine performing this repeatedly is highly inefficient.
As others have suggested use the LockBits method, although you will have to use unsafe code I imagine. Now if this is allowed, use it. It allows direct access to the Bitmap's pixels in an unmanaged memory buffer.
I'm writing a tool to automate some of our asset making for a game. What I want to do is take a folder of PNG files, combine them into a texture atlas and then export the atlas as a TGA and the UV coords to XML.
I'm not sure which method I should use to load the PNG files in C# as there seem to be several. What is the recommended method to load images in C# that gives access to the colour/alpha data so I can extract it to the TGA?
I also already have TGA creation code in C++ which I plan to move to C# but I'm wondering if there is anything already available in .Net to create/save TGAs?
Thanks for reading.
Loading a PNG file into a .Net Bitmap is easy:
Bitmap bmp = (Bitmap)Bitmap.FromFile("c:\wherever\whatever.png");
// yes, the (Bitmap) cast is necessary. Don't ask me why.
Once you have the Bitmap loaded, you can access all of its info (including alpha channel info) most efficiently using the Bitmap's LockBits method (there are many LockBits code samples on StackOverflow).
Update: here's a code sample that shows how to use LockBits to access the Bitmap's data pixel-by-pixel:
System.Drawing.Imaging.BitmapData data =
bmp.LockBits(new Rectangle(0, 0, bmp.Width, bmp.Height),
System.Drawing.Imaging.ImageLockMode.ReadOnly, bmp.PixelFormat);
unsafe
{
// important to use the BitmapData object's Width and Height
// properties instead of the Bitmap's.
for (int x = 0; x < data.Width; x++)
{
int columnOffset = x * 4;
for (int y = 0; y < data.Height; y++)
{
byte* row = (byte*)data.Scan0 + (y * data.Stride);
byte B = row[columnOffset];
byte G = row[columnOffset + 1];
byte R = row[columnOffset + 2];
byte alpha = row[columnOffset + 3];
}
}
}
bmp.UnlockBits(data);
You need to set the "Allow unsafe code" compiler option for your project to use this code. You could also use the GetPixel(x, y) method of the Bitmap, but this is amazingly slow.
I dont have a code sample, but i can give you a guideline
Load PNG using Image.FromFile(). .NET supports loading PNG
Open a file handle to your targa. Create an XmlDocument. .NET doesnt support targa, so you have to manually write it.
Lock the bitmap to get at the pixels. GetPixel() is very slow. I think the method is named LockBits(). You get a pointer to the surface to read the pixels
Write to targa. Targa is a container format, so any bitmap should fit.
Save the UV as Xml.
Targa format
Do you want to use a palette ? Since your making a game, i would recommend you compute a palette for your bitmaps and put that into the targa to reduce the file size.
Oh , before i forget, .NET doesnt use RGBA, instead, the pixels are BGRA. Dont ask me why, but its like that.