We Keep Coding

Tensorflowsharp results getvalue() is very slow

I am using TensorflowSharp to run evaluations using a neural network on an Android phone. I am building the project with Unity.
I am using the tensorflowsharp unity plugin listed under the requirements here: https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Using-TensorFlow-Sharp-in-Unity.md.
Everything is working, however extracting the result is very slow.
The network I am running is an autoencoder and the output is an image with dimensions of 128x128x16 (yes there is a lot of output channels).
The evaluation is done in ~ 0.2 seconds which is acceptable. However when i need to extract the result data using results[0].GetValue() it is VERY slow.
This is my code where i run the neural network
var runner = session.GetRunner();
runner.AddInput(graph[INPUT_NAME][0], tensor).Fetch(graph[OUTPUT_NAME][0]);
var results = runner.Run();
float[,,,] heatmaps = results[0].GetValue() as float[,,,]; // <- this is SLOW
The problem:
The last line where i convert the result to floats is taking ~1.2 seconds.
Can it realy be true that reading the result data into a float array is taking more than 5 times as long as the actual evaluation of the network?
Is there another way to extract the result values?

So I have found a solution to this. I still do not know why the GetValue() call is so slow, but I found another way to retrieve the data.
I chose to manually read the raw tensor data available at results[0].Data
I created a small function to handle this as a drop in for GetValue, (Here just with the dimensions i am expecting hardcoded)
private float[,,,] TensorToFLoats(TFTensor tensor)
{
IntPtr resData = tensor.Data;
UIntPtr dataSize = tensor.TensorByteSize;
byte[] s_ImageBuffer = new byte[(int)dataSize];
System.Runtime.InteropServices.Marshal.Copy(resData, s_ImageBuffer, 0, (int)dataSize);
int floatsLength = s_ImageBuffer.Length / 4;
float[] floats = new float[floatsLength];
for (int n = 0; n < s_ImageBuffer.Length; n += 4)
{
floats[n / 4] = BitConverter.ToSingle(s_ImageBuffer, n);
}
float[,,,] result = new float[1, 128, 128, 16];
int i = 0;
for (int y = 0; y < 128; y++)
{
for (int x = 0; x < 128; x++)
{
for (int p = 0; p < 16; p++)
{
result[0, y, x, p] = floats[i++];
}
}
}
return result;
}
Given this i can replace the code in my question with the following
var runner = session.GetRunner();
runner.AddInput(graph[INPUT_NAME][0], tensor).Fetch(graph[OUTPUT_NAME][0]);
var results = runner.Run();
float[,,,] heatmaps = TensorToFLoats(results[0]);
This is insanely much faster. Where GetValue took ~1 second the TensorToFloats function i created got the same data in ~0.02 seconds

Saving array of bitmaps to separate files with C#

I'm stuck at saving an array of System.Drawing.Bitmap type, each bitmap to separate file.
I have an array "survey". This array stores several Lists of double type.
For each List i want to create a bitmap and then save it as a bmp file.
The line raport[i].Save(Path.Combine(myfilepath, nets[i] + ".bmp")); returns TypeInitializationException - and i don't know why.
The piece nets[i] is a dictionary (int, string) with expected file names.
public void save_results()
{
System.Drawing.Bitmap[] raport = new System.Drawing.Bitmap[survey.Length];
for (int i = 0; i < survey.Length; i++)
{
raport[i] = new System.Drawing.Bitmap(survey[i].Count, 1000);
for (int x = 0; x < survey[i].Count; x++)
for (int y = 0; y < 1000; y++)
raport[i].SetPixel(x, y, Color.FromArgb(255, 255, 255));
for (int x = 0; x < survey[i].Count; x++)
raport[i].SetPixel(x, (int)(1000 - Math.Floor(survey[i][x] * 1000) >= 1000 ? 999 : 1000 - Math.Floor(survey[i][x] * 1000)), Color.FromArgb(0, 0, 0));
raport[i].Save(Path.Combine(myfilepath, nets[i] + ".bmp"));
}
}

Finally, the problem was associated with the variable "myfilepath".
The variable was 'compiled' from few file paths - and all of those strings should have been static:
public static string mydoc= Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments);
public static string myfilepath_p = Path.Combine(mydoc, "Demeter");
public static string myfilepath= Path.Combine(myfilepath_p, "regresja_liniowa");
Originally, only the 'final' variable used in the cited code was static, what caused an error.
Rest of the code worked fine.

Passing a List as a parameter to a Matlab function from C# project

I'm trying to run The Fast Compressive Tracking Algorithm from my C# project. What I want is that, (in C#)to pass the images after converting them into gray scale images to the Matlab function
Runtracker
instead of having the images in the Matlab code as I'm going to apply some operations to specify the index of the array that containing the images to start tracking from.
But when I pass the list of the grayscale images, I got an error says
" The best overload method match for 'trackerNative.FCT.Runtracker(int)' has some invalid arguments."
Can you help me solve this? to pass List of images in C# to a matlab function.
C# Code
//to convert to graScale
public double[,] to_gray(Bitmap img)
{
int w = img.Width;
int h = img.Height;
double[,] grayImage = new double[w, h];
for (int i = 0; i < w; i++)
{
for (int x = 0; x < h; x++)
{
Color oc = img.GetPixel(i, x);
grayImage[i, x] = (double)(oc.R + oc.G + oc.B) / 3;
}
}
return grayImage;
}
//to get the files from specified folder
public static String[] GetFilesFrom(String searchFolder, String[] filters, bool isRecursive)
{
List<String> filesFound = new List<String>();
var searchOption = isRecursive ? SearchOption.AllDirectories : SearchOption.TopDirectoryOnly;
foreach (var filter in filters)
{
filesFound.AddRange(Directory.GetFiles(searchFolder, String.Format("*.{0}", filter), searchOption));
}
return filesFound.ToArray();
}
//Button to run the matlab function 'Runtracker'
private void button1_Click(object sender, EventArgs e)
{
FCT obj = new FCT(); //FCT is a matlab class
OpenFileDialog openFileDialog1 = new OpenFileDialog();
if (openFileDialog1.ShowDialog() == DialogResult.OK)
{
string file_path = openFileDialog1.FileName;
var filters = new String[] { "png" };
var files = GetFilesFrom(file_path, filters, false);
List< double[,] > imgArr = new List< double[,] >();
for (int i = 0; i < files.Length; i++)
{
Bitmap image = new Bitmap(files[i]);
double[,] grayScale_image = to_gray(image);
imgArr[i] = grayScale_image;
}
object m = obj.Runtracker(imgArr); //the error occured
//Bitmap output = m as Bitmap;
}
}
Matlab Code
function Runtracker(input_imgArr)
clc;clear all;close all;
rand('state',0);
%%
[initstate] = initCt(1);% position of the detected face in the 1st frame
num = length(input_imgArr);% number of frames
%%
x = initstate(1);% x axis at the Top left corner
y = initstate(2);
w = initstate(3);% width of the rectangle
h = initstate(4);% height of the rectangle
%---------------------------
img = imread(input_imgArr(1));
img = double(img);
%%
trparams.init_negnumtrain = 50;%number of trained negative samples
trparams.init_postrainrad = 4;%radical scope of positive samples
trparams.initstate = initstate;% object position [x y width height]
trparams.srchwinsz = 25;% size of search window
%-------------------------
%% Classifier parameters
clfparams.width = trparams.initstate(3);
clfparams.height= trparams.initstate(4);
% feature parameters
% number of rectangle from 2 to 4.
ftrparams.minNumRect =2;
ftrparams.maxNumRect =4;
M = 100;% number of all weaker classifiers, i.e,feature pool
%-------------------------
posx.mu = zeros(M,1);% mean of positive features
negx.mu = zeros(M,1);
posx.sig= ones(M,1);% variance of positive features
negx.sig= ones(M,1);
lRate = 0.85;% Learning rate parameter
%% Compute feature template
[ftr.px,ftr.py,ftr.pw,ftr.ph,ftr.pwt] = HaarFtr(clfparams,ftrparams,M);
%% Compute sample templates
posx.sampleImage = sampleImgDet(img,initstate,trparams.init_postrainrad,1);
negx.sampleImage = sampleImg(img,initstate,1.5*trparams.srchwinsz,4+trparams.init_postrainrad,trpar ams.init_negnumtrain);
%% Feature extraction
iH = integral(img);%Compute integral image
posx.feature = getFtrVal(iH,posx.sampleImage,ftr);
negx.feature = getFtrVal(iH,negx.sampleImage,ftr);
[posx.mu,posx.sig,negx.mu,negx.sig] =
classiferUpdate(posx,negx,posx.mu,posx.sig,negx.mu,negx.sig,lRate);%
update distribution parameters
%% Begin tracking
for i = 2:num
img = imread(input_imgArr(i));
imgSr = img;% imgSr is used for showing tracking results.
img = double(img);
iH = integral(img);%Compute integral image
%% Coarse detection
step = 4; % coarse search step
detectx.sampleImage =
sampleImgDet(img,initstate,trparams.srchwinsz,step);
detectx.feature = getFtrVal(iH,detectx.sampleImage,ftr);
r = ratioClassifier(posx,negx,detectx.feature);% compute the classifier for all samples
clf = sum(r);% linearly combine the ratio classifiers in r to the final classifier
[c,index] = max(clf);
x = detectx.sampleImage.sx(index);
y = detectx.sampleImage.sy(index);
w = detectx.sampleImage.sw(index);
h = detectx.sampleImage.sh(index);
initstate = [x y w h];
%% Fine detection
step = 1;
detectx.sampleImage = sampleImgDet(img,initstate,10,step);
detectx.feature = getFtrVal(iH,detectx.sampleImage,ftr);
r = ratioClassifier(posx,negx,detectx.feature);% compute the classifier for all samples
clf = sum(r);% linearly combine the ratio classifiers in r to the final classifier
[c,index] = max(clf);
x = detectx.sampleImage.sx(index);
y = detectx.sampleImage.sy(index);
w = detectx.sampleImage.sw(index);
h = detectx.sampleImage.sh(index);
initstate = [x y w h];
%% Show the tracking results
imshow(uint8(imgSr));
rectangle('Position',initstate,'LineWidth',4,'EdgeColor','r');
hold on;
text(5, 18, strcat('#',num2str(i)), 'Color','y', 'FontWeight','bold', 'FontSize',20);
set(gca,'position',[0 0 1 1]);
pause(0.00001);
hold off;
%% Extract samples
posx.sampleImage = sampleImgDet(img,initstate,trparams.init_postrainrad,1);
negx.sampleImage =
sampleImg(img,initstate,1.5*trparams.srchwinsz,4+trparams.init_postrainrad,trparams.init_negnumtrain);
%% Update all the features
posx.feature = getFtrVal(iH,posx.sampleImage,ftr);
negx.feature = getFtrVal(iH,negx.sampleImage,ftr);
[posx.mu,posx.sig,negx.mu,negx.sig] = classiferUpdate(posx,negx,posx.mu,posx.sig,negx.mu,negx.sig,lRate);% update distribution parameters
end
end

This is because the Runtracker expects an argument of type int as per the error and you are passing an argument of type int[] i.e an integer array.
The passed parameter and the expected parameter should match in order to execute the method successfully.
Hope this helps.

Connected-component labeling algorithm optimization

I need some help with optimisation of my CCL algorithm implementation. I use it to detect black areas on the image. On a 2000x2000 it takes 11 seconds, which is pretty much. I need to reduce the running time to the lowest value possible to achieve. Also, I would be glad to know if there is any other algorithm out there which allows you to do the same thing, but faster than this one. So here is my code:
//The method returns a dictionary, where the key is the label
//and the list contains all the pixels with that label
public Dictionary<short, LinkedList<Point>> ProcessCCL()
{
Color backgroundColor = this.image.Palette.Entries[1];
//Matrix to store pixels' labels
short[,] labels = new short[this.image.Width, this.image.Height];
//I particulary don't like how I store the label equality table
//But I don't know how else can I store it
//I use LinkedList to add and remove items faster
Dictionary<short, LinkedList<short>> equalityTable = new Dictionary<short, LinkedList<short>>();
//Current label
short currentKey = 1;
for (int x = 1; x < this.bitmap.Width; x++)
{
for (int y = 1; y < this.bitmap.Height; y++)
{
if (!GetPixelColor(x, y).Equals(backgroundColor))
{
//Minumum label of the neighbours' labels
short label = Math.Min(labels[x - 1, y], labels[x, y - 1]);
//If there are no neighbours
if (label == 0)
{
//Create a new unique label
labels[x, y] = currentKey;
equalityTable.Add(currentKey, new LinkedList<short>());
equalityTable[currentKey].AddFirst(currentKey);
currentKey++;
}
else
{
labels[x, y] = label;
short west = labels[x - 1, y], north = labels[x, y - 1];
//A little trick:
//Because of those "ifs" the lowest label value
//will always be the first in the list
//but I'm afraid that because of them
//the running time also increases
if (!equalityTable[label].Contains(west))
if (west < equalityTable[label].First.Value)
equalityTable[label].AddFirst(west);
if (!equalityTable[label].Contains(north))
if (north < equalityTable[label].First.Value)
equalityTable[label].AddFirst(north);
}
}
}
}
//This dictionary will be returned as the result
//I'm not proud of using dictionary here too, I guess there
//is a better way to store the result
Dictionary<short, LinkedList<Point>> result = new Dictionary<short, LinkedList<Point>>();
//I define the variable outside the loops in order
//to reuse the memory address
short cellValue;
for (int x = 0; x < this.bitmap.Width; x++)
{
for (int y = 0; y < this.bitmap.Height; y++)
{
cellValue = labels[x, y];
//If the pixel is not a background
if (cellValue != 0)
{
//Take the minimum value from the label equality table
short value = equalityTable[cellValue].First.Value;
//I'd like to get rid of these lines
if (!result.ContainsKey(value))
result.Add(value, new LinkedList<Point>());
result[value].AddLast(new Point(x, y));
}
}
}
return result;
}
Thanks in advance!

You could split your picture in multiple sub-pictures and process them in parallel and then merge the results.
1 pass: 4 tasks, each processing a 1000x1000 sub-picture
2 pass: 2 tasks, each processing 2 of the sub-pictures from pass 1
3 pass: 1 task, processing the result of pass 2
For C# I recommend the Task Parallel Library (TPL), which allows to easily define tasks depending and waiting for each other. Following code project articel gives you a basic introduction into the TPL: The Basics of Task Parallelism via C#.

I would process one scan line at a time, keeping track of the beginning and end of each run of black pixels.
Then I would, on each scan line, compare it to the runs on the previous line. If there is a run on the current line that does not overlap a run on the previous line, it represents a new blob. If there is a run on the previous line that overlaps a run on the current line, it gets the same blob label as the previous. etc. etc. You get the idea.
I would try not to use dictionaries and such.
In my experience, randomly halting the program shows that those things may make programming incrementally easier, but they can exact a serious performance cost due to new-ing.

The problem is about GetPixelColor(x, y), it take very long time to access image data.
Set/GetPixel function are terribly slow in C#, so if you need to use them a lot, you should use Bitmap.lockBits instead.
private void ProcessUsingLockbits(Bitmap ProcessedBitmap)
{
BitmapData bitmapData = ProcessedBitmap.LockBits(new Rectangle(0, 0, ProcessedBitmap.Width, ProcessedBitmap.Height), ImageLockMode.ReadWrite, ProcessedBitmap.PixelFormat);
int BytesPerPixel = System.Drawing.Bitmap.GetPixelFormatSize(ProcessedBitmap.PixelFormat) / 8;
int ByteCount = bitmapData.Stride * ProcessedBitmap.Height;
byte[] Pixels = new byte[ByteCount];
IntPtr PtrFirstPixel = bitmapData.Scan0;
Marshal.Copy(PtrFirstPixel, Pixels, 0, Pixels.Length);
int HeightInPixels = bitmapData.Height;
int WidthInBytes = bitmapData.Width * BytesPerPixel;
for (int y = 0; y < HeightInPixels; y++)
{
int CurrentLine = y * bitmapData.Stride;
for (int x = 0; x < WidthInBytes; x = x + BytesPerPixel)
{
int OldBlue = Pixels[CurrentLine + x];
int OldGreen = Pixels[CurrentLine + x + 1];
int OldRed = Pixels[CurrentLine + x + 2];
// Transform blue and clip to 255:
Pixels[CurrentLine + x] = (byte)((OldBlue + BlueMagnitudeToAdd > 255) ? 255 : OldBlue + BlueMagnitudeToAdd);
// Transform green and clip to 255:
Pixels[CurrentLine + x + 1] = (byte)((OldGreen + GreenMagnitudeToAdd > 255) ? 255 : OldGreen + GreenMagnitudeToAdd);
// Transform red and clip to 255:
Pixels[CurrentLine + x + 2] = (byte)((OldRed + RedMagnitudeToAdd > 255) ? 255 : OldRed + RedMagnitudeToAdd);
}
}
// Copy modified bytes back:
Marshal.Copy(Pixels, 0, PtrFirstPixel, Pixels.Length);
ProcessedBitmap.UnlockBits(bitmapData);
}
Here is the basic code to access pixel data.
And I made a function to transform this into a 2D matrix, it's easier to manipulate (but little slower)
private void bitmap_to_matrix()
{
unsafe
{
bitmapData = ProcessedBitmap.LockBits(new Rectangle(0, 0, ProcessedBitmap.Width, ProcessedBitmap.Height), ImageLockMode.ReadWrite, ProcessedBitmap.PixelFormat);
int BytesPerPixel = System.Drawing.Bitmap.GetPixelFormatSize(ProcessedBitmap.PixelFormat) / 8;
int HeightInPixels = ProcessedBitmap.Height;
int WidthInPixels = ProcessedBitmap.Width;
int WidthInBytes = ProcessedBitmap.Width * BytesPerPixel;
byte* PtrFirstPixel = (byte*)bitmapData.Scan0;
Parallel.For(0, HeightInPixels, y =>
{
byte* CurrentLine = PtrFirstPixel + (y * bitmapData.Stride);
for (int x = 0; x < WidthInBytes; x = x + BytesPerPixel)
{
// Conversion in grey level
double rst = CurrentLine[x] * 0.0721 + CurrentLine[x + 1] * 0.7154 + CurrentLine[x + 2] * 0.2125;
// Fill the grey matix
TG[x / 3, y] = (int)rst;
}
});
}
}
And the website where the code comes
"High performance SystemDrawingBitmap"
Thanks to the author for his really good job !
Hope this will help !

Converting a simple JavaScript code

I've been trying to convert this JavaScript code that gets the dominant color from an image, so far with no success. I get errors with the colorCount & color variables. I don't know the suitable & equivalent data types to use for these variables. Here is my code:
public string dominantColor(Bitmap img)
{
int[] colorCount = new int[0];
int maxCount = 0;
string dominantColor = "";
// data is an array of a series of 4 one-byte values representing the rgba values of each pixel
Bitmap Bmp = new Bitmap(img);
BitmapData BmpData = Bmp.LockBits(new Rectangle(0, 0, Bmp.Width, Bmp.Height), ImageLockMode.ReadOnly, Bmp.PixelFormat);
byte[] data = new byte[BmpData.Stride * Bmp.Height];
for (int i = 0; i < data.Length; i += 4)
{
// ignore transparent pixels
if (data[i+3] == 0)
continue;
string color = data[i] + "." + data[i+1] + "," + data[i+2];
// ignore white
if (color == "255,255,255")
continue;
if (colorCount[color] != 0)
colorCount[color] = colorCount[color] + 1;
else
colorCount[color] = 0;
// keep track of the color that appears the most times
if (colorCount[color] > maxCount)
{
maxCount = colorCount[color];
dominantColor = color.ToString;
}
}
string rgb = dominantColor.Split(",");
return rgb;
}

I'll give you a complete managed version of your code:
static Color dominantColor(Bitmap img)
{
Hashtable colorCount = new Hashtable();
int maxCount = 0;
Color dominantColor = Color.White;
for (int i = 0; i < img.Width; i++)
{
for (int j = 0; j < img.Height; j++)
{
var color = img.GetPixel(i, j);
if (color.A == 0)
continue;
// ignore white
if (color.Equals(Color.White))
continue;
if (colorCount[color] != null)
colorCount[color] = (int)colorCount[color] + 1;
else
colorCount.Add(color, 0);
// keep track of the color that appears the most times
if ((int)colorCount[color] > maxCount)
{
maxCount = (int)colorCount[color];
dominantColor = color;
}
}
}
return dominantColor;
}
So what is the difference here?
- I use a Hashtable instead of your array (you never redefine the dimension of it - and the best way to use an extensible object from JavaScript is a Hashtable)
- I prefer to use the already included structure Color (which saves 4 bytes for Alpha, Red, Green, Blue)
- I also do the comparisons and return this structure (then you are free to do whatever you want to do - in JavaScript using those strings is just a workaround because the browser is just giving you such RGB(a) strings)
What is another problem in your code is the line containing byte[] data = new byte[BmpData.Stride * Bmp.Height]; - Your array is created and initialized but with no data (.NET will erase all previous data resulting in a lot of zeros). Therefore you will not anywhere.
Drawback of my version is that it is indeed very small (this is where your lockbits are coming into play). I can give you a non-managed version (using the lockbits and an unsafe-block) if you want to. Depends if performance matters a lot for you and if you are interested!