We Keep Coding

How to make Tesseract faster in C#?

I put a scaled (2x) and binarized image (text black, everything else white) into tesseract and want to do the OCR as fast as possible:
private OCRResultData getOCRResult(Bitmap image, int minConf)
{
// Prepare input image for tesseract
Pix imageAsPix = PixConverter.ToPix(image);
// Process image by engine
Page page = this.engine.Process(imageAsPix, PageSegMode.SparseText);
string result = page.GetTsvText(0);
// Get data (split string into array of arrays to make information easier to acces)
string[][] data = getDataArray(result);
// Create result object and return it (class defiend by myself)
OCRResultData resultData = new OCRResultData();
resultData.confidences = getConfidences(data, minConf);
resultData.boundingBoxes = getBoundingBoxes(data, minConf);
resultData.detectedText = getDetectedText(data, minConf);
resultData.labelCoordinates = getLabelCoordsList(data, minConf);
page.Dispose();
return resultData;
}
Problem:
The best method I found for getting information like the detected string, its confidence, the bounding box coords, etc. was page.GetTsvText(0). Now while the processing is really fast (this.engine.Process(imageAsPix, PageSegMode.SparseText) takes like 30 ms for a 3000px X 3000px image), the page.GetTsvText(0) method needs about 1800 ms, which is too slow for my needs.
Is there a way to get it faster?
As a last option, I would try to introduce a string property and call the GetTsvText(0) in an asynchronous method at programm start and set it that way; In the case the user does nothing in the first 1800ms after loading the image, he wouldn't feel the delay anymore when pressing the OCR button (the code uses a WinForms GUI);

How to display real time updates to an image in Unity?

I am trying to create an application that generates a bitmap image every frame based on user actions and have it display that image to the screen. I would like the application to also be able to update that image in unity in real time as soon as the user makes another action.
I have created an application that does this and it works. However, it is veryyyy slow. My Update() method is attached below.
My idea was:
Capture user data (mouse location).
Convert that data into a special signal format that another program recognizes.
Have that program return a bitmap image.
Use that bitmap as a texture and update the existing texture with the new image.
Code:
UnityEngine.Texture2D oneTexture;
Bitmap currentBitmap;
private int frameCount = 0;
void Update()
{
// Show mouse position in unity environment
double xValue = Input.mousePosition.x;
double yValue = Screen.height - Input.mousePosition.y;
myPoints = "" + xValue + "," + yValue + Environment.NewLine;
// Show heatmap being recorded.
signals = Program.ConvertStringToSignalsList(myPoints);
currentBitmap = Program.CreateMouseHeatmap(Screen.width, Screen.height, signals);
// Update old heatmap texture.
UpdateTextureFromBitmap();
ri.texture = oneTexture;
ri.rectTransform.sizeDelta = new Vector2(Screen.width, Screen.height);
frameCount++;
// Write points to Database.
StartCoroutine(WriteToDB(xValue, yValue)); // <<<<< Comment out when playback.
}
private void UpdateTextureFromBitmap()
{
// Convert Bitmap object into byte array instead of creating actual
// .bmp image file each frame.
byte[] imageBytes = ImageToBytes(currentBitmap);
BMPLoader loader = new BMPLoader();
BMPImage img = loader.LoadBMP(imageBytes);
// Only initialize the Texture once.
if (frameCount == 0)
{
oneTexture = img.ToTexture2D();
}
else
{
Color32[] imageData = img.imageData;
oneTexture.SetPixels32(imageData);
oneTexture.Apply();
}
}
I was wondering if someone could help me improve the rate at which the image updates to the screen? I know that it is possible to make this program much faster but I am so new to unity and C# that I don't know how to make that happen. Also if there is a completely different way that I should be going about doing this then I am open to that too. Any help would be appreciated. Thanks!
Also, below is a screenshot of the Profiler showing the breakdown of CPU Usage. Currently it looks like every frame is taking about 500ms.

Performance degrade when plotting animated graph

I plot animated graph with Dynamic Data Display - graph that has source with constantly adding new points (about 10 times per second and 10 points per time). The problem is the performance degrade if these points form line segments which are big in compare to plotter size. In other words I can add thousands of points but if they forms comparably small line segments the performance will be good. But if I add only few hundreds points which form big line segments (they occupy all the plotter's area) the performance will be very poor.
The following images illustrate the problem. The first image is just hundreds line segments and performance is very poor on my machine. In the second image I just add two points to make the rest points took small place on the plotter due to plotter's scaling.
It doesn't matter how fast your machine is (mine is Pentium G3220 with integrated GPU, 16 GB RAM, and another one is relatively same CPU with discrete GeForce 640 GT) - I can add much more points and will fall your machine down. The behavior will be the same - if I add two "defend scaling" points and make the line segments relatively small on the plotter the performance will be much better.
The following code - how I get such pictures. Leave the adding of two points in MainWindow() commented to obtain the first picture, and uncomment it to obtain the second one. This is just a plane WPF application with all code in one file - in MainWindow's "code behind":
public partial class MainWindow : Window
{
readonly DispatcherTimer _timer = new DispatcherTimer();
int _counter = 0;
int _batchSize= 10;
RingArray<Point> _data = new RingArray<Point>(2000);
EnumerableDataSource<Point> _ds;
public MainWindow()
{
InitializeComponent();
_ds = new EnumerableDataSource<Point>(_data);
_ds.SetXMapping(d => d.X);
_ds.SetYMapping(d => d.Y);
LineGraph chart = plotter.AddLineGraph(_ds, Colors.Blue, 2.0);
//_data.Add(
// new Point
// {
// X = -1,
// Y = 200
// });
//_data.Add(
// new Point
// {
// X = -0.5,
// Y = -200
// });
_timer.Tick += OnTimerTick;
_timer.Interval = TimeSpan.FromMilliseconds(100);
_timer.Start();
}
void OnTimerTick(object sender, EventArgs e)
{
_data.AddMany(
Enumerable.Range(0, _batchSize)
.Select(v =>
{
// I don't use the "v" here - the "_counter" is enough.
var p = new Point
{
X = (double)_counter,
Y = (double)(_counter % 2 == 0 ? 0 : 1)
};
_counter++;
return p;
}));
}
}
public class Point
{
public double X { get; set; }
public double Y { get; set; }
}
It doesn't matter whether I use EnumerableDataSource or ObservableDataSource with AppendAsync, Add or AddMany - it is all the same behavior: the performance depends only on the size of line segments relatively to screen size (or plotter size). The problem lays definitely in the field of 1) WPF's rendering system, and 2) rasterization.
1) bad WPF's rendering system https://jeremiahmorrill.wordpress.com/2011/02/14/a-critical-deep-dive-into-the-wpf-rendering-system/
2) rasterization - the more pixels the line segments occupy on the screen the less performance will be... but dude, several hundreds lines fall down my machine but Crysis on max settings not! Ridiculous!
I know the following approaches to cope with this problem:
reduce the size of the plotter (but in my case it have to be as large as possible because it is the main part of the user interface and it is dedicated for monitoring purposes);
reduce the size of line segments (not in my case because the data must be displayed as they are);
apply a filter to the data like plotting only each tenth point instead of every first one - already done but it has limited application (my graph looks too jugged);
What else would you offer?

parallel image processing artifacts

I capture images from a webcam, do some heavy processing on them, and then show the result. To keep the framerate high, i want to have the processing of different frames run in parallel.
So, I have a 'Producer', which captures the images and adds these to the 'inQueue'; also it takes an image from the 'outQueue' and displays it:
public class Producer
{
Capture capture;
Queue<Image<Bgr, Byte>> inQueue;
Queue<Image<Bgr, Byte>> outQueue;
Object lockObject;
Emgu.CV.UI.ImageBox screen;
public int frameCounter = 0;
public Producer(Emgu.CV.UI.ImageBox screen, Capture capture, Queue<Image<Bgr, Byte>> inQueue, Queue<Image<Bgr, Byte>> outQueue, Object lockObject)
{
this.screen = screen;
this.capture = capture;
this.inQueue = inQueue;
this.outQueue = outQueue;
this.lockObject = lockObject;
}
public void produce()
{
while (true)
{
lock (lockObject)
{
inQueue.Enqueue(capture.QueryFrame());
if (inQueue.Count == 1)
{
Monitor.PulseAll(lockObject);
}
if (outQueue.Count > 0)
{
screen.Image = outQueue.Dequeue();
}
}
frameCounter++;
}
}
}
There are different 'Consumers' who take an image from the inQueue, do some processing, and add them to the outQueue:
public class Consumer
{
Queue<Image<Bgr, Byte>> inQueue;
Queue<Image<Bgr, Byte>> outQueue;
Object lockObject;
string name;
Image<Bgr, Byte> image;
public Consumer(Queue<Image<Bgr, Byte>> inQueue, Queue<Image<Bgr, Byte>> outQueue, Object lockObject, string name)
{
this.inQueue = inQueue;
this.outQueue = outQueue;
this.lockObject = lockObject;
this.name = name;
}
public void consume()
{
while (true)
{
lock (lockObject)
{
if (inQueue.Count == 0)
{
Monitor.Wait(lockObject);
continue;
}
image = inQueue.Dequeue();
}
// Do some heavy processing with the image
lock (lockObject)
{
outQueue.Enqueue(image);
}
}
}
}
Rest of the important code is this section:
private void Form1_Load(object sender, EventArgs e)
{
Consumer[] c = new Consumer[consumerCount];
Thread[] t = new Thread[consumerCount];
Object lockObj = new object();
Queue<Image<Bgr, Byte>> inQueue = new Queue<Image<Bgr, Byte>>();
Queue<Image<Bgr, Byte>> outQueue = new Queue<Image<Bgr, Byte>>();
p = new Producer(screen1, capture, inQueue, outQueue, lockObj);
for (int i = 0; i < consumerCount; i++)
{
c[i] = new Consumer(inQueue, outQueue, lockObj, "c_" + Convert.ToString(i));
}
for (int i = 0; i < consumerCount; i++)
{
t[i] = new Thread(c[i].consume);
t[i].Start();
}
Thread pt = new Thread(p.produce);
pt.Start();
}
The parallelisation actually works fine, I do get a linear speed increase with each added thread (up to a certain point of course). The problem is that I get artifacts in the output, even if running only one thread. The artifacts look like part of the picture is not in the right place.
Example of the artifact (this is without any processing to keep it clear, but the effect is the same)
Any ideas what causes this?
Thanks

Displaimer: This post isn't supposed to fully describe an answer, but instead give some hints on why the artifact is being shown.
A quick analysis show that the the actifact is, in fact, a partial, vertically mirrored snippet of a frame. I copied it, mirrored, and placed it back over the image, and added an awful marker to show its placement:
Two things immediately come to attention:
The artifact is roughly positioned on the 'correct' place it would be, only that the position is also vertically mirrored;
The image is slightly different, indicating that it may belong to a different frame.
It's been a while since I played around with raw capture and ran into a similar issue, but I remember that depending on how the driver is implemented (or set up - this particular issue happened when setting a specific imaging device for interlaced capture) it may fill its framebuffer alternating between 'top-down' and 'bottom-up' scans - as soon as the frame is full, the 'cursor' reverts direction.
It seems to me that you're running into a race condition/buffer underrun situation, where the transfer from the framebuffer to your application is happening before the full frame is transferred by the device.
In that case, you'd receive a partial image, and the area still not refreshed would show a bit of the previously transferred frame.
If I'd have to bet, I'd say that the artifact may appear on sequential order, not on the same position but 'fluctuating' on a specific direction (up or down), but always as a mirrored bit.

Well, I think the problem is here . The section of code is not guarantee that you will be access by one thread in here between two queue. The image is pop by inQueue is not actually received in order in outQueue
while (true)
{
lock (lockObject)
{
if (inQueue.Count == 0)
{
Monitor.Wait(lockObject);
continue;
}
image = inQueue.Dequeue();
}
// Do some heavy processing with the image
lock (lockObject)
{
outQueue.Enqueue(image);
}
}

Similar to #OnoSendai, I'm not trying to solve the exact problem as stated. I would have to write an app and I just don't have the time. But, the two things that I would change right away would be to use the ConcurrentQueue class so that you have thread-safety. And, I would use the Task library functions in order to create parallel tasks on different processor cores. These are found in the System.Net and System.Net.Task namespaces.
Also, vertically flipping a chunk like that looks like more than an artifact to me. If it also happens when executing in a single thread as you mentioned, then I would definitely re-focus on the "heavy processing" part of the equation.
Good luck! Take care.

You may have two problems:
1) parallism doesn't ensure that images are added to the out queue in the right order. I imagine that displaying image 8 before image 6 and 7 can produce some artifacts. In consumer thread, you have to wait previous consumer have posted its image to the out queue to post next image. Tasks can help greatly for that because of their inherent synchronisation mecanism.
2) You may also have problems in the rendering code.

Out of memory with multi images in one picturebox

I have a problem with out of memory when I'm trying load a few images into one picturebox.
public void button2_Click(object sender, EventArgs e)
{
FolderBrowserDialog dialog = new FolderBrowserDialog();
dialog.ShowDialog();
string selected = dialog.SelectedPath;
string[] imageFileList = Directory.GetFiles(selected);
int iCtr = 0,zCtr = 0;
foreach(string imageFile in imageFileList)
{
if (Image.FromFile(imageFile) != null)
{
Image.FromFile(imageFile).Dispose();
}
PictureBox eachPictureBox = new PictureBox();
eachPictureBox.Size = new Size(100,100);
// if (iCtr % 8 == 0)
//{
// zCtr++;
// iCtr = 0;
//}
eachPictureBox.Location = new Point(iCtr * 100 + 1, 1);
eachPictureBox.Image = Image.FromFile(imageFile);
iCtr++;
panel1.Controls.Add(eachPictureBox);
}
}`enter code here`

if (Image.FromFile(imageFile) != null)
{
Image.FromFile(imageFile).Dispose();
}
Bad. You're loading the image from the file, checking to see if the result is null...then loading it again into a new result so that you can dispose it. While the latter portion is silly, it isn't harmful. The first portion is, however, as the resulting Image is never properly disposed of (if/when the GC collects it, the finalizer on the Image type should dispose of the unmanaged resources, but this is not a wise thing to rely on).
Incidentally, Image.FromFile will never return null. If it cannot read the image, then it will throw an OutOfMemoryException.
The code also appears to do nothing, since there's no else block and nothing meaningful is done in the if block.
My guess is that your OutOfMemoryException is coming from the fact that one or more of the files in that directory is stored in a corrupted or unsupported image format, or isn't an image at all.
Try replacing the code in your foreach with this:
try
{
Image image = Image.FromFile(imageFile);
PictureBox eachPictureBox = new PictureBox();
eachPictureBox.Size = new Size(100,100);
eachPictureBox.Location = new Point(iCtr * 100 + 1, 1);
eachPictureBox.Image = Image.FromFile(imageFile);
iCtr++;
panel1.Controls.Add(eachPictureBox);
}
catch(OutOfMemoryException) { } // skip the file

The picture box internally holds a reference to the bitmap that you place in it. Unless you get rid of the picture box, it's holding a reference to every bitmap you load into it.

Something you have to consider that regardless of the type of picture stored on disk, when you open it for display the picture will become a bitmap and require 4 bytes per displayed pixel.
Your code seems to suggest an attempt at a thumbnail operation. You are in fact loading 70 files into memory and regardless of the display size, in memory they will be very large.
For example let's say you have 70 jpegs at 32bit color depth and say 1920x1080 pixels in size. Your memory requirement to load that many images all at once then is:
70 pics x 1920 pixels x 1080 pixels x 4 bytes/pixel = 580,608,000 bytes!
And that's a fairly low estimate.
You could consider loading many fewer pictures or trying for a real thumbnailing solution.