How to make Tesseract faster in C#? - c#

I put a scaled (2x) and binarized image (text black, everything else white) into tesseract and want to do the OCR as fast as possible:
private OCRResultData getOCRResult(Bitmap image, int minConf)
{
// Prepare input image for tesseract
Pix imageAsPix = PixConverter.ToPix(image);
// Process image by engine
Page page = this.engine.Process(imageAsPix, PageSegMode.SparseText);
string result = page.GetTsvText(0);
// Get data (split string into array of arrays to make information easier to acces)
string[][] data = getDataArray(result);
// Create result object and return it (class defiend by myself)
OCRResultData resultData = new OCRResultData();
resultData.confidences = getConfidences(data, minConf);
resultData.boundingBoxes = getBoundingBoxes(data, minConf);
resultData.detectedText = getDetectedText(data, minConf);
resultData.labelCoordinates = getLabelCoordsList(data, minConf);
page.Dispose();
return resultData;
}
Problem:
The best method I found for getting information like the detected string, its confidence, the bounding box coords, etc. was page.GetTsvText(0). Now while the processing is really fast (this.engine.Process(imageAsPix, PageSegMode.SparseText) takes like 30 ms for a 3000px X 3000px image), the page.GetTsvText(0) method needs about 1800 ms, which is too slow for my needs.
Is there a way to get it faster?
As a last option, I would try to introduce a string property and call the GetTsvText(0) in an asynchronous method at programm start and set it that way; In the case the user does nothing in the first 1800ms after loading the image, he wouldn't feel the delay anymore when pressing the OCR button (the code uses a WinForms GUI);

Related

Detecting if a Face is Upside down with Dlib.Net(FaceRecognition.Net)

Basically i'm trying to check if a face is upside down in an image using this library https://github.com/takuya-takeuchi/FaceRecognitionDotNet.
Take example of the image below
This is an image that is successfully detected using the FaceRecognition.Net library.The image is upside down.I have marked all face landmarks in the image with a blue ellipses.
This is the approach i follow
// Finding faceparts
var faceparts = dparameters._FaceRecognition.FaceLandmark(dparameters.FCImage);
// Drawing Ellipses over all points got from faceparts
foreach(var facepart in faceparts) {
foreach(var mypoint in facepart.Values) {
foreach(var x in mypoint) {
tempg.DrawEllipse(Pens.Blue, x.Point.X, x.Point.Y, 2, 2);
}
}
}
Now i'm checking if the image is rotated by comparing maximum Y coordinates of the lip and eyepoints
var temp = faceparts.FirstOrDefault();
IEnumerable < FacePoint > lippoints;
temp.TryGetValue(FacePart.BottomLip, out lippoints);
IEnumerable < FacePoint > eyepoints;
temp.TryGetValue(FacePart.LeftEye, out eyepoints);
var lippoint = lippoints.Max(r => r.Point.Y);
var topeyepoint = eyepoints.Max(r => r.Point.Y);
if (lippoint > topeyepoint) {
bool isinverted = true;
} else {
bool isinverted = false;
}
The issue is that even when the image is not upside down, the eyecoordinate is less than the face coordinate.This is because a false face is detected as outlined in the image.How to get over this issue?
It looks like this library does not provide a confidence ratio for the results. Otherwise, I would suggest to try both the input and its flipped copy and take the one with higher confidence before doing the "eye over mouth" check.
So maybe what could help is:
using the CNN model, in the original library it is called by
face_locations = face_recognition.face_locations(image, number_of_times_to_upsample=0, model="cnn")
in the C# port it should be
_FaceRecognition.FaceLocations(image, 0, Model.Cnn)
That should give you a more accurate face bounding box which you can then compare with the bounding box of the landmarks. If you do the same for a flipped copy of the image, you can "emulate" the confidence I mentioned earlier and assume the orientation where the boxes match better. Then you can identify the orientation by the "eyes over mouth" test.
as far as I noticed the library does not provide pre-trained data, so in order to use the Cnn model you need to train it by yourself. Selection of the dataset for training is of course very important. If you already performed the training, more/better training data might improve the accuracy.

How to display real time updates to an image in Unity?

I am trying to create an application that generates a bitmap image every frame based on user actions and have it display that image to the screen. I would like the application to also be able to update that image in unity in real time as soon as the user makes another action.
I have created an application that does this and it works. However, it is veryyyy slow. My Update() method is attached below.
My idea was:
Capture user data (mouse location).
Convert that data into a special signal format that another program recognizes.
Have that program return a bitmap image.
Use that bitmap as a texture and update the existing texture with the new image.
Code:
UnityEngine.Texture2D oneTexture;
Bitmap currentBitmap;
private int frameCount = 0;
void Update()
{
// Show mouse position in unity environment
double xValue = Input.mousePosition.x;
double yValue = Screen.height - Input.mousePosition.y;
myPoints = "" + xValue + "," + yValue + Environment.NewLine;
// Show heatmap being recorded.
signals = Program.ConvertStringToSignalsList(myPoints);
currentBitmap = Program.CreateMouseHeatmap(Screen.width, Screen.height, signals);
// Update old heatmap texture.
UpdateTextureFromBitmap();
ri.texture = oneTexture;
ri.rectTransform.sizeDelta = new Vector2(Screen.width, Screen.height);
frameCount++;
// Write points to Database.
StartCoroutine(WriteToDB(xValue, yValue)); // <<<<< Comment out when playback.
}
private void UpdateTextureFromBitmap()
{
// Convert Bitmap object into byte array instead of creating actual
// .bmp image file each frame.
byte[] imageBytes = ImageToBytes(currentBitmap);
BMPLoader loader = new BMPLoader();
BMPImage img = loader.LoadBMP(imageBytes);
// Only initialize the Texture once.
if (frameCount == 0)
{
oneTexture = img.ToTexture2D();
}
else
{
Color32[] imageData = img.imageData;
oneTexture.SetPixels32(imageData);
oneTexture.Apply();
}
}
I was wondering if someone could help me improve the rate at which the image updates to the screen? I know that it is possible to make this program much faster but I am so new to unity and C# that I don't know how to make that happen. Also if there is a completely different way that I should be going about doing this then I am open to that too. Any help would be appreciated. Thanks!
Also, below is a screenshot of the Profiler showing the breakdown of CPU Usage. Currently it looks like every frame is taking about 500ms.

reduce CPU overhead while proccessing Video Stream

I am developing C# WPF Auto Number Plate Recognition Using an OCR.
The Flow is, i am getting a pictures from a video stream MJPEG and this images should be passed to the OCR to get the plate number and other details.
The problem is : the Video stream is producing about 30 Frame/second and the CPU can't handle this much of processing also it will take around 1 Sec to process 1 frame, Also when i will get many frames on the Queue the CPU will be 70% used (Intel I7 4th G).
Can anyone suggest solution and better implementation.
//This is the queue where it will hold the frames
// produced from the video streaming(video_Newfram1)
private readonly Queue<byte[]> _anpr1Produces = new Queue<byte[]>();
//I am using AForg.Video to read the MJPEG Streaming
//this event will be triggered for every frame
private void video_NewFrame1(object sender, NewFrameEventArgs eventArgs)
{
var frameDataAnpr = new Bitmap(eventArgs.Frame);
AnprCam1.Source = GetBitmapimage(frameDataAnpr);
//add current fram to the queue
_anpr1Produces.Enqueue(imgByteAnpr);
//this worker is the consumer that will
//take the frames from the queue to the OCR processing
if (!_workerAnpr1.IsBusy)
{
_workerAnpr1.RunWorkerAsync(imgByteAnpr);
}
}
//This is the consumer, it will take the frames from the queue to the OCR
private void WorkerAnpr1_DoWork(object sender, DoWorkEventArgs e)
{
while (true)
{
if (_anpr1Produces.Count <= 0) continue;
BgWorker1(_anpr1Produces.Dequeue());
}
}
//This method will process the frames that sent from the consumer
private void BgWorker1(byte[] imageByteAnpr)
{
var anpr = new cmAnpr("default");
var objgxImage = new gxImage("default");
if (imageByteAnpr != null)
{
objgxImage.LoadFromMem(imageByteAnpr, 1);
if (anpr.FindFirst(objgxImage) && anpr.GetConfidence() >= Configs.ConfidanceLevel)
{
var vehicleNumber = anpr.GetText();
var vehicleType = anpr.GetType().ToString();
if (vehicleType == "0") return;
var imagename = string.Format("{0:yyyy_MMM_dd_HHmmssfff}", currentDateTime) + "-1-" +
vehicleNumber + ".png";
//this task will run async to do the rest of the process which is saving the vehicle image, getting vehicle color, storing to the database ... etc
var tsk = ProcessVehicle("1", vehicleType, vehicleNumber, imageByteAnpr, imagename, currentDateTime, anpr, _anpr1Produces);
}
else
{
GC.Collect();
}
}
}
What you should do is this:
First, figure out if a frame is worth processing or not. If you're using a compressed video stream, you can usually quickly read the frame's compressed size. It stores the difference between the current frame and the previous one.
When it's small, not much changed (i.e: no car drove by).
That's a low-tech way to do motion detection, without even having to decode a frame, and it should be extremely fast.
That way, you can probably decide to skip 80% of the frames in a couple of milliseconds.
Once and a while you'll get frames that need processing. Make sure that you can buffer enough frames so that you can keep recording while you're doing your slow processing.
The next thing to do is find a region of interest, and focus on those first. You could do that by simply looking at areas where the color changed, or try to find rectangular shapes.
Finally, one second of processing is SLOW if you need to process 30 fps. You need to make things faster, or you'll have to build up a gigantic buffer, and hope that you'll ever catch up if it's busy on the road.
Make sure to make proper use of multiple cores if they are available, but in the end, knowing which pieces of the image are NOT relevant is the key to faster performance here.

How can I mix output images from two different methods in C#?

I have a problem as follow,
I have an application based on C# which has two lines of image production:
Line-1 generates a series of frames per second based on some mathematical calculation. Each frame (image) is consist of black and white pixels which they form a pattern.
Line-2 is generating another series of frames which are made based on random noise generator. So they are just different frames which contain only noise!
Now, my question is I need to mix this frames from these two lines together randomly. By meaning that, for example I need to select 5 frames from line-1 and 3 frames from line-2 and so on mixing them together randomly. This mixing procedure will change randomly.
My own solution is if I store images from line-1 production in a separate list, and line-2 production in another list so I can have a flag to select from these two lists randomly. But as a fact that this images are generating real time I don’t know if that solution works or not. Does anyone have any alternative solution for my question?;)
I would use queues (Queue(T) or ConcurrentQueue(T)) to store the images. Assuming you use a thread to fill each queue and one to consume from both queues.
Example:
private ConcurrentQueue<Bitmap> line1 = new ConcurrentQueue<Bitmap>();
private ConcurrentQueue<Bitmap> line2 = new ConcurrentQueue<Bitmap>();
private Random randomGenerator = new Random();
//thread 1
private void FillLine1()
{
//your line 1 image producation code
Bitmap yourCalculatedBitmap = new Bitmap(100,100);
line1.Enqueue(yourCalculatedBitmap);
}
//thread 2
private void FillLine2()
{
//your line 2 image production code
Bitmap yourCalculatedBitmap = new Bitmap(100,100);
line1.Enqueue(yourCalculatedBitmap);
}
//thread 3
private Bitmap RandomImageSelection()
{
Bitmap image;
if (randomGenerator.Next(2) == 0 && line1.TryDequeue(out image))
{
return image;
}
if (line2.TryDequeue(out image))
{
return image;
}
return null;
}

How can I render text on a WriteableBitmap on a background thread, in Windows Phone 7?

I am trying to render text on a bitmap in a Windows Phone 7 application.
Code that looks more or less like the following would work fine when it's running on the main thread:
public ImageSource RenderText(string text, double x, double y)
{
var canvas = new Canvas();
var textBlock = new TextBlock { Text = text };
canvas.Children.Add(textBloxk);
Canvas.SetLeft(textBlock, x);
Canvas.SetTop(textBlock, y);
var bitmap = new WriteableBitmap(400, 400);
bitmap.Render(canvas, null);
bitmap.Invalidate();
return bitmap;
}
Now, since I have to render several images with more complex stuff, I would like to render the bitmap on a background thread to avoid an unresponsive UI.
When I use a BackgroundWorker to do so, the constructor for TextBlock throws an UnauthorizedAccessException claiming that this is an invalid cross-thread access.
My question is: how can I render text on a bitmap without blocking the UI?
Please don't suggest using a web service to do the rendering. I need to render a large number of images and the bandwidth cost is not acceptable for my needs, and the ability to work offline is a major requirement.
The solution doesn't necessarily has to use WriteableBitmap or UIElements, if there is another way to render text.
EDIT
Another thought: does anyone know if it should be possible to run a UI message loop in another thread, and then have that thread do the work? (instead of using a BackgroundWorker)?
EDIT 2
To consider alternatives to WriteableBitmap, the features I need are:
Draw a background image.
Measure the width and height of a 1-line string, given a font familiy and size (and preferably style). No need for word wrapping.
Draw a 1-line string, with given font family, size, style, at a given coordinate.
Text rendering should support a transparent background. I.e. you should see the background image between the characters.
This method copies the letters from an pre-made image instead of using TextBlock, it's based on my answer to this question. The main limitation is requiring a different image for each font and size needed. A size 20 Font needed about 150kb.
Using SpriteFont2 export the font and the xml metrics file in the sizes you require. The code assumes they're named "FontName FontSize".png and "FontName FontSize".xml add them to your project and set the build action to content. The code also requires WriteableBitmapEx.
public static class BitmapFont
{
private class FontInfo
{
public FontInfo(WriteableBitmap image, Dictionary<char, Rect> metrics, int size)
{
this.Image = image;
this.Metrics = metrics;
this.Size = size;
}
public WriteableBitmap Image { get; private set; }
public Dictionary<char, Rect> Metrics { get; private set; }
public int Size { get; private set; }
}
private static Dictionary<string, List<FontInfo>> fonts = new Dictionary<string, List<FontInfo>>();
public static void RegisterFont(string name,params int[] sizes)
{
foreach (var size in sizes)
{
string fontFile = name + " " + size + ".png";
string fontMetricsFile = name + " " + size + ".xml";
BitmapImage image = new BitmapImage();
image.SetSource(App.GetResourceStream(new Uri(fontFile, UriKind.Relative)).Stream);
var metrics = XDocument.Load(fontMetricsFile);
var dict = (from c in metrics.Root.Elements()
let key = (char) ((int) c.Attribute("key"))
let rect = new Rect((int) c.Element("x"), (int) c.Element("y"), (int) c.Element("width"), (int) c.Element("height"))
select new {Char = key, Metrics = rect}).ToDictionary(x => x.Char, x => x.Metrics);
var fontInfo = new FontInfo(new WriteableBitmap(image), dict, size);
if(fonts.ContainsKey(name))
fonts[name].Add(fontInfo);
else
fonts.Add(name, new List<FontInfo> {fontInfo});
}
}
private static FontInfo GetNearestFont(string fontName,int size)
{
return fonts[fontName].OrderBy(x => Math.Abs(x.Size - size)).First();
}
public static Size MeasureString(string text,string fontName,int size)
{
var font = GetNearestFont(fontName, size);
double scale = (double) size / font.Size;
var letters = text.Select(x => font.Metrics[x]).ToArray();
return new Size(letters.Sum(x => x.Width * scale),letters.Max(x => x.Height * scale));
}
public static void DrawString(this WriteableBitmap bmp,string text,int x,int y, string fontName,int size,Color color)
{
var font = GetNearestFont(fontName, size);
var letters = text.Select(f => font.Metrics[f]).ToArray();
double scale = (double)size / font.Size;
double destX = x;
foreach (var letter in letters)
{
var destRect = new Rect(destX,y,letter.Width * scale,letter.Height * scale);
bmp.Blit(destRect, font.Image, letter, color, WriteableBitmapExtensions.BlendMode.Alpha);
destX += destRect.Width;
}
}
}
You need to call RegisterFont once to load the files then you call DrawString. It uses WriteableBitmapEx.Blit so if your font file has white text and a transparent background alpha is handled correctly and you can recolour it. The code does scale the text if you draw at a size you didn't load but the results aren't good, a better interpolation method could be used.
I tried drawing from a different thread and this worked in the emulator, you still need to create the WriteableBitmap on the main thread. My understanding of your scenario is that you want to scroll through tiles similar to how mapping apps work, if this is the case reuse the old WriteableBitmaps instead of recreating them. If not the code could be changed to work with arrays instead.
I'm not sure if this will fully resolve your issues, but there are 2 tools that I use in my comic book reader (I won't shamelessly plug it here, but I'm tempted.. a hint if you are searching for it.. it is "Amazing"). There are times where I need to stitch together a bunch of images. I use Rene Schulte's (and a bunch of other contributors) WriteableBitmapExtensions (http://writeablebitmapex.codeplex.com/). I have been able to offload rendering/stitching of an image to a background thread and then set the resulting WriteableBitmap as the source of some image on the UI thread.
Another up and comer in this space is the .NET Image Tools (http://imagetools.codeplex.com/). They have a bunch of utilities for saving/reading various image formats. They also have a few of the low levels, and I wish there were an easy way to use both (but there isn't).
All of the above work in WP7.
I guess the major difference is with these tools you won't be using XAML you will be writing directly to your image (so you may need to do size detection of your text and stuff like that).
The very nature of UI elements requires interaction with them on the UI thread. Even if you could create them on a background thread, when you came to try to render them into the WriteableBitmap you'd get a similar exception, and even then if it allowed you to do that, the elements wouldn't actually have a visual representation until they were added into the visual tree. You might need to use a generic image manipulation library instead of using UI elements.
Perhaps you could describe your scenario on a wider basis, we might have a better solution for you :)
First off, are you sure about rendering this as a bitmap? How about generating a Canvas with an image and TextBlock?
I need to render a large number of images
I have a feeling that this generating will kill phone performance. Generally, for bitmap mainupulation, the best way is to use XNA. Some parts of the XNA framework do a great job Silverlight projects. (BTW the refreshed Windows Phone Developer Tools will allow Silverlight and XNA coexist in the same project)
I would step back and think about this feature. Developing something like this for a week and then end up with unacceptable performance would make me a sad panda.
EDIT
As far I understand you need some kind of popup with image as a background and the message.
Make a Canvas with TextBlock but hide it.
<Canvas x:Name="userInfoCanvas" Height="200" Width="200" Visibility="Collapsed">
<Image x:Name="backgroundImage"> </Image>
<TextBlock x:Name="messageTextBlock" Canvas.ZIndex="3> </TextBlock> <!--ZIndex set the order of elements -->
</Canvas>
When you got the new message, show the Canvas to the user (a opacity animation would be nice), when you finish rendering on the background thread.
messageTextBlock.Text = message;
backgroundImage.Source = new BitmapImage(renderedImage);
Obviouslly, here is a problem with update. UIelements can be updated only form the UI Thread, hence update must be queue with Dispatcher
Dispatcher.BeginInvoke(DispatcherPriority.Background, messageUpdate); //messageUpdate is an Action or anthing that can be infered to Delegate
PS. didn't compile, this is more pseudocode.
You can draw on WriteableBitmap in thread, but You have to
create WriteableBitmap in main UI thread
do draw work in background thread
assign BitmapSource in main UI thread
i'll agree with Derek's answer: you're trying to use UI controls without a UI.
If you want to render a bitmap you need to stick to classes for drawing text on bitmaps.
i presume Windows Phone 7 has the .NET Compact Framework.
psudeo-code:
public Bitmap RenderText(string text, double x, double y)
{
Bitmap bitmap = new Bitmap(400, 400);
using (Graphics g = new Graphics(bitmap))
{
using (Font font = SystemFonts....)
{
using (Brush brush = new SolidColorBrush(...))
{
g.DrawString(text, font, brush, new Point(x, y));
}
}
}
return bitmap;
}

Categories