I'm working on a small personal application that should read some text (2 sentences at most) from a really simple Android screenshot. The text is always the same size, same font, and in approx. the same location. The background is very plain, usually a few shades of 1 color (think like bright orange fading into a little darker orange). I'm trying to figure out what would be the best way (and most importantly, the fastest way) to do this.
My first attempt involved the IronOcr C# library, and to be fair, it worked quite well! But I've noticed a few issues with it:
It's not 100% accurate
Despite having a community/trial version, it sometimes throws exceptions telling you to get a license
It takes ~400ms to read a ~600x300 pixel image, which in the case of my simple image, I consider to be rather long
As strange as it sounds, I have a feeling that libraries like IronOcr and Tesseract may just be too advanced for my needs. To improve speeds I have even written a piece of code to "treshold" my image first, making it completely black and white.
My current IronOcr settings look like this:
ImageReader = new AdvancedOcr()
{
CleanBackgroundNoise = false,
EnhanceContrast = false,
EnhanceResolution = false,
Strategy = AdvancedOcr.OcrStrategy.Fast,
ColorSpace = AdvancedOcr.OcrColorSpace.GrayScale,
DetectWhiteTextOnDarkBackgrounds = true,
InputImageType = AdvancedOcr.InputTypes.Snippet,
RotateAndStraighten = false,
ReadBarCodes = false,
ColorDepth = 1
};
And I could totally live with the results I've been getting using IronOcr, but the licensing exceptions ruin it. I also don't have $399 USD to spend on a private hobby project that won't even leave my own PC :(
But my main goal with this question is to find a better, faster or more efficient way to do this. It doesn't necessarily have to be an existing library, I'd be more than willing to make my own kind of letter-detection code that would work (only?) for screenshots like mine if someone can point me in the right direction.
I have researched about this topic and the best solution which I could find is Azure cognitive services. You can use Computer vision API to read text from an image. Here is the complete document.
How fast does it have to be?
If you are using C# I recommend the Google Cloud Vision API. You pay per request but the first 1000 per month are free (check pricing here). However, it does require a web request but I find it to be very quick
using Google.Cloud.Vision.V1;
using System;
namespace GoogleCloudSamples
{
public class QuickStart
{
public static void Main(string[] args)
{
// Instantiates a client
var client = ImageAnnotatorClient.Create();
// Load the image file into memory
var image = Image.FromFile("wakeupcat.jpg");
// Performs label detection on the image file
var response = client.DetectText(image);
foreach (var annotation in response)
{
if (annotation.Description != null)
Console.WriteLine(annotation.Description);
}
}
}
}
I find it works well for pictures and scanned documents so it should work perfectly for your situation. The SDK is also available in other languages too like Java, Python, and Node
I'm using NTwain to scan documents into memory and I have it all working except for one part: When trying to set the size of the page to scan, it scans the entire width and height, rather than just the specified size.
I went and examined the details of NTwain's capabilities on the current source, and found the the ICapSupportedSizes was not supported for any action.
Here's how I'm setting the capabilities (this is on an open, valid source)
_twain.CurrentSource.Capabilities.ICapXResolution.SetValue(new TWFix32() { Whole = 600 });
_twain.CurrentSource.Capabilities.ICapYResolution.SetValue(new TWFix32() { Whole = 600 });
_twain.CurrentSource.Capabilities.ICapPixelType.SetValue(PixelType.BlackWhite);
_twain.CurrentSource.Capabilities.ICapSupportedSizes.SetValue(SupportedSize.USLegal);
_twain.CurrentSource.Capabilities.CapDuplexEnabled.SetValue(BoolType.False);
UPDATE:
I've found out that none of the settings are actually working. I set it as black and white, even if it shows black and white in the settings, it displays in color. Doesn't matter what DPI I set it at either as it defaults to 300 no matter what. I've updated it to grab out the source and use that to change the settings and call Enable, but it still doesn't work.
Any help is appreciated.
Thanks!
Enviornment Information
.NET Framework 4.6.1
Win Forms
C#
NTwain
Scanner: Canon Flatbed Scanner Unit 102
You will need to use the DGImage.ImageLayout property to set the page size in a generally-supported way.
For example:
var ds = _twain.CurrentSource;
ds.Capabilities.ICapUnits.SetValue(Unit.Inches);
ds.DGImage.ImageLayout.Get(out TWImageLayout imageLayout);
imageLayout.Frame = new TWFrame
{
Left = 0,
Right = pageWidthInInches,
Top = 0,
Bottom = pageHeightInInches
};
ds.DGImage.ImageLayout.Set(imageLayout);
I have having alot of issues loading in textures into my simple game. First off, I am able to load in a texture when im inside of "Game1.cs". However, I am currently trying to create a level. So I want to load in all the pictures in the Level class.
public Level(IServiceProvider _serviceProvider)
{
content = new ContentManager(_serviceProvider, "Content");
mNrOfTextures = 3;
mTextures[] = new Texture2D[mNrTextures];
mTextures[0] = Content.Load<Texture2D>("sky");
//And then more textures and other stuff..
}
But the program can never find the file sky. I dont really get any useful error messages and im moving away from any tutorials currently. Can anyone point me into the right direction?
Full path to file: C:\c++\ProjIV\ProjIV\ProjIVContent\
I personally just pass my ContentManager to my level class, instead of passing the service provider as others do.
In this case, you need to use your local content instance, not the static Content
mTextures[0] = content.Load<Texture2D>("sky");
EDIT: I see this did not work, can you attach a picture of your solution layout with the content?
I'm trying to convert a sample from objective C to Monotouch, and I have run into some difficulties.
Basically I want to read a video file, and decode the frames one by one to an opengl texture.
The key to doing this is to use the AVAssetReader, but I am not sure how to set this up properly in Monotouch.
This is my code:
AVUrlAsset asset=new AVUrlAsset(NSUrl.FromFilename(videoFileName),null);
assetReader=new AVAssetReader(asset,System.IntPtr.Zero);
AVAssetTrack videoTrack=asset.Tracks[0];
NSDictionary videoSettings=new NSDictionary();
NSString key = CVPixelBuffer.PixelFormatTypeKey;
NSNumber val=0x754b9d0; //NSNumber* value = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA]; - Had to hardcode constant as it is not defined in Monotouch?
videoSettings.SetNativeField(key,val);
//**The program crashes here:
AVAssetReaderTrackOutput trackOutput=new AVAssetReaderTrackOutput(videoTrack,videoSettings);
assetReader.AddOutput(trackOutput);
assetReader.StartReading();
The program crashes on the line indicated above, with an invalid argument exception, indicating that the content of the NSDictionary is not in the right format? I have checked the video file, and it loads well, "asset" contains valid information about the video.
This is the original Objective C code:
NSString* key = (NSString*)kCVPixelBufferPixelFormatTypeKey;
NSNumber* value = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA];
NSDictionary* videoSettings = [NSDictionary dictionaryWithObject:value forKey:key];
AVAssetReaderTrackOutput *trackOutput = [AVAssetReaderTrackOutput assetReaderTrackOutputWithTrack:videoTrack outputSettings:videoSettings];
[_assetReader addOutput:trackOutput];
[_assetReader startReading];
I'm not that into Objective C, so any help is appreciated.
EDIT: I used the code suggested below
var videoSettings = NSDictionary.FromObjectAndKey (
new NSNumber ((int) MonoTouch.CoreVideo.CVPixelFormatType.CV32BGRA),
MonoTouch.CoreVideo.CVPixelBuffer.PixelFormatTypeKey);
And the program no longer crashes. By using the following code:
CMSampleBuffer buffer=assetReader.Outputs[0].CopyNextSampleBuffer();
CVImageBuffer imageBuffer = buffer.GetImageBuffer();
I get the image buffer which should contain the next frame in the video file. By inspecting the imageBuffer object, I find it has valid data such as the width and height, matching that of the video file.
However, the imageBuffer BaseAddress is always 0, which indicates the image has no data? I tried to do this as a test:
CVPixelBuffer buffer=(CVPixelBuffer)imageBuffer;
CIImage image=CIImage.FromImageBuffer(buffer);
And image is always returned as null. Does this mean the actual image data is not present, and my imageBuffer object only contains the frame header info?
And if so, is this a bug in Monotouch, or am I setting this up wrong?
I had an idea that I may need to wait for the image data to be ready, but in that case, I do not know how either. Pretty stuck now...
You need to create the NSDictionary like this:
var videoSettings = NSDictionary.FromObjectAndKey (
new NSNumber ((int) MonoTouch.CoreVideo.CVPixelFormatType.CV32BGRA),
MonoTouch.CoreVideo.CVPixelBuffer.PixelFormatTypeKey);
SetNativeField is something completely different (you're setting the field named CVPixelBuffer.PixelFormatTypeKey to 0x754b9d0, not adding a key/value pair to the dictionary).
[NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA]; - Had to hardcode constant as it is not defined in Monotouch?
You should be able to replace this with:
CVPixelFormatType.CV32BGRA
Note that MonoTouch defines this value as 0x42475241 which differs from yours. That could be your error. If not I suggest you to make a small, self contained, test case and attach it to a bug report on http://bugzilla.xamarin.com and we'll have a look at it.
A link to the objective-c sample, if available, would be helpful too (here to an update to your question or on the bug report).
After several days of tracking down bizarre GDI+ errors, I've stumbled across this little gem on MSDN:
Classes within the System.Drawing namespace are not supported for use within a Windows or ASP.NET service. Attempting to use these classes from within one of these application types may produce unexpected problems, such as diminished service performance and run-time exceptions.
I don't know whether "ASP.NET service" means "web application" in this context, but "diminished service performance" certainly seems to cover the random assortment of "A generic error occurred in GDI+" and "Out of memory" errors that my app is throwing - intermittent, non-reproducible errors reading and writing JPEG images that - in many cases - were actually created by System.Drawing.Imaging in the first place.
So - if GDI+ can't read and write JPEG files reliably in a Web app, what should I be using instead?
I want users to be able to upload images (JPEG required, other formats nice-to-have), resample them reliably, and display useful error messages if anything goes wrong. Any ideas? Are the System.Media namespaces from WPF worth considering?
EDIT: Yeah, I know GDI+ works "most of the time". That's not good enough, because when it fails, it does so in a way that's impossible to isolate or recover from gracefully. I am not interested in examples of GDI+ code that works for you: I am looking for alternative libraries to use for image processing.
There is an excellent blog post including C# code about using the ImageMagick graphics library through Interop over at TopTen Software Blog. This post deals specifically with running ASP.net on linux under mono; however, the C# code should be perfectly copy-paste-able, the only thing you'll need to change is the Interop attributes if you are running under windows referencing a window binary (DLL).
ImageMagick® is a software suite to create, edit, compose, or convert
bitmap images. It can read and write images in a variety of formats
(over 100) including DPX, EXR, GIF, JPEG, JPEG-2000, PDF, PhotoCD,
PNG, Postscript, SVG, and TIFF. Use ImageMagick to resize, flip,
mirror, rotate, distort, shear and transform images, adjust image
colors, apply various special effects, or draw text, lines, polygons,
ellipses and Bézier curves.
There is also an ImageMagick .Net development project on codeplex that wraps up everything for you. But it doesn't show active development since 2009, so it may be lagging behind the current ImageMagick library version. For a small trivial resizing routine, I'd probably stick with the interop. You just need to watch your implementation carefully for your own memory leak or unreleased resources (the library itself is well tested and vetted by the community).
The library is free and open source. The Apache 2 license appears to be compatible with both personal and commercial purposes. See ImageMagick License Page.
The library is totally cross platform and implements many powerful image handling and transformation routines that are not found in GDI+ (or not implemented under mono) and has a good reputation as an alternative for ASP.net image processing.
Update: Looks like there is an updated version of a .NET wrapper here: http://magick.codeplex.com/
Yes, use the WPF System.Windows.Media classes. Being fully managed they don't suffer the same problems as the GDI stuff.
Here's an excerpt from some MVC code I use to render gradients, to give you an idea how to get from a WPF Visual to a PNG:
using System;
using System.IO;
using System.Web.Mvc;
using System.Windows;
using System.Windows.Media;
using System.Windows.Media.Imaging;
namespace MyMvcWebApp.Controllers
{
public class ImageGenController : Controller
{
// GET: ~/ImageGen/Gradient?color1=red&color2=pink
[OutputCache(CacheProfile = "Image")]
public ActionResult Gradient(Color color1, Color color2, int width = 1, int height = 30, double angle = 90)
{
var visual = new DrawingVisual();
using (DrawingContext dc = visual.RenderOpen())
{
Brush brush = new LinearGradientBrush(color1, color2, angle);
dc.DrawRectangle(brush, null, new Rect(0, 0, width, height));
}
return new FileStreamResult(renderPng(visual, width, height), "image/png");
}
static Stream renderPng(Visual visual, int width, int height)
{
var rtb = new RenderTargetBitmap(width, height, 96, 96, PixelFormats.Default);
rtb.Render(visual);
var frame = BitmapFrame.Create(rtb);
var encoder = new PngBitmapEncoder();
encoder.Frames.Add(frame);
var stream = new MemoryStream();
encoder.Save(stream);
stream.Position = 0;
return stream;
}
}
}
You can find a very good article from a Microsoft Employee here: Resizing images from the server using WPF/WIC instead of GDI+ that proposes to use WPF instead of GDI+. It's more about thumbnailing but it's overall the same issues.
Anyway, at the end it states this:
I contacted the WPF team to have the final word on whether this is
supported. Unfortunately, it's not, and the documentation is being
updated accordingly. I apologize about any confusion this may have
caused. We're looking at ways to make that story more acceptable in
the future.
So WPF is also unsupported in web apps and still is I believe :-S
ImageSharp
ImageSharp is an open-source cross-platform 2D graphics library. It's written in C# on top of the new .NET Standard, with no dependency on any OS-specific API.
It's currently still in pre-release on MyGet (you'll have to add the package source in the VS options or a NuGet.config file), but we are already using it with some very positive results.
Most of the issues I have read about pertain to resources not being disposed properly.
I have used variants of this code time and time again with no issues from web applications:
public void GenerateThumbNail(HttpPostedFile fil, string sPhysicalPath,
string sOrgFileName,string sThumbNailFileName,
System.Drawing.Imaging.ImageFormat oFormat, int rez)
{
try
{
System.Drawing.Image oImg = System.Drawing.Image.FromStream(fil.InputStream);
decimal pixtosubstract = 0;
decimal percentage;
//default
Size ThumbNailSizeToUse = new Size();
if (ThumbNailSize.Width < oImg.Size.Width || ThumbNailSize.Height < oImg.Size.Height)
{
if (oImg.Size.Width > oImg.Size.Height)
{
percentage = (((decimal)oImg.Size.Width - (decimal)ThumbNailSize.Width) / (decimal)oImg.Size.Width);
pixtosubstract = percentage * oImg.Size.Height;
ThumbNailSizeToUse.Width = ThumbNailSize.Width;
ThumbNailSizeToUse.Height = oImg.Size.Height - (int)pixtosubstract;
}
else
{
percentage = (((decimal)oImg.Size.Height - (decimal)ThumbNailSize.Height) / (decimal)oImg.Size.Height);
pixtosubstract = percentage * (decimal)oImg.Size.Width;
ThumbNailSizeToUse.Height = ThumbNailSize.Height;
ThumbNailSizeToUse.Width = oImg.Size.Width - (int)pixtosubstract;
}
}
else
{
ThumbNailSizeToUse.Width = oImg.Size.Width;
ThumbNailSizeToUse.Height = oImg.Size.Height;
}
Bitmap bmp = new Bitmap(ThumbNailSizeToUse.Width, ThumbNailSizeToUse.Height);
bmp.SetResolution(rez, rez);
System.Drawing.Image oThumbNail = bmp;
bmp = null;
Graphics oGraphic = Graphics.FromImage(oThumbNail);
oGraphic.CompositingQuality = CompositingQuality.HighQuality;
oGraphic.SmoothingMode = SmoothingMode.HighQuality;
oGraphic.InterpolationMode = InterpolationMode.HighQualityBicubic;
Rectangle oRectangle = new Rectangle(0, 0, ThumbNailSizeToUse.Width, ThumbNailSizeToUse.Height);
oGraphic.DrawImage(oImg, oRectangle);
oThumbNail.Save(sPhysicalPath + sThumbNailFileName, oFormat);
oImg.Dispose();
}
catch (Exception ex)
{
Response.Write(ex.Message);
}
}
You may have a look at http://gd-sharp.sourceforge.net/ which is a wrapper for the GD library. I haven't tested it but it seems promising.
I've had good behavior from the Cairo library (http://www.cairographics.org) in an ASP.Net webserver environment. I actually moved to cairo from WPF due to WPF's poor memory usage model for web-based stuff.
WPF actually tends to run your worker process out of memory. None of the WPF objects implement IDisposable, and many of them reference unmanaged memory that's only freed via a finalizer. Heavy use of WPF (especially if your server is significantly CPU-taxed) will eventually run you out of memory because your finalizer queue gets saturated. When I was profiling my app, for instance, the finalization queue had upwards of 50,000 objects on it, many of them holding references to unmanaged memory. Cairo has behaved much better for me, and its memory usage pattern has been much more predictable than WPF's.
If you're interested in using cairo, grab the libs from GTK+'s website. They have an x86 as well as an x64 set of binaries.
The only downside is that cairo can't read/write JPG natively; however, you could easily adapt WPF's stuff for reading/writing JPG and do the resampling/scaling/drawing/whatever else using Cairo.
Aspose.Drawing is a drop-in replacement for System.Drawing that is fully managed and can be safely used in web applications. (I'm one of the developers.)