We Keep Coding

How to avoid "adding up" of opacities in SkiaSharp

I'm currently creating a Xamarin.Forms app. One of my pages uses SkiaSharp to allow users to highlight parts of an image in a text marker style (i.e. a yellow brush with low opacity).
This is how the related SKPaint object is defined:
var strokePaint = new SKPaint()
{
Color = Color.FromRgba(255, 255, 0, 100).ToSKColor(),
Style = SKPaintStyle.Stroke,
StrokeWidth = StrokeWidth
};
That's working fine so far, but what bothers me is that the opacity "increases" when I have multiple overlapping paths, until at some point the underlying picture isn't visible anymore.
What could I do to avoid this overlapping? I was thinking about merging all paths into one, but that doesn't seem to work because the user is allowed to change StrokeWidth in between strokes and I didn't see any way of drawing paths with varying width.
I hope any of you guys has some help for me. Any idea is appreciated!

I'm not super familiar with Skia, but I took a look at the documentation for SKPaint, and it looks like it has a BlendMode property. Based on how similar things work in other systems, that should control how colors are combined. You might have to try different values to get the effect you are looking for. Dst, or Modulate look like good candidates. – Bradley Uffner
Thanks for your answer Bradley! I went with the Darken blend mode and set opacity to 255, which creates a very nice effect when highlighting text (only the darker color is visible, so dark text on a light background becomes dark text on a background of my marker color).

Find image within image (object detection)

I do have different images which all have some kind of border around the "real" image. What I would like to achieve is to find the "real" image (size and location in pixels).
For me the challenge is that the border is not always black (can be any kind of black or grey with a lot of noise) and the "real" image (water with shark in this example) can have any combination of color, saturation, ...
Now in general I'm aware of algorithms like Canny, Blob detection, hough lines, ..., but I have just started using them. So far I managed to find the border for a specific image, but as soon as I try to apply the same algorithms and parameters to the next image it doesn't work. My current approach looks like this (pseudo code):
convert to gray CvInvoke.CvtColor(_processedImage, tempMat, CvEnum.ColorConversion.Rgb2Gray)
downsample with CvInvoke.PyrDown(srcImage, targetImage) and CvInvoke.PyrUp(srcImage, targetImage)
blur image with CvInvoke.GaussianBlur(_processedImage, bluredImage, New Drawing.Size(5, 5), 0)
Binarize with CvInvoke.Threshold(_processedImage, blackWhiteImage, _parameters.BinarizeThreshold, 255, CvEnum.ThresholdType.Binary)
Detect Edges with CvInvoke.Canny(_processedImage, imgEdges, 60, 100)
Find Contours with CvInvoke.FindContours(_processedImage, contours, Nothing, CvEnum.RetrType.External, CvEnum.ChainApproxMethod.ChainApproxSimple)
Assume that largest contour is the real image
I already tried different approaches based on for example:
Thresholding saturation channel and bounding box
Thresholding, canny edge and finding contours
Any hint especially on how to find proper parameters (that apply for all images) for algorithms like (adaptive) threshold and canny as well as ideas for improving the processing pipeline would be highly appreciated.

you can try to subtract black image from this image , and you will get the inside image , way to do this:
Use image subtraction to compare images in C# ,

If the border was uniform, this would be easy. Use cv::reduce to find MIN and MAX of each row and column; then count the top,left,bottom,right rows/columns whose MIN and MAX are equal (or very close) to the pixel value in a nearby corner. For sanity, maybe check the border colour is the same on all sides.
In your example the border contains faint red stuff, but a row/column approach might still be a useful way to simplify the problem. Maybe, as Nofar suggests, take an absolute difference with what you think is the background colour; square it, convert to grey, then reduce to Sums of rows and columns. You still need to find edges, but have reduced the data from two dimensions to one.
If there's a large border and lots of noise, maybe iterate: in the second pass, exclude the rows you think comprise the border, from statistics on columns (and vice versa).
EDIT: The above only works for an upright rectangle! If it could be rotated then the row/column projection method won't work. In that case I might go for sum-of-squared differences as above (don't start by converting to grey as it could throw away information), followed by blurring or some morphology, edge detection then some kind of Hough transform to find straight edges.

Eliminating a border effect

This is a work project. I inherited some code using SharpDX (a DirectX layer). One of my tasks is to fix a piece of code where certain image effects are applying to a geometric shape containing a fill. If the filter is applied to the fill itself, it doesn't conform to the edges. I've figured out the code to pull out an excerpt using the Geometry of the object. For various reasons, they want to keep the fill that exists outside of the shape (namely, we have some distortion effects that pull in pixels outside of the shape), so I need to overlay it over the background. The problem I'm running into is that I'm getting this single-pixel border...
Applying the Soft Edge filter to the visible part
The background with the shape cut out
The two composited together in the program
What I'm actually getting
I can't share a good bit of the code, due to parts of it being proprietary, but the mask is a byte array. I'm building it using the following code:
SingleChannelBitmap mask = new SingleChannelBitmap(MaxRequiredPixels.Width, MaxRequiredPixels.Height, 255);
mask.FillShape(new RectangleF(new PointF(0,0), mask.Size), this.Geometry, 0);
255 is the maximum Alpha value (transparent). I invert it to take the slice out of the background. The only thing I can think of is that, when I do the masking, it's not including the outer edge of the Geometry. I'm going to try expanding the mask by one pixel in the crudest way possible (basically, scanning through and taking anything which is 0 transparency and adding a 0 transparency pixel to the left, right, up, and down), but I know there has to be a more elegant solution.
This has to work for the 3D Edge bevel filter as well, so doing an arbitrarily large whitespace probably won't work for me either.

What you describe is essentially the same haloing problem that sometimes occurs with displaying PNG images. The PNG export process from several programs will store a solid color for any portions of the PNG that has zero alpha, instead of the actual color at those pixels. This makes them function similar to other image formats (GIF) which use a specific color to encode transparent pixels. This significantly reduces the size of the file, however, can cause issues when sampling the image.
Your situation is similar. Although the masked pixels have zero alpha, when doing bilinear sampling, you may sample in between pixels, mixing both color and alpha values (unless pixel and texel centers are perfectly aligned). For example, if you have a 100% alpha, white pixel, next to a 0% alpha red pixel, and sample in between both, the result will be a pink pixel at 50% alpha.
There are several possible solutions:
You could extend the borders of the color layer, such that the 0% alpha border has the same color as its non-0% alpha adjacent pixels.
Intentionally line up the pixel and texel centers, although this can be tricky and/or not possible, depending on your requirements (mostly dependent on resolution).
Use 'nearest' sampling, instead of bilinear when displaying the image. This way, you will never blend in a 0% alpha pixel. However, this may also not be desirable, because your image will likely exhibit more aliasing effects.

Blurred picture after applying of shader effect

I've come across strange behavior of pixel shader in WPF.
This problem is 100% reproducible, so I wrote small demo program. You can download source code here.
The root of all evil is tiny class titled MyFrameworkElement:
internal sealed class MyFrameworkElement : FrameworkElement
{
public double EndX
{
get
{
return (double)this.GetValue(MyFrameworkElement.EndXProperty);
}
set
{
this.SetValue(MyFrameworkElement.EndXProperty, value);
}
}
public static readonly DependencyProperty EndXProperty =
DependencyProperty.Register("EndX",
typeof(double),
typeof(MyFrameworkElement),
new FrameworkPropertyMetadata(0d, FrameworkPropertyMetadataOptions.AffectsRender));
protected override void OnRender(DrawingContext dc)
{
dc.DrawLine(new Pen(Brushes.Red, 2), new Point(0, 0), new Point(this.EndX, 100));
dc.DrawLine(new Pen(Brushes.Green, 3), new Point(10, 300), new Point(200, 10));
}
}
As you can see this framework element renders 2 lines: lower line has permanent coordinates but upper line depends on EndX dependency property.
So this framework element is target for pixel shader effect. For simplicity's sake I use grayscale shader effect found here. So I applied GrayscaleEffect to MyFrameworkElement. You can see result, it looks nice.
Until I increase EndX property drastically.
Small line is blurred and big line is fine!
But if I remove grayscale effect, all lines will look as they should.
Can anybody explain what's the reason of this blurring?
Or even better how can I solve this problem?

With a custom pixel shader it has to create an Intermediate Bitmap and then that texture gets sampled by the pixel shader.
You're creating a massive rendering, so your hitting some limitation in the render path.
A quick fix is to clip what you want rendered as follows:
Geometry clip = new RectangleGeometry(new Rect(0,0,this.ActualWidth, this.ActualHeight));
dc.PushClip(clip);
dc.DrawLine(new Pen(Brushes.Red, 2), new Point(0, 0), new Point(this.EndX, 100));
dc.DrawLine(new Pen(Brushes.Green, 3), new Point(200, 10), new Point(10, 300));
dc.Pop();
UPDATE:
One theory is that it's using a filter to scale the bitmap when it exceeds the maximum texture size (which can vary depending on your graphics card architecture)...so it goes through the pixel shader at a different size....then it gets scaled back to original size.
Thus the scaling filter is causing artifacts depending on the content of your bitmap (i.e. horizontal lines and vertical lines survive a scale down and up better than diagonal lines).
.NET 4 changed the default filter it uses for filtering to a lowerquality one...Bilinear, instead of Fant...maybe this impacts the quality that you get too.
http://10rem.net/blog/2010/05/16/more-on-image-resizing-in-net-4-vs-net-35sp1-bilinear-vs-fant
UPDATE2:
This kind of confirms what I was thinking above.
If you use the Windows Performance Toolkit/Suite (part of Windows SDK), then you can see the Video Memory being gobbled up in the orange graph while you increase the slider value because a bigger Intermediate Bitmap texture is being created. It keeps increasing until it hits a limit, then it flatlines...and thats when the pixelation becomes evident.
UPDATE3:
If you set the render mode to the "Software Renderer" (Tier 0) then you can see how it copes with rendering such a large visual - the artifacts start appearing at a different point....presumably because the texture size limit is larger/different to your GPUs. But the artifacts still appear because it's using a Bilinear filter internally.
Trying to use RenderOptions.SetBitmapScalingMode to up the filter to Fant doesn't seem to change the rendering quality in any way (I guess because it isn't honoured when it goes through the custom pixel shader path).
Put this in Application_Startup to see the software renderer results:
RenderOptions.ProcessRenderMode = RenderMode.SoftwareOnly;

Note that image is normally blurred in vertical direction, but is jagge in horizontal.
Since shaders are applied to raster images, not vector, the lines are rasterized into texture. Hardware usually supports textures up to 8198*8192.
In my case the "blurring", as you call it, appears at slider value of 16384. So, my virtualBox virtual graphics card supports up to 16384*16384.
Your limit may differ.
So just keep this value lower than that.
But it's strange that WPF rasterizes whole image, since only small part of it visible.
So there is also another possible reason, that lies inside shader itself, but it is compiled into binary, so i can't check it.
Update:
In my case it looks this way:
Looks like it is filtered vertically but not horizontally.

Ok, I've got this!
I decompiled the library with your grayscale effect and also decompiled WCF PresentationCore library to check why BlurEffect works perfect in the same situation.
And i found that BlurEffect implements abstract method Effect.GetRenderBounds which is absent in GrayscaleEffect. I also noticed that GrayscaleEffect is built against PresentationCore v 3.0.0 where Effect does not have GetRenderBound.
So this is an incompatibility between 3rd and 4th versions of WPF.
There are three ways to fix it:
If you have source code of GrayscaleEffect - add needed methods and compile it against 4.0.0 version of runtime.
You can switch the runtime your application use to version 3.*.
If you don't have sources of GrayscaleEffect but can't use 3rd version of runtime, write wrapper for GrayscaleEffect that inherits Effect (v4) and implements absent methods.
I tried 2nd way and the problem disappeared.

old question, but might be useful for someone having problem with blurring of image after applying Custom ShaderEffect.
Also problem OP mentioned might be releated to scale of rendered content,
I had similar problem with blurring after applying ShaderEffects from WPFShadersLibrary to video, text and any other content within normal window.
What I noticed that that image shifts down by a tiny bit, resulting in "pixel splitting", so I created two new properties for chosen ShaderEffect : XOffset and YOffset, and applied them in HLSL (see code below), then binded to Sliders in XAML :
float2 newPos;
newPos.x = uv.x + offsetX;
newPos.y = uv.y + offsetY;
Then I experimented with some arbitrary offsets and was able to re-align the picture. There is still some minimal blurring (or loss in detail) but result was noticeably better.
Problem with this solution currently, that I don't know how to predict offset either depending on resolution or window size.

Segmenting Part of the image using threshold

Im trying to isolate and segment the yellow car body to change the color of it. in order to do that i need to separately identify the body from the image. And continue oration with the remaining white pixels. And im using C#, here the plan
Color d;
Color newColor = Color.YellowGreen;
for(inti =0;i<carimage.Width;i++){
for(intj =0;j<carimage.Height;j++){
d = carimage.GetPixel(i, j);
if(d.R == 255 && d.G==255 && d.B == 255)
image.SetPixel(i, j, newColor );
}
}
simple thresholding will trow the second image where car body is not separated correctly. i tried Aforge.net Fill holes image filter but no significant change has been done to the threshold image. I tried to use color filter but it i did not return a correct output due to color vary of the body. can anyone suggest and solution for this?
Original Image
Threshold Image

Instead of thresholding, you might want to look into clustering.
As a quick&dirty test, I've increased the image brightness in HSB space (using Mathematica):
brightnessAdjusted = Image[
Map[#^{1, 1, 0.2} &, ImageData[ColorConvert[img, "HSB"]], {2}],
ColorSpace -> "HSB"]
Then I've used simple K-Nearest clustering:
(clusters = ClusteringComponents[ColorConvert[brightnessAdjusted, "RGB"], 3,
Method -> "KMeans"]) // Colorize
to find clusters of similar colors in the image (there are many more, probably more suitable clustering algorithms, so you should experiment a little). Then I can just adjust the color in one of the clusters:
Image[MapThread[If[#1 == 2, #2[[{1, 3, 2}]], #2] &, {clusters, ImageData[brightnessAdjusted]}, 2]]
If you want to use thresholding, you should probably use a CIE color space, since euclidian distances in that color space are closer to human perception.

I had a similar project few years ago. I can't remember the exact details, but the idea was to shift a (not too small) sliding window over the image, and calculate the average intensity (maybe for R, G and B separately) inside the window at each position. I filled a "threshold image" with these averages, and subtracted it from the original image. There was a scaling factor somewhere, and other tuning stuff, but the point is, such an approach was way better than using a constant threshold.

If you are going to use a set of thresholds, you might be better of selecting yellow hues in the Hue Saturation Value colorspace. See the related SO question.

I=imread('test.jpg');
I=im2double(rgb2gray(I));
BW=im2bw(I,0.64);imshow(BW)
Gives me :
I got the 0.64 threshold by looking at the image's histogram. I suggest you use MATLAB to do image processing as it is much easier. Hope that helps you in colouring the image.