We Keep Coding

C# move cursor on screen based on where text appears on desktop

So I have searched and can't seem to find the right resources or tools for what I am wanting to do so I thought I'd ask for some help. Below is what I am wanting to do.
I am making a .Net core application that will simulate some key strokes and that is all working fine, but I want to also move the mouse based on where certain text appears on the screen, however, I will not have control over the application where the text appears, so I will need to somehow either be streaming the desktop view and using something to analyze it, or constantly taking a screen shot every 10 seconds or so and analyzing that screen shot if the text is there and then move the mouse to that position of text and simulate a left click.
I am not really confident in the screenshot approach as I am not sure how I'd get the mouse coordinates, so I have a feeling I will need to feed a desktop stream into something in order to get the image I am analyzing and then overlay my own transparent overlay on top of that in order to move the cursor to appropriate position.
I hope this makes sense and any libraries or whatever that is recommended to do this is appreciated. I know how to move the cursor on the screen and left click, just haven't found on google the right library for analyzing a desktop screen in real time and getting coordinates based on searching conditions.
Thank you everyone.

So I figured out what I needed to do. Trying to match the text with and OCR proved to be impossible as the text was layered into an image where they actively tried to prevent detection, this is because its a video game and they don't want automation I guess. So what I did was determine I will always know what the button is going to look like, so I took a screen shot of the button and now I take screenshots of the game every 10 seconds and then search that image for my button template image. If the image has a match, then I generate coordinates and send mouse there and click it. Below is the code I used and you will need the libraries from AForge to use this code.
Bitmap sourceImage = (Bitmap)Bitmap.FromFile(#"C:\testjpg.jpg");
Bitmap template = (Bitmap)Bitmap.FromFile(#"C:\buttonjpg.jpg");
// create template matching algorithm's instance
//Resize value
var resizePercent = 0.4;
//Resizing images to increase process time. Please note this will result in skewed coordinates.
//You must divide the coordinates by your resizePercent to get native coordinates taken from screen shots.
sourceImage = new ResizeBicubic((int)(sourceImage.Width * resizePercent), (int)(sourceImage.Height * resizePercent)).Apply(sourceImage);
template = new ResizeBicubic((int)(template.Width * resizePercent), (int)(template.Height * resizePercent)).Apply(template);
// (set similarity threshold to 0.951f = 95.1%)
ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0.951f);
// find all matchings with specified above similarity
TemplateMatch[] matchings = tm.ProcessImage(sourceImage, template);
// highlight found matchings
BitmapData data = sourceImage.LockBits(
new Rectangle(0, 0, sourceImage.Width, sourceImage.Height),
ImageLockMode.ReadWrite, sourceImage.PixelFormat);
foreach (TemplateMatch m in matchings)
{
Drawing.Rectangle(data, m.Rectangle, Color.White);
Console.WriteLine(m.Rectangle.Location.ToString());
var x = m.Rectangle.Location.X;
var y = m.Rectangle.Location.Y;
//Fixing the coordinates to reflex the origins of the original screenshot, not the resized one.
var xResized = (int)(x / resizePercent);
var yResized = (int)(y / resizePercent);
// do something else with matching
}
sourceImage.UnlockBits(data);
I did notice my coordinates were off usually around ~4.5%-8% from dead center(the button still gets clicked, but I want a perfect center click), so for me I am calculating a range of coordinates, which in this range are not very many to ensure the button I want to click is clicked and not missed and just having program click all the potential coordinates.
So I hope this helps someone!
Kind Regards,
Aaron

Depending on the application there are a number of approaches you might want to try.
For example, using Application Insights and UI Automation, you might be able to capture the text directly.
If you have the exe file, then you might be able to decompile it with something like dotPeek and then use Visual Studio's Live debugging feature to access the text.
You might be able to inspect the heap of the running process via a tool like HeapMemView.
Optical Character Recognition might be an option worth considering. In that case Tesseract is an open source OCR engine worth considering. This can be used in conjunction with screenshots to extract text.

I can't get the correct coordinate of an image with mouse over C#

This is my first question ever on Stackoverflow. So I hope it for the best answer.
I want to get correct X and Y coordinate of an Image with mouse over Event (WFA .NET Framework).
Take a look at my cursor
The coordinate should be somewhere between 500 for X and 427 for Y, but I only get as I posted. I already maxed out the scroll. And I think the image resolution is correct, here's the image properties
Here's my code:
private void pbInput_MouseMove(object sender, MouseEventArgs e) {
mouseX.Text = e.X.ToString();
mouseY.Text = e.Y.ToString();
}
And I have a plan to zooming the image for the future, so I put "auto scroll" panel below the picture box.
Could you help me? Thank you so much.
PS: Sorry for my bad English

Try putting picturebox inside panel
set panel to:
this.panel1.AutoScroll = true;
and picturebox to
this.pictureBox1.SizeMode = System.Windows.Forms.PictureBoxSizeMode.AutoSize;
this will allow panel to have scroll bars and picturebox will have its full size adjusted to actual picture size

UWP Magnifier control (tool) that follows cursor

I am working on an assignment right now and was asked to create an app to select an area on image with ability to magnify a part of the image around cursor.
Right now I stuck on the magnifier part. There is a Magnifier control in WPF, but how about UWP? Has anyone had any experience creating magnifier in UWP?
SO far I've found this, but UWP has different API's:
http://csharphelper.com/blog/2015/06/zoom-and-crop-a-picture-in-c/
My logic is:
1. Draw circle around the cursor and re-draw it every time the cursor moves.
2. Take a screenshot (render) specified area around it
3. Magnify the are
4. Fill the circle with the magnified image (Bitmap)
Any tips or suggestions would be much appreciated. Thank you

Draw circle around the cursor and re-draw it every time the cursor moves.
You could register the PointerMoved event for your panel(e.g, Canvas) and get current pointer by using the following method:
private void Canvas_PointerMoved(object sender, PointerRoutedEventArgs e)
{
var pointer = e.GetCurrentPoint(sender as UIElement);
}
And then, you could add a Ellipse on it and set its position by current pointer.
Take a screenshot (render) specified area around it
You could use RenderTargetBitmap class APIs to render specific area.
Magnify the are
You could resize the rendertargetbitmap. Check this thread How to Resize the RenderTargetBitmap.
Fill the circle with the magnified image (Bitmap)
After you get the final rendertargetbitmap, you could use it to make a ImageBrush, then you could specify this ImageBrush to the Ellipse's Fill property like the following:
ellipse.Fill = new ImageBrush() { ImageSource = renderTargetBitmap};

Interact with drawings

I'm using bmp but I'm open to other suggestions since I found barely any info while searching for bmp interaction.
I finally plotted my drawing into a pictureBox
Everything is working properly, but when i click the drawing, I'd like to read where is the pointer on the bmp. If it is above the displaying graph, I would wanna take the X position out of it.
Edit:
Another not important but interesting question:
Is it possible to stick the mouse position whenever it is above the bmp, to move only above or below the graph function?

I've found my own answer, here :)
private void obterposicaonografico()
{
// Set the Current cursor, and display how many points in X axis you took
this.Cursor = new Cursor(Cursor.Current.Handle);
Cursor.Position = new Point(Cursor.Position.X, Cursor.Position.Y);
MessageBox.Show(Cursor.Position.X.ToString());
//Define it's position on X axis, subtracting the whole form X location
//This will ensure that if the user moves the form around, it will still be relative to the own form
//and not the windows positioning
decimal posicaoX = Cursor.Position.X -this.Location.X
MessageBox.Show(Cursor.Position.X.ToString());
}

Simulate Mouse Drag Programmatically

I have a Windows Forms application with a control. The control consists of a chart panel with a canvas on which I paint. What I would like to do is to programmatically mouse drag the panel so that I have a specific distance between the right edge of the canvas and the last item painted on the canvas. I have tried two approaches. The both work in the sense that the panel is dragged as desired BUT I cannot seem to be able to get the precision of movement I desire. I coded a mouse simulator and have tried two approaches.
Approach 1:
if(this.ChartControl.ChartPanel.CanFocus)
{
// ... Focus the chart panel to be adjusted.
this.ChartControl.ChartPanel.Focus();
// ... Move cursor to lastBarScreenCoordinates on the chart panel to be adjusted.
Cursor.Position = new Point(lastBarScreenCoordinates.X, lastBarScreenCoordinates.Y);
MouseSimulator.SetMouseDragThresholds();
// ... Move chart panel to required position.
MouseSimulator.LeftMouseButtonDown(lastBarScreenCoordinates.X, lastBarScreenCoordinates.Y);
MouseSimulator.MouseMove(lastBarScreenCoordinates.X-positionShift,
lastBarScreenCoordinates.Y);
MouseSimulator.LeftMouseButtonUp(lastBarScreenCoordinates.X-positionShift,
lastBarScreenCoordinates.Y);
MouseSimulator.ResetMouseDragThresholds(_cx_default, _cy_default);
// ... Redraw the chart panel.
this.ChartControl.ChartPanel.Refresh();
// ... Reset cursor to its starting position.
Cursor.Position = new Point(startingCursorX, startingCursorY);
}
Approach 2:
if(this.ChartControl.ChartPanel.CanFocus)
{
// ... Focus the chart panel to be adjusted.
this.ChartControl.ChartPanel.Focus();
// ... Move cursor to lastBarScreenCoordinates on the chart panel to be adjusted.
Cursor.Position = new Point(lastBarScreenCoordinates.X, lastBarScreenCoordinates.Y);
MouseSimulator.SetMouseDragThresholds();
// ... Move chart panel to required position.
MouseSimulator.LeftMouseButtonDown(lastBarScreenCoordinates.X, lastBarScreenCoordinates.Y);
Cursor.Position = new Point(lastBarScreenCoordinates.X-positionShift,
lastBarScreenCoordinates.Y);
WindowsCommunication.SendMessage(this.ChartControl.Handle, 0x200, IntPtr.Zero,IntPtr.Zero);
MouseSimulator.LeftMouseButtonUp(lastBarScreenCoordinates.X-positionShift,
lastBarScreenCoordinates.Y);
MouseSimulator.ResetMouseDragThresholds(_cx_default, _cy_default);
// ... Redraw the chart panel.
this.ChartControl.ChartPanel.Refresh();
// ... Reset cursor to its starting position.
Cursor.Position = new Point(startingCursorX, startingCursorY);
}
I am using SendInput for simulating mouse clicks. Here is sample left mouse button down code ...
public static void LeftMouseButtonDown(int x, int y)
{
INPUT mouseInput = new INPUT();
mouseInput.type = SendInputEventType.InputMouse;
mouseInput.mkhi.mi.dx = CalculateAbsoluteCoordinateX(x);
mouseInput.mkhi.mi.dy = CalculateAbsoluteCoordinateY(y);
mouseInput.mkhi.mi.mouseData = 0;
mouseInput.mkhi.mi.time = 0;
mouseInput.mkhi.mi.dwFlags = MouseEventFlags.MOUSEEVENTF_LEFTDOWN;
SendInput(1, ref mouseInput, Marshal.SizeOf(new INPUT()));
}
And I calculate normalized absolute coordinates for the mouse as follows ...
private static int CalculateAbsoluteCoordinateX(int x)
{
return ((x * 65536) + GetSystemMetrics(SystemMetric.SM_CXSCREEN) - 1) /
GetSystemMetrics(SystemMetric.SM_CXSCREEN);
}
So here are the precision issues. If I use Approach 1 (mouse move), the measured distance between the last item painted and the right edge of the canvas is different from what I set in positionShift and the cursor position difference does not equal positionShift. I initially thought it was due to pointer ballistics issues so I tried using Approach 2. Approach 2 does give me precision in pointer positioning but I am still having difficulty in that the panel moves but the distance between the last bar painted and the right edge of the canvas does not equal the positionShift amount as it should. It always seems to be off. I have been working on this for a long time now and am at my wits end. I am not sure what is going on here. How to improve the precision in my canvas drag by simulated mouse drag?

Well what you can do is this, First of all I believe the SendInput API allows for an AbsoluteValue flag so there is no need to calculate those values which may be the issue but most likely not.
Although I am curious as to why you are using a Mouse Drag opperation for this. It seems like all you want to do is reposition the canvas on every draw by some specified amount. if this is the case why not just set it explicitly on the canvas itself. Also it is unclear if you are using pure WinForms or WPF. The unclear bit being Canvas which I am fairly certain is only usable with WPF enabled windows.
That being said
WPF fix,
Depending on the Object containing the canvas just set its margin appropriately for the situation, since I do not know the data you are working with I cant say much about that. But this is a relatively simple idea so let me know if that works, it should give you, at least close too, a pixel perfect alignment.
WinForms,
Just do the above for the "Canvas" object you were talking about, or use absolute coordinates of the object to move it around.
If you could supply a sample of what you were working on looked like roughly maybe we could have a better idea of what you mean.