We Keep Coding

How to get cursor position within another app

I'm using .Net c# winforms. I want to move my mouse over another application and see the cusor X,Y position as I move the mouse over it's interface. Displaying the X,Y on my forms title bar is ok. I want to see the X,Y location for a specific spot on this app's form.
The reason I want to do this is because there are controls on this app's interface that I can mouse click on to turn a knob, one mouse click per knob turn. I want to write an app that I can position the mouse cursor to that specific X,Y position on this app form and then do a software mouse click to turn that same knob one turn. But I want to do this from my app, kind of like remote control I guess you could say. The other app knobs responds to mouse clicks when you are over the correct X,Y location.
Thanks for any pointers in the right direction.

Add a Label to your Form and wire up its MouseMove() and QueryContinueDrag() events. Use the WindowFromPoint() and GetAncestor() APIs to get a handle to the main window containing the cursor position, then use the ScreenToClient() API to convert the screen coordinate to the client coordinate of that Form. Run the app and left drag the Label in your Form over to the knobs in your target application. The title bar should update with the client coords of the current mouse position relative to the app it is over:
private const uint GA_ROOT = 2;
[System.Runtime.InteropServices.StructLayout(System.Runtime.InteropServices.LayoutKind.Sequential)]
public struct POINT
{
public int X;
public int Y;
}
[System.Runtime.InteropServices.DllImport("user32.dll")]
private static extern IntPtr WindowFromPoint(int xPoint, int yPoint);
[System.Runtime.InteropServices.DllImport("user32.dll", ExactSpelling = true)]
private static extern IntPtr GetAncestor(IntPtr hwnd, uint gaFlags);
[System.Runtime.InteropServices.DllImport("user32.dll")]
private static extern bool ScreenToClient(IntPtr hWnd, ref POINT lpPoint);
private void label1_MouseMove(object sender, MouseEventArgs e)
{
if (e.Button == MouseButtons.Left)
{
label1.DoDragDrop(label1, DragDropEffects.Copy);
}
}
private void label1_QueryContinueDrag(object sender, QueryContinueDragEventArgs e)
{
Point pt = Cursor.Position;
IntPtr wnd = WindowFromPoint(pt.X, pt.Y);
IntPtr mainWnd = GetAncestor(wnd, GA_ROOT);
POINT PT;
PT.X = pt.X;
PT.Y = pt.Y;
ScreenToClient(mainWnd, ref PT);
this.Text = String.Format("({0}, {1})", PT.X.ToString(), PT.Y.ToString());
}

Obtain mouse click position relative to active window

Having worked out how to obtain the mouse click position anywhere along the monitor boundaries using low level hooks I receive an X Y coordinate that will contain a value typically between x: -1680 to +1920 and y: 0 to 1200 in my pcs case. Easy enough!
Now the problem is that I now want to calculate the mouse position relative to a given window that I have so I use GetForegroundWindow() and GetWindowRect(HandleRef hWnd, out RECT lpRect) to obtain my active window coordinates.
Where I am stuck is I require the current active desktop (By active I mean which monitor the click occurred on) to calculate the coordinates of my mouse click relative to a window.
Unfortunately I have not been able to find an API call like GetActiveMonitor() or similar so hopefully someone can point me in the right direction?

[DllImport("user32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
static extern bool GetWindowRect(IntPtr hWnd, ref RECT lpRect);
[StructLayout(LayoutKind.Sequential)]
private struct RECT
{
public int Left;
public int Top;
public int Right;
public int Bottom;
}
Call it as:
RECT rct = new RECT();
GetWindowRect(hWnd, ref rct);
after get your mouse position like this
int mouserelativepositionX = mousePosition.X - rct.Left;
int mouserelativepositionY = mousePosition.Y - rct.Top;

My guess is that you can know where your mouse is by using an if:
if(mousePosition.X > -1680 && mousePosition.X < 0)
//We are in monitor 1;
else
//Monitor 2;

How to programmatically (C# or Java) launch an app in Windows and invoke click in it's window?

There is a simple application that works in Windows. It has very simple interface: squre window with buttons in fixed coordinates.
I need to write a program that makes use of this application: to launch it and to click one of buttons (let's say invoke a click at (150,200)).
Is there any way to do it in Java or .NET?

The Java based solution is to launch the app. in a Process and use the Robot to interact with it.
The best solution on this thread was by #HFoE but deleted by a moderator. For reference, it basically came down to..
If you want to control another Windows application, use a tool that was built specifically for this such as AutoIt V3.
Since "Don't do it" seems to be considered a valid answer when an alternative is supplied (by general opinion on Meta), I cannot understand why the answer was deleted.

As Hovercraft Full Of Eels if you can - use autoit - it's much easier. If AutoIt is not an option then you will need to use winAPI functions in order to do it.
For example to call mouseclick at coordinates:
[DllImport("user32.dll")]
static extern bool SetCursorPos(int x, int y);
[DllImport("user32.dll")]
static extern bool GetCursorPos(ref Point lpPoint);
[DllImport("user32.dll")]
public static extern void mouse_event(int dwFlags, int dx, int dy, int cButtons, int dwExtraInfo);
public void LeftMouseClick(int xpos, int ypos) //Make a click at specified coords and return mouse back
{
Point retPoint = new Point();
GetCursorPos(ref retPoint); // set retPoint as mouse current coords
SetCursorPos(xpos, ypos); //set mouse cursor position
mouse_event(MOUSEEVENTF_LEFTDOWN, xpos, ypos, 0, 0);
mouse_event(MOUSEEVENTF_LEFTUP, xpos, ypos, 0, 0); //click made
SetCursorPos(retPoint.X, retPoint.Y); //return mouse position to coords
}
But be aware, that to make click inside a window it needs to be at front of you - you cannot click to a minimized app for example.
If you want to try - you can find all needed functions(how to run a programm, get needed window by hwnd and so on) at PInvoke

For .Net you can pretty much use AutomationElement which I prefer. There's a bit of learning time, but it shouldn't take much. You can start your app with ProcessStartInfo.
If you have VS2010 Pro or Ultimate you can use the CodedUITests to generate a couple of button pushes.
As #Hovercraft Full Of Eels suggested - Autoit, Python could do the same

Yes - in C#...
Use the Process class to start the process (there are plenty of resources on the web on how to do this.
Wait until the process has started (either just wait for a fixed amount of time which is probably going to be long enough, or you could try and do something fancy like IPC or monitoring for a window being created)
To simulate the click take a look at How to simulate Mouse Click in C#? which uses a P/Invoke call to the mouse_event function.
However note that there are several things that can go wrong with this
Someone might move the window, or place another window on top of that window in the time it takes to launch the application
On a slower PC it may take longer to load the application (this risk can be mitigated by doing things like monitoring open windows and waiting for the expected application window to appear)

In .net you can Process.Start from System.Diagnostics to launch an application, you can even pass parameters, and to simulate mouse events you can use P/Invoke there is already an answer to that on SO here

Here is my working test app to play with clicking in windows.
We just start some app and hope to click it in right place)
It would be nice to have some solution for capturing windows this way =)
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.Threading;
namespace ConsoleApplication8
{
class Program
{
static void Main(string[] args)
{
var startInfo = new ProcessStartInfo(#"C:\Users\Bodia\Documents\visual studio 2010\Projects\ConsoleApplication8\WindowsFormsApplication1\bin\Debug\WindowsFormsApplication1.exe");
startInfo.WindowStyle = ProcessWindowStyle.Maximized;
Console.WriteLine(1);
var process = Process.Start(startInfo);
Console.WriteLine(2);
Thread.Sleep(400);
Console.WriteLine(3);
LeftMouseClick(1000, 200);
Console.WriteLine(4);
}
static void CursorFun()
{
Point cursorPos = new Point();
GetCursorPos(ref cursorPos);
cursorPos.X += 100;
Thread.Sleep(1000);
SetCursorPos(cursorPos.X, cursorPos.Y);
cursorPos.X += 100;
Thread.Sleep(1000);
SetCursorPos(cursorPos.X, cursorPos.Y);
cursorPos.X += 100;
Thread.Sleep(1000);
SetCursorPos(cursorPos.X, cursorPos.Y);
cursorPos.X += 100;
Thread.Sleep(1000);
SetCursorPos(cursorPos.X, cursorPos.Y);
}
[DllImport("user32.dll")]
static extern bool SetCursorPos(int x, int y);
[DllImport("user32.dll")]
static extern bool GetCursorPos(ref Point lpPoint);
[DllImport("user32.dll")]
public static extern void mouse_event(int dwFlags, int dx, int dy, int cButtons, int dwExtraInfo);
public static void LeftMouseClick(int xpos, int ypos) //Make a click at specified coords and return mouse back
{
Point retPoint = new Point();
GetCursorPos(ref retPoint); // set retPoint as mouse current coords
SetCursorPos(xpos, ypos); //set mouse cursor position
mouse_event(MOUSEEVENTF_LEFTDOWN, xpos, ypos, 0, 0);
mouse_event(MOUSEEVENTF_LEFTUP, xpos, ypos, 0, 0); //click made
SetCursorPos(retPoint.X, retPoint.Y); //return mouse position to coords
}
struct Point
{
public int X;
public int Y;
}
private const int MOUSEEVENTF_LEFTDOWN = 0x02;
private const int MOUSEEVENTF_LEFTUP = 0x04;
private const int MOUSEEVENTF_RIGHTDOWN = 0x08;
private const int MOUSEEVENTF_RIGHTUP = 0x10;
}
}

How to get mouse position related to desktop in WPF?

Problem
When you search for such question using google you get a lot of hits but all solutions assume you have at least one window.
But my question is just like I phrased it -- not assumptions at all. I can have a window, but I could have zero windows (because I didn't even show one or I just closed the last one). So in short the solution cannot rely on any widget or window -- the only thing is known, is there is a desktop (and app running, but it does not have any windows).
So the question is -- how to get the mouse position?
Background
I would like to show windows centered to mouse position. There is no such mode in WPF (there are only center to owner, or center to screen) so I have to do it manually. The missing piece is mouse position.
Edits
Thank you all, so now I have the first part of the solution -- raw position. Now there is a problem how to convert the data for WPF. I found such topic:
WPF Pixels to desktop pixels
but again, it assumes having some window.
Then I googled more and I found solution:
http://jerryclin.wordpress.com/2007/11/13/creating-non-rectangular-windows-with-interop/
the code includes class for scaling up/down coordinates relying only on info about desktop. So joining those two pieces, I finally get the solution :-). Thanks again.

Getting the Screen Coordinates:
[DllImport("user32.dll")]
[return: MarshalAs(UnmanagedType.Bool)]
static extern bool GetCursorPos(out POINT lpPoint);
[StructLayout(LayoutKind.Sequential)]
public struct POINT
{
public int X;
public int Y;
public POINT(int x, int y)
{
this.X = x;
this.Y = y;
}
}
private void WritePoint(object sender, RoutedEventArgs e)
{
POINT p;
if (GetCursorPos(out p))
{
System.Console.WriteLine(Convert.ToString(p.X) + ";" + Convert.ToString(p.Y));
}
}
Converting Pixels to WPF Units:
[DllImport("User32.dll")]
static extern IntPtr GetDC(IntPtr hwnd);
[DllImport("gdi32.dll")]
static extern int GetDeviceCaps(IntPtr hdc, int nIndex);
[DllImport("user32.dll")]
static extern bool ReleaseDC(IntPtr hWnd, IntPtr hDC);
private Point ConvertPixelsToUnits(int x, int y)
{
// get the system DPI
IntPtr dDC = GetDC(IntPtr.Zero); // Get desktop DC
int dpi = GetDeviceCaps(dDC, 88);
bool rv = ReleaseDC(IntPtr.Zero, dDC);
// WPF's physical unit size is calculated by taking the
// "Device-Independant Unit Size" (always 1/96)
// and scaling it by the system DPI
double physicalUnitSize = (1d / 96d) * (double)dpi;
Point wpfUnits = new Point(physicalUnitSize * (double)x,
physicalUnitSize * (double)y);
return wpfUnits;
}
Putting both together:
private void WriteMouseCoordinatesInWPFUnits()
{
POINT p;
if (GetCursorPos(out p))
{
Point wpfPoint = ConvertPixelsToUnits(p.X, p.Y);
System.Console.WriteLine(Convert.ToString(wpfPoint.X) + ";" + Convert.ToString(wpfPoint.Y));
}
}

Two options:
Use System.Windows.Forms.Control.MousePosition, or p/invoke
[DllImport("user32.dll", CharSet=CharSet.Auto, ExactSpelling=true)]
public static extern bool GetCursorPos([In, Out] NativeMethods.POINT pt);
The first option already does the p/invoke for you. I'm not entirely sure it requires you have some UI splashed up, but I don't think so. Yes, its winforms and not wpf, but it really doesn't have anything to do with where its located at.
If you want to skip any dependencies on system.windows.forms.dll then check out more information about the second on pinvoke.net.

I stumbled over that thread while looking for a solution for the same problem. In the meantime, I found PointToScreen, which does not require any P/Invoke. The method is available on any Visual starting .NET 3.0 (and thus UIElement, Control, etc.) and an implementation would look like this:
protected void OnMouseLeave(object Sender, MouseEventArgs e) {
var relativePosition = e.GetPosition(this);
var screenPosition = this.PointToScreen(relativePosition);
}

Move a window on keypress + mouse (like linux ALT + mouse down)

Simple, i want to move a windows pressing ALT+MOUSE, like linux os (ALT+drag).
It's possible to pass a win32 api (move api) to the windows interested clicking on it?
I have a windows services that hook key pressed (ALT button in specific).
When ALT key is pressed and a mouse down event is verified, i want to move window clicking anywhere, not only on the title bar!
Currently i move my form windows in this way:
using System.Runtime.InteropServices;
[DllImport( "user32.dll", CharSet = CharSet.Auto, SetLastError = false )]
static extern IntPtr SendMessage( IntPtr hWnd, uint Msg, int wParam, int lParam );
[DllImportAttribute( "user32.dll", CharSet = CharSet.Auto, SetLastError = false )]
public static extern bool ReleaseCapture();
private void Form1_MouseDown( object sender, MouseEventArgs e )
{
ReleaseCapture();
SendMessage( this.Handle, 0xa1, 0x2, 0 );
}
How can I get windows handle of the specific windows by clicking and after call SendMessage() on it?
It's possible?

You can do this by trapping the WM_NCHITTEST message that Windows sends to see what area of the window got clicked. You can fool it by returning HTCAPTION and it will dutifully perform the mouse actions you'd normally get when clicking the caption of a window. Including moving the window. Paste this code into your form:
protected override void WndProc(ref Message m) {
base.WndProc(ref m);
// Trap WM_NCHITTEST when the ALT key is down
if (m.Msg == 0x84 && (Control.ModifierKeys == Keys.Alt)) {
// Translate HTCLIENT to HTCAPTION
if (m.Result == (IntPtr)1) m.Result = (IntPtr)2;
}
}

I worked this out my self, came up with something interesting of my own calculations, worked perfectly for me, for any window (any active foreground window). Kinda long, but really easy to understand if you follow along the comments, hope it helps :)
The way it works, is that you press a certain registered key-combo, like Ctrl+Alt+M
and the mouse will stick in the center of an active window, you move the mouse, the windows follows it, press the SAME combo again, to release, no need for mouse clicks or anything.
public void MoveWindow_AfterMouse()
{
// 1- get a handle to the foreground window (or any window that you want to move).
// 2- set the mouse pos to the window's center.
// 3- let the window move with the mouse in a loop, such that:
// win(x) = mouse(x) - win(width)/2
// win(y) = mouse(y) - win(height)/2
// This is because the origin (point of rendering) of the window, is at its top-left corner and NOT its center!
// 1-
IntPtr hWnd = WinAPIs.GetForegroundWindow();
// 2- Then:
// first we need to get the x, y to the center of the window.
// to do this, we have to know the width/height of the window.
// to do this, we could use GetWindowRect which will give us the coords of the bottom right and upper left corners of the window,
// with some math, we could deduce the width/height of the window.
// after we do that, we simply set the x, y coords of the mouse to that center.
RECT wndRect = new RECT();
WinAPIs.GetWindowRect(hWnd, out wndRect);
int wndWidth = wndRect.right - wndRect.left;
int wndHeight = wndRect.bottom - wndRect.top; // cuz the more you go down, the more y value increases.
Point wndCenter = new Point(wndWidth / 2, wndHeight / 2); // this is the center of the window relative to itself.
WinAPIs.ClientToScreen(hWnd, out wndCenter); // this will make its center relative to the screen coords.
WinAPIs.SetCursorPos(wndCenter.X, wndCenter.Y);
// 3- Moving :)))
while (true)
{
Point cursorPos = new Point();
WinAPIs.GetCursorPos(out cursorPos);
int xOffset = cursorPos.X - wndWidth / 2;
int yOffset = cursorPos.Y - wndHeight / 2;
WinAPIs.MoveWindow(hWnd, xOffset, yOffset, wndWidth, wndHeight, true);
Thread.Sleep(25);
}
}
And now:
int moveCommandToggle = 0;
protected override void WndProc(ref Message m)
{
if (m.Msg == 0x0312)
{
int keyID = m.WParam.ToInt32();
if(keyID == some_key_combo_you_registered_for_the_moving)
{
if (moveCommandToggle++ % 2 == 0)
{
mover = new Thread(() => MoveWindow_AfterMouse());
mover.Start();
}
else mover.Abort();
}
}
}
If you're wondering about RECT:
public struct RECT
{
public int left; // xCoor of upper left corner.
public int top; // yCoor of upper left corner.
public int right; // xCoor of lower right corner.
public int bottom; // yCoor of lower right corner.
};
WinAPIs was just a static class that I did my DllImports in.