I have an external windows application (no source code) that has a grid within it. This runs as a separate process. When the user selects a cell within the grid via mouse click, I need to be able to read the value within that cell. Can anyone provide some direction on what API's I would need to use to be able to trap and listen to the events?
You best option is UI Automation Overview or accessibility as older technology.
Also you could take a look at this Pinvoke SetFocus to a particular control on how to invoke things on another process (pretty much unrelated to automation, automation works w/o that)
UI Automation is the best tool for the job, however, the downside is that not every app supports that - so this very much depends on the app you're targeting. Some support only the legacy acessibility (IAccessible, IAccessible2 etc.), usually there is a 'combined' approach. Older techniques don't work very good any more but you could try traversing windows, child windows in the target window (for that direction you'll probably need the above technique sooner or later) and hoping you could get it from standard controls, windows text, via messages etc.
Related
I've started using FlaUI for Automating my thick client .net application. The application is Windows Form based. The start was good and Login Form was identified and I could Login, but after that came the dead end and I found that almost everything in the application is developed as Pane control type.
So, there is grid, table etc. but they all just appear as Pane type when I see the object hierarchy using Inspect.exe or FLAUInspect tools. And nothing really appears in thier property, so it seems that nothing could be read. But before giving up I just wanted to check with experienced audience on this forum if there is really any way to get the data from Pane objects.
Please suggest if there is any way, even that means using other libraries like UIAutomation, TestStack.White, etc.
UPDATE: I now understand little more about this. So, the objects that are there in the pane are developed in syncfusion and devexpress. Is it possible to identify objects developed in syncfusion and devexpress using FlaUI or UIAutomation or TestStack.White, etc ?
I don't know if you have already tried the following steps. Have you add automationId's to your objects in xaml code with:
AutomationProperties.AutomationId="AnyID"
In the testcode, first initialize the main window of the application.
MainWindow = fApplication.GetMainWindow(fAutomation, null)?.AsWindow()
After that you can find your objects by the automationId's, like:
MainWindow .FindFirstDescendant(cf => cf.ByAutomationId(AnyID))
I did it this way, and don't have to know the hierarchy of my application. Maybe this will work?
Most UI Frameworks nowadays fully support UI Automation. So first make sure that you have a recent version of your framework (syncfusion, devexpress). In addition, some frameworks provide settings to enable UI Automation. Like for devexpress, you need to set
ClearAutomationEventsHelper.IsEnabled = false;
at the start of your application to test so it exposes way more things (like tabs) to FlaUI.
I want to develop a tool for test automation.
I have source code of the application under test, so I have the privilege to add some custom logic inside the app.
One part of the custom logic is to detect the content change, analyze the change and finally report the result outside to the test tool, such as: a message "the login window is ready" followed by locations of user_id and password control.
By using VisualTreeHelper and LogicalTreeHelper class, I can know the current status of the window, but I do not know WHEN to walk through the tree.
I found a similar question but this is for 3rd party window, I guess there may be better solution for app that I have access to source code.
In win32, I can hook WM_PAINT to detect window content change.
Do you have any hint about how to do this in WPF?
By the way, although I would like to add custom logic to the app, I also want to change the app logic as little as possible.
I am new to WPF, sorry if anything totally wrong.
You might want to specify exactly what you want to achieve, like describing an examlpe of what you want to do.
Are you aware of the VisualTreeHelper class? https://learn.microsoft.com/en-us/uwp/api/windows.ui.xaml.media.visualtreehelper
Based on what you said you wanted to achieve. I would probably subscribe to some "ready" or "loaded" event of a UI element. You should be able to get access to the UI element through the VisualTreeHelper.
You should also be able to interact with the UI elements through it, eg. click and enter information. And you could also run tests based on the state of the UI (I think).
I'm also sure there are plenty of Automated UI Testing frameworks for WPF, just Google: "automated UI Testing frameworks for WPF".
Hope this helps.
I am starting some research on a project that involves navigating the web with speech feedback from SAPI. This application will be used by the blind and visually impaired to navigate the web with full speech feedback of current selected object and the ability to read the entire page and fill out forms. I have a mass amount of experience in SAPI and the accessibility word, however I'm not so familiar with HTML and the best approach on taking on this project and i would like to ask for guidance from someone that may have experience in this area. The basic concepts, which I've already explained a little is to enable a blind user to use this standalone product without a screen reader and have the ability to navigate the web page with a keyboard and receive speech feedback from a synthesizer i have created. I can use the Accessibility API's built into windows to display location for low vision users.
Thank you!
IE supports all accessibility interfaces (I believe all other browsers do too - sample on SO ).
There are 2 parts - code access using IAccessible and related interfaces and good HTML (i.e. correct use of semantic tags like UL/LI) markup with possible use of additional CSS accessibility aria-XXXX attributes (i.e. aria-expanded)
Possible starting link - What's New for Accessibility in IE8
I have come to the conclusion in order to get a talking web browser i actually had to hook an instance of the WebBrowser Control with Set setwindowshookex. In my setwindowshookex callback i monitor for the event: OB_FOCUS (focus changed, could be keyboard or mouse) i then pass the object instance that has focus to the AccessibleObjectFromEvent API. Next, i get the IAccessible interface to this object and then enumerate any children if present and compare them against a known object list that relates to HTML/Web objects to use the proper class that will parse this field and announce text if available, if the user is in an editable field, focused buttons, etc... I also use the IAccessible class to get the objects location and present location to the user by drawing a rectangle on the current focused object. I have also created a class that extracts just link, text, removes images and more to make for an easier, more enjoyable web experience for people with visual impairments. I still have some quarks to work out, but progress is moving forward!
Im a c# developer and I believe that what I want to achieve is going to move out of the realms of some drop in .NET component so I am looking for advise on what I use externally which .NET can inter op with.
My requirements are to have an embedded web browser control in a WPF/Winforms applciation BUT I will also need to keep track of the following:
User interaction i.e. what pages they visit, forms submitted where they click etc.
DOM manipulation and traversing
I am guessing here but it seems that I might need to start looking at open source html/web browsers out there like WebKit etc. Is this the right track or is there anything currently available in the form of a control/COM object that I can use directly.
Cheers, Chris.
You should be able to accomplish point 1 using the webbrowser control in Visual Studio, but I dont think DOM manipulation is available, i do know you can traverse all the tags.
reference
I am investigating automated testing of an old Win32 application that used ActiveX controls. I am spiking use White (from Thougthworks) that uses Microsoft UIAutomation. I can find the AutomationElement related to the control, but how do I interact with it?
Spy++ sees the grid control as a single window, so I can't talk to rows, columns, or cells directly. How do I talk to the SSUltraGrid control from my test code?
Cheers
Nigel
The basic problem with some ActiveX and other custom controls like SSUltraGrid is what you mentioned, they're presented as just one window. So unless they've provided an API that makes them "friendly" to your GUI automation tool, you'll always face this challenge. Of course many companies offer newer versions of their products that are friendlier to UI Automation than before... perhaps upgrading the control is worthwhile...
Failing that... what the test engineers at my job have told me is that when they encounter that situation, if there are well known keystrokes to invoke the activity they desire, they send keystrokes to the control window. If they're lucky enough to have things in a fixed location, they might even be able to get away with sending mouse events. However, that's dicey at best.