System.Windows.Automation is EXTREMELY slow.
I execute:
element.FindAll(TreeScope.Children, Condition.TrueCondition);
Obtaining only 30 child elements may take 1000ms on a very fast computer.
I have even seen it hanging forever while getting the child elements of a Tree in a QT application.
Is this a known problem?
I cannot find any usefull answer after googling a lot.
System.Windows.Automation is EXTREMELY slow.
System.Windows.Automation is full of bugs. It may not return all children of an AutomationElement, which is a very severe bug.
Apart from that the implementation is not thread safe.
System.Windows.Automation is deprecated. Do not use it!
In the MSDN you find the following note:
UI Automation was first available in Windows XP as part of the
Microsoft .NET Framework. Although an unmanaged C++ API was also
published at that time, the usefulness of client functions was limited
because of interoperability issues. For Windows 7, the API has been
rewritten in the Component Object Model (COM).
Although the library functions introduced in the earlier version of
UI Automation are still documented, they should not be used in new
applications.
The solution to slow performance is to use the new IUIAutomationElement COM interface instead of the old System.Windows.Automation C# interface. After that the code will be running lightning fast!
Apart from that the new interface offers much more patterns and Microsoft is extending it continously. In the Windows 10 SDK (UIAutomationClient.h and UIAutomationCore.h) several patterns and properties have been added which are not available in the .NET Automation framework.
The following patterns are available in the COM version of UIAutomation which do not exist in System.Windows.Automation:
IUIAutomationLegacyIAccessiblePattern
IUIAutomationObjectModelPattern
IUIAutomationAnnotationPattern
IUIAutomationTextPattern2
IUIAutomationStylesPattern
IUIAutomationSpreadsheetPattern
IUIAutomationSpreadsheetItemPattern
IUIAutomationTransformPattern2
IUIAutomationTextChildPattern
IUIAutomationDragPattern
IUIAutomationDropTargetPattern
IUIAutomationTextEditPattern
IUIAutomationCustomNavigationPattern
Additionally the following Control types have been added:
AppBar
SemanticZoom
Additionally the following Element's have been added:
IUIAutomationElement2
IUIAutomationElement3
IUIAutomationElement4
And what concerns the bugs: The new COM UIAutomation Framework is very well designed and I could not find bugs on the client side of the framework which is a great progress compared to System.Windows.Automation. But several missing features and even bugs on the server side of the framework. On the server side each GUI framework must implement an UIAutomation provider (see MSDN: Interfaces for Providers). So these problems differ depending on what type of application you are automating because each GUI framework has it's own problems:
In the Native Windows GUI features are missing: Lots of controls do not implement the patterns that they should implement. For example a SplitButton in a native Toolbar should implement the Invoke pattern to click the button and the ExpandCollapse pattern to open the drop-down menu. But the ExpandCollapse pattern is missing which makes it difficult to use SplitButtons. If you obtain a Toolbar SplitButton by IUIAutomation->ElementFromPoint() and then ask for it's parent you will get a crippled element. And the Pager control cannot be automated at all.
Also in WPF applications there are controls that are implemented buggy by Microsoft: For example if you have a Calendar control you see two buttons at the top to switch to the next/previous month. If you execute the Invoke pattern on these buttons you will get an UIA_E_NOTSUPPORTED error. But this is not a bug on the client side of the framework, because for other buttons the Invoke pattern works correctly. This is a bug in the WPF Automation server. And if you test IUIAutomationTextRange with a WPF RichTextBox, you will find that several commands are not implemented: Select() and ScrollIntoView() do simply nothing.
For .NET Forms applications Microsoft did not make much effort to support them. The .NET Calendar control cannot be automated at all. The entire control is not even recognized as Calendar. It has the ControlType "Pane" with no child elements in it. The same applies to the DateTimePicker. And for complex controls like DataGrid and PropertyGrid the only implemented pattern is LegacyIAccessible which is a poor support. These controls should implement at least the Table and the Grid and the ScrollItem pattern.
Also Internet Explorer cannot be automated because elements outside the visible area cannot be scrolled automatically into view due to missing coordinates. (The Bounds are returned as an empty rectangle) And the ScrollItem pattern is not implemented. (Yes, I know that Internet Explorer has been replaced with Edge in Windows 10, but the UIAutomation framework exists since Windows 7 and Microsoft did not implement a usefull automation support in Internet Explorer in all these years)
I saw even complete crashes of the automated application. For example Visual Studio and TotalCommander will crash if you execute certain automation commands on a certain control. Here - once again - the bug lies in the server side implementation of the framework.
Summary: We have a great framework with limited usefullness. The Microsoft team that developed the new UIAutomation framework did a great job, but the other areas in Microsoft (the native GUI, WPF, .NET and Internet Explorer team) do not support this framework. This is very sad because only a small effort would have to be made to offer a better functionality. But it seems that the users who use UIAutomation in the first place (handicapped people) are not a profitable market.
Related
Blueprism gives the possibility to spy elements (like buttons and textboxes) in both web-browsers and windows applications. How can I spy (windows-based only) applications using Python, R, Java, C++, C# or other, anything but not Blueprism, preferrably opensource.
For web-browsers, I know how to do this, without being an expert. Using Python or R, for example, I can use Selenium or RSelenium, to spy elements of a website using different ways such as CSS selector, xpath, ID, Class Name, Tag, Text etc.
But for Applications, I have no clue. BluePrism has mainly two different App spying modes which are WIN32 and Active Accessibility. How can I do this type of spying and interacting with an application outside of Blueprism, preferrably using an opensource language?
(only interested in windows-based apps for now)
The aim is of course to create robots able to navigate the apps as a human would do.
I guess you are using selenium for web-browsers
There is also some projects for windows based applications working with a Windows Driver.
Take a look to the project on github, it may be what you are looking for.
https://github.com/2gis/Winium
https://github.com/microsoft/WinAppDriver
Autoit: https://www.autoitscript.com/site/
It also comes with an identify mode for application elements and is has a big community
There is a free version of Blue Prism now :) Also Blue Prism uses win32, active accessibility and UI Automation which is a newer for of the older active accessibility.
To do this yourself without looking into Blue Prism you would need to know how to use UIA with C#/VB.new or C++. There are libraries however given that Blue Prism now has a free version I would recommend using that. Anything specific can be developed withing a code stage within Blue Prism.
I need to interact with a windows application by clicking a button in my application. More precisely: how can I write in a textbox or scroll a dropdown menu?
There are multiple ways to do this depending on what kind of application you are automating. If you are automating a WPF application I would suggesting using Microsoft UI Automation (UIA). There are some nice wrappers written around UIA like TestStack.White and FlaUI. FlaUI is the more modern of the two and supports UIAv3 using a COM wrapper. TestStack.White is built on top of a UIAv2 using the managed wrapper in the .NET Framework which is no longer supported.
If you are automating anything else besides a WPF application you can do straight PInvokes to SendMessage. I would suggest staying away from that method and using the Microsoft UIA framework since sending windows messages can get quite verbose. If you really want to go with the Win32 route I suggest using something like AutoIT to automate your application.
Lastly, you will want to download the Windows SDK and run Inspect.exe from that. Inspect.exe is the application that will allow you to see the properties you are attempting to query and the patterns that are available. Applications like AutoIT have essentially built their own Inspect.exe or reference and application call UISpy which is also in the Windows SDK but mostly superseded by Inspect.exe.
I'm trying to automate a hidden .NET application, with another .NET application (written in c#) using the easiest way possible. It's NOT for testing purposes, it's a way to fulfill the lack of scripting for this application.
I already tried white framework, but there is one major problems with it: the way it's working. It's slow and it's not working on hidden windows and controls (like the winAPI does). Whats more, when "clicking" white moves the mouse, brings it's targeted window to the front and so on.
I was also thinking about using a user32.dll wrapper, because the way it's handling it's target is what I need, but I've red it's not working with .NET applications. It also would be a problem working with it, because my targeted application got 5 button labeled "...", and would be really hard finding 2 of them I need. I also would like to use the controls .NET id (the name the developer gave to it's controls when designing the GUI).
BTW, my targeted application is MeGUI if that helps. We do a lot of video encoding and a tool like this would help us a lot. I need the MeGUI to be hidden, because I'm the only programmer, others using my tool shouldn't see what happens in the background, not to talk about the many windows popping all around.
You can add a reference to the exe from your project and then create an AppDomain to run its main method. From there, it should be possible to queue delegates to its main thread's loop. With a bit of reflection, you could have those delegates invoke the click events and whatnot directly.
I've never attempted this approach, but it should work.
You should try Stephens idea instead of scripting a hidden app. A .NET Windows Forms App (EXE) is still a .NET Assembly and that means you can use that the same way as a DLL, just add a reference and use the public classes.
If you still want to try some scripting, take a look on the "Microsoft UI Automation" API and the "System.Windows.Automation" namespace.
Nice article here: http://msdn.microsoft.com/en-us/magazine/cc163465.aspx
MSDN Doc: http://msdn.microsoft.com/en-us/library/system.windows.automation.aspx
I've inherited a C# window's application that I'm not real crazy about. I've got a looming deadline and I'm scared to death that some of my changes might be having adverse effects on existing functionality.
I've got a hobbyist background to RoR and I'm fairly comfortable with testing in that framework (using both RSpec and Cucumber).
I love having test scripts that can be ran on a regular basis and I'm willing to spend my personal time developing those for this particular project. I purchased a book from PragProg.com on scripted GUI testing with Ruby (http://pragprog.com/titles/idgtr/scripted-gui-testing-with-ruby). So far, I'm digging what I'm seeing and I think that this should work well.
Unfortunately, I've got a fundamental lack of understanding concerning Windows app development. I'm making calles to FindWindowEx (via Win32API) to "attempt" to retrieve sub-controls in my application.
A big part of my confusion is how I should retrieve the Class Name of the control that I'm trying to capture. The example provided in the text is as follows:
edit = find_window_ex.call #main_window, 0, 'ATL:00434310', nil
Where #main_window is my application's main window handle, and 'ATL:...' is the class of a text box area. There is no explanation given as to how the author arrived at 'ATL:...'.
I've read some very old posts concerning MS's SPY++, but those seem to be obsolete (or for some reason it wasn't installed when I installed vs2010).
So, what's the best way for me to find control classes to be used with the findWindowEx call? I do have the source code - should I be pulling from there? What if I don't have the source code and I want to automate an application? Is there a utility that allows you to somehow "browse" controls on a running application?
Sorry for the length - thanks in advance for the help!
Bob
The best is for you to install the components so that you get Spy++, this is the best way I know of to get to the actual class names esp. if you do not have the source to the original controls, which might be from a library or possibly some standard ActiveX controls that Microsoft ships.
The ATL class name is probably for controls developed using Microsoft Active Template Library (ATL), this is a C++ template library which significantly simplifies the development of ActiveX controls, and COM objects etc. in C++.
I have a project that i started as a WinForms application as that was the format i was confortable with at the time. I have since started dabbling in WPF an introduced some WPF UserControls (mainly grids) into my project and absolutely love them.
The question i have is, is there any real advantage to me changing the UI Project of my solution into a purely WPF project, and get rid of any WinForms?
I am fully aware that each format suits a certain environment, and you wouldnt be able to give a definitive answer without knowing more of the details, but i would like to know peoples opinions, and if anyone has done a silimar thing of converting an existing WinForms App into a WPF frontend, and any observations they made in doing so.
Thanks
Underlying the new features in WPF is a powerful new infrastructure based on DirectX, the hardware-accelerated graphics API that’s commonly used in cutting-edge computer games. This means that you can use rich graphical effects without incurring the performance overhead that you’d suffer with Windows Forms. In fact, you even get advanced features such as support for video files and 3-D content. Using these features (and a good design tool), it’s possible to create eye-popping user interfaces and visual effects that would have been all but impossible with Windows Forms.
WPF enhances features that appeal directly to business developers, including a vastly improved data binding model, a new set of classes for printing content and managing print queues, and a document feature for displaying large amounts of formatted text.
But if you’ve done a substantial amount of work creating a Windows Forms application, you don’t need to migrate it wholesale to WPF to get access to new features such as animation. Instead, you can add WPF content to your existing Windows Forms application, or you can create a WPF application that incorporates your legacy Windows Forms content.
Reference: Pro WPF in C# 2008: Windows Presentation Foundation with .NET 3.5, Second Edition