Blueprism gives the possibility to spy elements (like buttons and textboxes) in both web-browsers and windows applications. How can I spy (windows-based only) applications using Python, R, Java, C++, C# or other, anything but not Blueprism, preferrably opensource.
For web-browsers, I know how to do this, without being an expert. Using Python or R, for example, I can use Selenium or RSelenium, to spy elements of a website using different ways such as CSS selector, xpath, ID, Class Name, Tag, Text etc.
But for Applications, I have no clue. BluePrism has mainly two different App spying modes which are WIN32 and Active Accessibility. How can I do this type of spying and interacting with an application outside of Blueprism, preferrably using an opensource language?
(only interested in windows-based apps for now)
The aim is of course to create robots able to navigate the apps as a human would do.
I guess you are using selenium for web-browsers
There is also some projects for windows based applications working with a Windows Driver.
Take a look to the project on github, it may be what you are looking for.
https://github.com/2gis/Winium
https://github.com/microsoft/WinAppDriver
Autoit: https://www.autoitscript.com/site/
It also comes with an identify mode for application elements and is has a big community
There is a free version of Blue Prism now :) Also Blue Prism uses win32, active accessibility and UI Automation which is a newer for of the older active accessibility.
To do this yourself without looking into Blue Prism you would need to know how to use UIA with C#/VB.new or C++. There are libraries however given that Blue Prism now has a free version I would recommend using that. Anything specific can be developed withing a code stage within Blue Prism.
Related
I need to interact with a windows application by clicking a button in my application. More precisely: how can I write in a textbox or scroll a dropdown menu?
There are multiple ways to do this depending on what kind of application you are automating. If you are automating a WPF application I would suggesting using Microsoft UI Automation (UIA). There are some nice wrappers written around UIA like TestStack.White and FlaUI. FlaUI is the more modern of the two and supports UIAv3 using a COM wrapper. TestStack.White is built on top of a UIAv2 using the managed wrapper in the .NET Framework which is no longer supported.
If you are automating anything else besides a WPF application you can do straight PInvokes to SendMessage. I would suggest staying away from that method and using the Microsoft UIA framework since sending windows messages can get quite verbose. If you really want to go with the Win32 route I suggest using something like AutoIT to automate your application.
Lastly, you will want to download the Windows SDK and run Inspect.exe from that. Inspect.exe is the application that will allow you to see the properties you are attempting to query and the patterns that are available. Applications like AutoIT have essentially built their own Inspect.exe or reference and application call UISpy which is also in the Windows SDK but mostly superseded by Inspect.exe.
System.Windows.Automation is EXTREMELY slow.
I execute:
element.FindAll(TreeScope.Children, Condition.TrueCondition);
Obtaining only 30 child elements may take 1000ms on a very fast computer.
I have even seen it hanging forever while getting the child elements of a Tree in a QT application.
Is this a known problem?
I cannot find any usefull answer after googling a lot.
System.Windows.Automation is EXTREMELY slow.
System.Windows.Automation is full of bugs. It may not return all children of an AutomationElement, which is a very severe bug.
Apart from that the implementation is not thread safe.
System.Windows.Automation is deprecated. Do not use it!
In the MSDN you find the following note:
UI Automation was first available in Windows XP as part of the
Microsoft .NET Framework. Although an unmanaged C++ API was also
published at that time, the usefulness of client functions was limited
because of interoperability issues. For Windows 7, the API has been
rewritten in the Component Object Model (COM).
Although the library functions introduced in the earlier version of
UI Automation are still documented, they should not be used in new
applications.
The solution to slow performance is to use the new IUIAutomationElement COM interface instead of the old System.Windows.Automation C# interface. After that the code will be running lightning fast!
Apart from that the new interface offers much more patterns and Microsoft is extending it continously. In the Windows 10 SDK (UIAutomationClient.h and UIAutomationCore.h) several patterns and properties have been added which are not available in the .NET Automation framework.
The following patterns are available in the COM version of UIAutomation which do not exist in System.Windows.Automation:
IUIAutomationLegacyIAccessiblePattern
IUIAutomationObjectModelPattern
IUIAutomationAnnotationPattern
IUIAutomationTextPattern2
IUIAutomationStylesPattern
IUIAutomationSpreadsheetPattern
IUIAutomationSpreadsheetItemPattern
IUIAutomationTransformPattern2
IUIAutomationTextChildPattern
IUIAutomationDragPattern
IUIAutomationDropTargetPattern
IUIAutomationTextEditPattern
IUIAutomationCustomNavigationPattern
Additionally the following Control types have been added:
AppBar
SemanticZoom
Additionally the following Element's have been added:
IUIAutomationElement2
IUIAutomationElement3
IUIAutomationElement4
And what concerns the bugs: The new COM UIAutomation Framework is very well designed and I could not find bugs on the client side of the framework which is a great progress compared to System.Windows.Automation. But several missing features and even bugs on the server side of the framework. On the server side each GUI framework must implement an UIAutomation provider (see MSDN: Interfaces for Providers). So these problems differ depending on what type of application you are automating because each GUI framework has it's own problems:
In the Native Windows GUI features are missing: Lots of controls do not implement the patterns that they should implement. For example a SplitButton in a native Toolbar should implement the Invoke pattern to click the button and the ExpandCollapse pattern to open the drop-down menu. But the ExpandCollapse pattern is missing which makes it difficult to use SplitButtons. If you obtain a Toolbar SplitButton by IUIAutomation->ElementFromPoint() and then ask for it's parent you will get a crippled element. And the Pager control cannot be automated at all.
Also in WPF applications there are controls that are implemented buggy by Microsoft: For example if you have a Calendar control you see two buttons at the top to switch to the next/previous month. If you execute the Invoke pattern on these buttons you will get an UIA_E_NOTSUPPORTED error. But this is not a bug on the client side of the framework, because for other buttons the Invoke pattern works correctly. This is a bug in the WPF Automation server. And if you test IUIAutomationTextRange with a WPF RichTextBox, you will find that several commands are not implemented: Select() and ScrollIntoView() do simply nothing.
For .NET Forms applications Microsoft did not make much effort to support them. The .NET Calendar control cannot be automated at all. The entire control is not even recognized as Calendar. It has the ControlType "Pane" with no child elements in it. The same applies to the DateTimePicker. And for complex controls like DataGrid and PropertyGrid the only implemented pattern is LegacyIAccessible which is a poor support. These controls should implement at least the Table and the Grid and the ScrollItem pattern.
Also Internet Explorer cannot be automated because elements outside the visible area cannot be scrolled automatically into view due to missing coordinates. (The Bounds are returned as an empty rectangle) And the ScrollItem pattern is not implemented. (Yes, I know that Internet Explorer has been replaced with Edge in Windows 10, but the UIAutomation framework exists since Windows 7 and Microsoft did not implement a usefull automation support in Internet Explorer in all these years)
I saw even complete crashes of the automated application. For example Visual Studio and TotalCommander will crash if you execute certain automation commands on a certain control. Here - once again - the bug lies in the server side implementation of the framework.
Summary: We have a great framework with limited usefullness. The Microsoft team that developed the new UIAutomation framework did a great job, but the other areas in Microsoft (the native GUI, WPF, .NET and Internet Explorer team) do not support this framework. This is very sad because only a small effort would have to be made to offer a better functionality. But it seems that the users who use UIAutomation in the first place (handicapped people) are not a profitable market.
Short story: is there a way to write a desktop application with a GUI in HTML5 and core in a cross-platform language like python (or even C#/Mono)?
Longer story: I'm a C# developer, for small personal projects I seldom do, running both under Windows and OSX, I use C# (Mono) with a frontend leveraging on Eto.Forms
I'd like to understand if there's a mature way to achieve the same results using an HTML5 GUI, since I'd like to learn that and believe it could be a good option for near-future Windows desktop UIs (or otherwise a nice tool to have in my skillset). Of course if the code running behind the scenes is C# I'll be more than happy, but also getting my feet wet in another, maybe more cross-platform like python would be good.
At this stage I'm not interested in any mobile-oriented solution.
Electron (formerly Atom shell) has really matured as of late. In fact it's what VSCode is built on.
There's a great tutorial and starter code on using Angular 2 and Typescript, and you can even use VSCode to write and build it.
For me this is the best way to transition from the WPF world to HTML5.
the NW.js look pretty promising... you might even be able to use TypeScript which would be much closer to C# than plain ol' js. If you're open to using PHP, you can check out the nightrain project https://github.com/naetech/nightrain.
Give http://www.tidesdk.org/ a try.
Your app will run on Windows 8, MacOS and Linux. You can use HTML5, Javascript and CSS3.
But you can also extend the functionality of your app with a scripting language you are comfortable with. TideSDK currently supports Python, PHP, and Ruby.
I have recently worked with Chromium Embedded Framework, basically a browser component for WPF an WinForms. It works very well and provides kind of a two-way interoperability from website to .NET-app and vice versa. Basically, you:
Create a WPF desktop app
Include CEFSharp and place a full-screen browser on the window
Call methods in JavaScript:
// .NET
var mainFrame = browser.GetMainFrame();
mainFrame.ExecuteJavaScriptAsync("any js code");
Bind a .NET-object
// .NET
browser.RegisterJsObject("boundObject", this);
Call methods on a bound .NET-object from the website/JS:
// JS
boundObject.someMethod();
On this basis you could build a mediation layer (ViewModels, controllers, ...) between HTML/JS-UI and .NET logic...
I wrote an APP with http://kivy.org/ it is capable to create apps for different systems.
Qt node https://github.com/arturadib/node-qt seems also interesting, but i did not test it myself.
And last https://chrome.google.com/webstore/launcher
You can create web apps for chrome, which should run in supported systems.
Kivy is a Python solution. Qt node is maybe what you are looking for.
Here are some nice tutorials for kivy:
https://www.youtube.com/playlist?list=PLQVvvaa0QuDe_l6XiJ40yGTEqIKugAdTy
NW.js
But it's Javascript (node), not python nor C#.
A very interesting project I think is Chromium Embedded Framework. You basically embed a (stripped down) web browser in your application. For python, many GUI Toolkits are supported. Check this for more information.
Since you are used to C#, maybe Java with JavaFX and FXML is an option. FXML is not HTML but you can style it with CSS as well. You can also use Scene Builder with it for faster UI creation. Many JVM languages support this toolkit so Jython instead of Java will also work. I will recommend Java however, because the support for other languages, while there, is not perfect yet.
Using C# and the Windows App Store WebView is also an option. You can check MSDN for more information.
I am sure there are other options (Kivy, Node.js, etc.) as well. Some of them are already mentioned in this thread.
I would recommend Node-Webkit which is based on nodejs.You can still use some python scripts to do some backend job integrated with Node-Webkit which is easy to deal with.I've already saw some successful applications using this(like wunderlist).TideSDK is another choice but the python support in TideSDK is not mature enough.The Node-Webkit project hosted at
https://github.com/nwjs/nw.js/
I'm looking for a library that can be used in native .NET code, just like any .NET assembly. The purpose of the library must be to automate Windows (push a button, select a window, send keys, record & playback, that sort of thing).
So: the library is supposed to be used natively in .NET, but the automation itself must be able to target any native or .NET Windows application that can receive user input.
Suggestions so far:
benPearce suggested AutoIt. It has a DLL, which is native Win32 but not native .NET and cannot be used without use of .NET Interop.
Chris Dunaway suggested Global Mouse Keyboard Lib. This came closest, but is not an automation lib. It just helps setting up keyboard and mouse hooks.
pm100 suggested Microsoft's WPF UI Automation. This one is pretty good, albeit that it's not available if you develop in .NET 2.0 and it requires the WPF to be installed on the system. It can, however, automate everything from Win32 apps to HTML in a browser.
JasonTrue suggested WebAI from ArtOfTest. This is a testing framework mainly geared towards browsers and web applications. It is unfortunately not well suitable for use for Windows automation.
If nothing else appears available, I'll probably choose Microsoft's UI Automation and upgrade any projects that require it that are still in .NET 2.0 to .NET 3.5, if possible. But I hope for a more widely applicable automation framework (.NET prior to 2.0 does not need to be supported).
Have you looked at the White framework?
I have used AutoIt in the past with success.
microsoft's own built in one is fine
http://msdn.microsoft.com/en-us/library/ms747327.aspx
not restricted to wpf as some seem to think.
If you haven't seen it yet, and a commercial library is acceptable, you might check out Ranorex:
http://www.ranorex.com/
I used Ranorex 1.5 quite a bit to write small C# UI automation utilities. It was pretty effective! Development seemed faster compared to using the MS UI Automation API directly, since Ranorex has a lot of useful convenience methods already available.
I haven't used Ranorex 2 very much yet, though.
In Ranorex 1.5, there was also support for traditional Win32 development in C++, but I didn't use it. As far as I know, that's still available in Ranorex 2.
I can't speak to the quality of the record/playback support in Ranorex since I never used that feature.
One final plus: Their support team was really responsive and helpful anytime I emailed them.
This library is pretty interesting and is fairly simple. Perhaps it will help you.
Check out Tools for automated GUI testing on windows
I would still suggest FlaUI for autoamating .Net Desktop,Mobile apps. Its based on Microsoft UIA libraries and have support for external controls like the DevExpress Grid too
Moreover, it is built on top of TestStack.White so indeed a very good library and has a github page also
I have used WebAii from ArtOfTest with a fair degree of success in automating integration testing for a Silverlight app. It also supports WinForms and Web applications.
Microsoft UI Automation, the successor to Active Accessibility, can do almost all of the Windows UI automation you would need.
How about CSharpScript, here's an article about it on Codeproject, and here's the link to the main website. Furthermore, it is familiar C#, scripted which can be used to automate anything.
Hope this helps,
Best regards,
Tom.
I've inherited a C# window's application that I'm not real crazy about. I've got a looming deadline and I'm scared to death that some of my changes might be having adverse effects on existing functionality.
I've got a hobbyist background to RoR and I'm fairly comfortable with testing in that framework (using both RSpec and Cucumber).
I love having test scripts that can be ran on a regular basis and I'm willing to spend my personal time developing those for this particular project. I purchased a book from PragProg.com on scripted GUI testing with Ruby (http://pragprog.com/titles/idgtr/scripted-gui-testing-with-ruby). So far, I'm digging what I'm seeing and I think that this should work well.
Unfortunately, I've got a fundamental lack of understanding concerning Windows app development. I'm making calles to FindWindowEx (via Win32API) to "attempt" to retrieve sub-controls in my application.
A big part of my confusion is how I should retrieve the Class Name of the control that I'm trying to capture. The example provided in the text is as follows:
edit = find_window_ex.call #main_window, 0, 'ATL:00434310', nil
Where #main_window is my application's main window handle, and 'ATL:...' is the class of a text box area. There is no explanation given as to how the author arrived at 'ATL:...'.
I've read some very old posts concerning MS's SPY++, but those seem to be obsolete (or for some reason it wasn't installed when I installed vs2010).
So, what's the best way for me to find control classes to be used with the findWindowEx call? I do have the source code - should I be pulling from there? What if I don't have the source code and I want to automate an application? Is there a utility that allows you to somehow "browse" controls on a running application?
Sorry for the length - thanks in advance for the help!
Bob
The best is for you to install the components so that you get Spy++, this is the best way I know of to get to the actual class names esp. if you do not have the source to the original controls, which might be from a library or possibly some standard ActiveX controls that Microsoft ships.
The ATL class name is probably for controls developed using Microsoft Active Template Library (ATL), this is a C++ template library which significantly simplifies the development of ActiveX controls, and COM objects etc. in C++.