I have a specific scenario where parts of an Internet Explorer are accessible through COM and IHtmlElement, but not (directly) through the GUI Tree of AutomationElements. (I have to use the tree/hierarchy to find my element. And the GUI Tree of that particular Internet Explorer is broken (don't know whether it is through customization or because it is an old version - it is a given).)
Now there are properties of the AutomationElement that I need and cannot get off the IHtmlElement (actually, getting properties off IHtmlElement for this version of IE is broken, but getting them off AutomationElement works).
My naive approach is to simply get the coordinates of the IHtmlElement (recursively via offsetParent), and then convert them to an AutomationElement using AutomationElement.FromPoint(x, y). This of course is not really stable (though working kind of ok).
I tried converting through Window Handles, but.. HTMLElements seem to only possess the Window Handle of the outmost element (else websites would use up a lot of handles, I understand).
I tried some more "fancy" approaches trying to get (again) a COM object with IID_IAccessible interface (which seems to work if I use the Main Document as a IServiceProvider), but unfortunately, i do not know later how to make an AutomationElement out of my IAccessible object (idea from IHTMLElement -> IAccessible ).
Any help with this approach or my general problem as stated above is very welcome!
Some rough code to give you an idea on my approach (though now I see that I should not use mMainDocument in the first place, but rather my current element - but this fills ae with null):
Guid IID_IAccessible = new Guid("618736E0-3C3D-11CF-810C-00AA00389B71");
IServiceProvider sp = (IServiceProvider)mMainDocument;
Object ae = null;
sp.QueryService(ref IID_IAccessible, ref IID_IAccessible, out ae);
System.Windows.Automation.AutomationElement ae2 = System.Windows.Automation.AutomationElement.FromLocalProvider((System.Windows.Automation.Provider.IRawElementProviderSimple)ae);
// ae is no AutomationElement, and the last line (with ae2) fails
Related
given 2 applications:
1 WPF application, where not all controls have a AutomationId, and which I cannot change. In addition the application adds controls at runtime without setting the AutomationId.
1 console application which automates the above WPF application
I need that, because I want to access all elements within nearly the same amount of time. (searching for an automationelement right before using seems to differ a lot (from ms to s) in the time - depending on the amount of elements / and tree-tiers)
I would like to set AutomationIds of WPFs controls within the console application during WPFs runtime. Would be great to know, if you can think of any solution for this problem!
What I have tried until now:
1.)Reflections
Type type = element.GetType();
FieldInfo fi = type.GetField("AutomationIdProperty");
fi.SetValue(element, "x"); //Error
Error message: "Object of type 'System.String' cannot be converted to type 'System.Windows.Automation.AutomationProperty'" But I would like to hand over a value, not a property type...
if I use the following instead, it throws no error, but changes nothing in the XAML
fi.SetValue(element, AutomationElement.AutomationIdProperty);
2.)directly
AutomationElement element; // = my AutomationElement
element.Current.AutomationId = "x"; //since AutomationId is Readonly - its not possible
3.) DependencyObjects and DependencyProperties seem also promising, but I couldn't come up with an solution so far. Does someone have experience with that??
IDK how it is possible but WPF Inspector is exactly able to do what I was looking for (now I need to know how they attach to the WPF application :) ).
____OLD ANSWER____
It seems impossible to change the XAML of other programs. If the developers are "too lazy" to set the AutomationId, I've come up with a alternative solution.
The automation app iterates over all controls in the beginning, giving them unique names which are stored in a dictionary, together with their references. In case a component gets added/deleted/changed in the hierarchy, the component and their descendants get deleted in the dictionary and the app re-iterates over this sub-tree again.
In my test application, I am constantly opening and re-opening a form. Everytime the form is opened, I must get all the elements on the form into an AutomationElementCollection so that I can operate on the elements. However, it seems expensive to repeatedly get these elements (due to tree navigation/context-switches etc.).
I attempted to set a boolean around the method that gets the elements. If the method was called for the first time, it would run normally, and set the boolean to true. If the method gets called a second time it will do nothing, as the array has already been populated.
However when I try to perform operations on any AutomationElement in the array (for a second time), the elements do not seem to be available. Does closing the form somehow "disable" these elements? Do I HAVE to find these elements each time I open the form, so that they are "fresh"?
I looked at the CacheRequest way, but that seems to only pertain to accessing properties/patterns, not elements.
Here is code/error message:
AutomationElement GAP;
AutomationElementcollection GAP1;
private bool initGAP1 = false;
public void initGAP()
{
if (!initGAP1)
{
int refnum = ...;
int refnum2 = ...;
AutomationElementCollection temp = MMChildren[refnum].FindAll(TreeScope.Children, findCondition);
GAP = temp.FindAll(TreeScope.Children, findCondition)[refnum2];
GAP1 = GAP.FindAll(TreeScope.Children, findCondition); //this contains the elements I want to operate on
initGAP1 = true;
}
}
System.Windows.Automation.ElementNotEnabledException: Exception of type 'System.Windows.Automation.ElementNotEnabledException' was thrown.
You would need to re-get the Automation Elements for each new window. As I understand the UI Automation framework it gives you the means to investigate running windows. It will collect information with different techniques, depending on what kind of framework the target application uses. In your case, if you create and destroy instances of windows, they are treated as different AutomationElements since they are different windows (basically they have different window handles in the OS). Even if the underlying controlling code is the same they are different instances towards the OS, and therefore UI automation.
If you experience that you are suffering from the performance in the traversion, it might be worth considering to use the UI Automation COM API instead, that is vastly faster on some operations.
I would like to be able to:
compare Word Interop COM proxies on a "reference equality" basis; and
map from a specific object (say a paragraph) to the collection it comes from, OR at least
determine whether two paragraphs are from the same section and which one comes relatively before the previous one
Why do I want to do this? I am trying to build a Word Add-In that acts similarly to a spell-checker in the sense that it runs in the background (by background I mean by regularly stealing time from the main Word thread using SendMessage) and scans the document for certain text "tokens". I want to be able to keep a collection of the tokens around and update them as the document changes. A specific example of this is if the user edits a given paragraph, I want to rescan the paragraph and update my data structure which points to that paragraph. If there is no way to map between the paragraph the user edited in (i.e. the paragraph where the start of the selection range is) and a paragraph that I have "stored" in a data structure, I can't do this.
Example Code for item #1, above
If I write the following VBA code:
Dim Para1 As Paragraph
Dim Para2a As Paragraph
Dim Para2b As Paragraph
Set Para1 = ActiveDocument.Paragraphs(1)
Set Para2a = Para1.Next
Set Para2b = Para1.Next.Next.Previous
If Para2a Is Para2b Then
Debug.Print ("Para2a Is Para2b")
Else
Debug.Print ("Para2a Is Not Para2b")
End If
Then I am getting the output:
"Para2a Is Not Para2b"
Which is perhaps physically true (different COM proxies) but not logically true. I need to be able to compare those paragraphs and determine if they are logically the same underlying paragraph.
(I am planning to write the add-in in C#, but the above VBA code demonstrates the kind of problem I need to overcome before doing too much coding).
For items 2 and 3 above, hopefully they will be self-explanatory. Say I have a paragraph (interop proxy) reference. I want to figure out "where" it is in the document. Does it belong to Section 1? Is it in a footer? Without this ability, all I can reasonably do to obtain an idea of where things come from is rescan the entire document every time it changes, which is of course absurdly inefficient and won't be timely enough for the app user.
Any thoughts greatly appreciated! I'm happy to post additional information as needed.
Navigating the particulars of reference equality in the context of COM Interop is always an interesting exercise.
I wouldn't be privy to the implementation details of the Paragraph.Next() and Paragraph.Previous() methods, however the behavior they exhibit is very similar to how COM-based collections act in general in regards to Runtime Callable Wrapper creation.
Typically, if possible, the framework avoids creating new RCW instances in response to additional references being made to COM objects that already have an RCW initialized and assigned. If an RCW already exists for a particular pointer to IUnknown, an internal reference count maintained by that RCW is incremented, and then the RCW is returned. This allows the framework to avoid incrementing the actual COM object's reference count (AddRef).
COM-based collections, which are COM objects that have managed representations implementing IEnumerable, seem to generate a new RCW each time an item is accessed, even if that item has already been accessed during the session.
For example:
Word.Document document = Application.ActiveDocument;
Paragraphs paragraphs = document.Paragraphs;
Paragraph first = paragraphs[1];
Paragraph second = paragraphs[1];
bool thisIsFalse = (first == second);
If you want to do any sort of "reference equality" checking, you need to escape from the COM based collection, specifically in your case: the Paragraphs object. You can do this simply by grabbing its kids and storing them in your own, purely managed and predictable collection, like so:
List<Paragraph> niceParagraphs = paragraphs.Cast<Paragraph>().ToList();
Although using LINQ with COM Interop may look a bit scary (if it doesn't to you...it really should!) I'm fairly certain the above code is safe and will not leave any dangling references out there, or anything else nasty. I have not tested the above code exhaustively, however.
Don't forget to properly release those resources when you are done with them, at least if your requirements require that level of prudence.
I'm making a jquery clone for C#. Right now I've got it set up so that every method is an extension method on IEnumerable<HtmlNode> so it works well with existing projects that are already using HtmlAgilityPack. I thought I could get away without preserving state... however, then I noticed jQuery has two methods .andSelf and .end which "pop" the most recently matched elements off an internal stack. I can mimic this functionality if I change my class so that it always operates on SharpQuery objects instead of enumerables, but there's still a problem.
With JavaScript, you're given the Html document automatically, but when working in C# you have to explicitly load it, and you could use more than one document if you wanted. It appears that when you call $('xxx') you're essentially creating a new jQuery object and starting fresh with an empty stack. In C#, you wouldn't want to do that, because you don't want to reload/refetch the document from the web. So instead, you load it once either into a SharpQuery object, or into an list of HtmlNodes (you just need the DocumentNode to get started).
In the jQuery docs, they give this example
$('ul.first').find('.foo')
.css('background-color', 'red')
.end().find('.bar')
.css('background-color', 'green')
.end();
I don't have an initializer method because I can't overload the () operator, so you just start with sq.Find() instead, which operates on the root of the document, essentially doing the same thing. But then people are going to try and write sq.Find() on one line, and then sq.Find() somewhere down the road, and (rightfully) expect it to operate on the root of the document again... but if I'm maintaining state, then you've just modified the context after the first call.
So... how should I design my API? Do I add another Init method that all queries should begin with that resets the stack (but then how do I force them to start with that?), or add a Reset() that they have to call at the end of their line? Do I overload the [] instead and tell them to start with that? Do I say "forget it, no one uses those state-preserved functions anyway?"
Basically, how would you like that jQuery example to be written in C#?
sq["ul.first"].Find(".foo") ...
Downfalls: Abuses the [] property.
sq.Init("ul.first").Find(".foo") ...
Downfalls: Nothing really forces the programmer to start with Init, unless I add some weird "initialized" mechanism; user might try starting with .Find and not get the result he was expecting. Also, Init and Find are pretty much identical anyway, except the former resets the stack too.
sq.Find("ul.first").Find(".foo") ... .ClearStack()
Downfalls: programmer may forget to clear the stack.
Can't do it.
end() not implemented.
Use two different objects.
Perhaps use HtmlDocument as the base that all queries should begin with, and then every method thereafter returns a SharpQuery object that can be chained. That way the HtmlDocument always maintains the initial state, but the SharpQuery objects may have different states. This unfortunately means I have to implement a bunch of stuff twice (once for HtmlDocument, once for the SharpQuery object).
new SharpQuery(sq).Find("ul.first").Find(".foo") ...
The constructor copies a reference to the document, but resets the stack.
I think the major stumbling block you're running into here is that you're trying to get away with just having one SharpQuery object for each document. That's not how jQuery works; in general, jQuery objects are immutable. When you call a method that changes the set of elements (like find or end or add), it doesn't alter the existing object, but returns a new one:
var theBody = $('body');
// $('body')[0] is the <body>
theBody.find('div').text('This is a div');
// $('body')[0] is still the <body>
(see the documentation of end for more info)
SharpQuery should operate the same way. Once you create a SharpQuery object with a document, method calls should return new SharpQuery objects, referencing a different set of elements of the same document. For instance:
var sq = SharpQuery.Load(new Uri("http://api.jquery.com/category/selectors/"));
var header = sq.Find("h1"); // doesn't change sq
var allTheLinks = sq.Find(".title-link") // all .title-link in the whole document; also doesn't change sq
var someOfTheLinks = header.Find(".title-link"); // just the .title-link in the <h1>; again, doesn't change sq or header
The benefits of this approach are several. Because sq, header, allTheLinks, etc. are all the same class, you only have one implementation of each method. Yet each of these objects references the same document, so you don't have multiple copies of each node, and changes to the nodes are reflected in every SharpQuery object on that document (e.g. after allTheLinks.text("foo"), someOfTheLinks.text() == "foo".).
Implementing end and the other stack-based manipulations also becomes easy. As each method creates a new, filtered SharpQuery object from another, it retains a reference to that parent object (allTheLinks to header, header to sq). Then end is as simple as returning a new SharpQuery containing the same elements as the parent, like:
public SharpQuery end()
{
return new SharpQuery(this.parent.GetAllElements());
}
(or however your syntax shakes out.)
I think this approach will get you the most jQuery-like behavior, with a fairly easy implementation. I'll definitely be keeping an eye on this project; it's a great idea.
I would lean towards a variant on option 2. In jQuery $() is a function call. C# doesn't have global functions, a static function call is the closest. I would use a method that indicates you're creating a wrapper like..
SharpQuery.Create("ul.first").Find(".foo")
I wouldn't be concerned about shortening SharpQuery to sq since intellisense means users won't have to type the whole thing (and if they have resharper they only need to type SQ anyways).
I got this error when trying to update an image.
It was a cross-thread update, but I used .Invoke(), so that shouldn't be the problem, should it.
(Answering my own question, for others, and for future reference)
I think (not yet entirely sure) that this is because InvokeRequired will always return false if the control has not yet been loaded/shown. I have done a workaround which seems to work for the moment, which is to simple reference the handle of the associated control in its creator, like so:
var x = this.Handle;
(See http://ikriv.com:8765/en/prog/info/dotnet/MysteriousHang.html - down? cached version)
(Related question: Boiler plate code replacement - is there anything bad about this code?)
If the handle doesn't yet exist, you can force it by subclassing the control and calling CreateHandle; however, the bigger question is: why are you doing things with a form that hasn't been loaded? Personally I'd only start such an operation after Load.