I am starting some research on a project that involves navigating the web with speech feedback from SAPI. This application will be used by the blind and visually impaired to navigate the web with full speech feedback of current selected object and the ability to read the entire page and fill out forms. I have a mass amount of experience in SAPI and the accessibility word, however I'm not so familiar with HTML and the best approach on taking on this project and i would like to ask for guidance from someone that may have experience in this area. The basic concepts, which I've already explained a little is to enable a blind user to use this standalone product without a screen reader and have the ability to navigate the web page with a keyboard and receive speech feedback from a synthesizer i have created. I can use the Accessibility API's built into windows to display location for low vision users.
Thank you!
IE supports all accessibility interfaces (I believe all other browsers do too - sample on SO ).
There are 2 parts - code access using IAccessible and related interfaces and good HTML (i.e. correct use of semantic tags like UL/LI) markup with possible use of additional CSS accessibility aria-XXXX attributes (i.e. aria-expanded)
Possible starting link - What's New for Accessibility in IE8
I have come to the conclusion in order to get a talking web browser i actually had to hook an instance of the WebBrowser Control with Set setwindowshookex. In my setwindowshookex callback i monitor for the event: OB_FOCUS (focus changed, could be keyboard or mouse) i then pass the object instance that has focus to the AccessibleObjectFromEvent API. Next, i get the IAccessible interface to this object and then enumerate any children if present and compare them against a known object list that relates to HTML/Web objects to use the proper class that will parse this field and announce text if available, if the user is in an editable field, focused buttons, etc... I also use the IAccessible class to get the objects location and present location to the user by drawing a rectangle on the current focused object. I have also created a class that extracts just link, text, removes images and more to make for an easier, more enjoyable web experience for people with visual impairments. I still have some quarks to work out, but progress is moving forward!
Related
I've started using FlaUI for Automating my thick client .net application. The application is Windows Form based. The start was good and Login Form was identified and I could Login, but after that came the dead end and I found that almost everything in the application is developed as Pane control type.
So, there is grid, table etc. but they all just appear as Pane type when I see the object hierarchy using Inspect.exe or FLAUInspect tools. And nothing really appears in thier property, so it seems that nothing could be read. But before giving up I just wanted to check with experienced audience on this forum if there is really any way to get the data from Pane objects.
Please suggest if there is any way, even that means using other libraries like UIAutomation, TestStack.White, etc.
UPDATE: I now understand little more about this. So, the objects that are there in the pane are developed in syncfusion and devexpress. Is it possible to identify objects developed in syncfusion and devexpress using FlaUI or UIAutomation or TestStack.White, etc ?
I don't know if you have already tried the following steps. Have you add automationId's to your objects in xaml code with:
AutomationProperties.AutomationId="AnyID"
In the testcode, first initialize the main window of the application.
MainWindow = fApplication.GetMainWindow(fAutomation, null)?.AsWindow()
After that you can find your objects by the automationId's, like:
MainWindow .FindFirstDescendant(cf => cf.ByAutomationId(AnyID))
I did it this way, and don't have to know the hierarchy of my application. Maybe this will work?
Most UI Frameworks nowadays fully support UI Automation. So first make sure that you have a recent version of your framework (syncfusion, devexpress). In addition, some frameworks provide settings to enable UI Automation. Like for devexpress, you need to set
ClearAutomationEventsHelper.IsEnabled = false;
at the start of your application to test so it exposes way more things (like tabs) to FlaUI.
The Problem
In the .NET Framework, it would just be a matter of setting a property true and hooking to the link clicked event. But how would I do this in WinRT using the limited .NET framework.
Why Is This Necessary?
Well I want to create a text based web browser (like Lynx but with a GUI), that is designed with people who don't want to use a fancy web browser or just need accessibility. Some of the features would be to display the URLs for instance 'http://stackoverflow.com/' to a URL in which I hookup the link clicked event and then redirect within the program and display the text.
Conclusion
Requesting the HTML and converting it to text, was simple, and I already have it done and working fine. So the only thing I need from the Stack Overflow community. Is whether the RichTextBox support hyper-links? and how can it be done? No code examples necessary, but pointers will just do. I have already done research, but Metro Style applications are just fresh out of the oven; not a lot of articles out there.
The Windows-8 App style way of doing this is to put a button inside an InlineUIContainer inside a textblock.
See here for some related discussion.
I have an external windows application (no source code) that has a grid within it. This runs as a separate process. When the user selects a cell within the grid via mouse click, I need to be able to read the value within that cell. Can anyone provide some direction on what API's I would need to use to be able to trap and listen to the events?
You best option is UI Automation Overview or accessibility as older technology.
Also you could take a look at this Pinvoke SetFocus to a particular control on how to invoke things on another process (pretty much unrelated to automation, automation works w/o that)
UI Automation is the best tool for the job, however, the downside is that not every app supports that - so this very much depends on the app you're targeting. Some support only the legacy acessibility (IAccessible, IAccessible2 etc.), usually there is a 'combined' approach. Older techniques don't work very good any more but you could try traversing windows, child windows in the target window (for that direction you'll probably need the above technique sooner or later) and hoping you could get it from standard controls, windows text, via messages etc.
Im a c# developer and I believe that what I want to achieve is going to move out of the realms of some drop in .NET component so I am looking for advise on what I use externally which .NET can inter op with.
My requirements are to have an embedded web browser control in a WPF/Winforms applciation BUT I will also need to keep track of the following:
User interaction i.e. what pages they visit, forms submitted where they click etc.
DOM manipulation and traversing
I am guessing here but it seems that I might need to start looking at open source html/web browsers out there like WebKit etc. Is this the right track or is there anything currently available in the form of a control/COM object that I can use directly.
Cheers, Chris.
You should be able to accomplish point 1 using the webbrowser control in Visual Studio, but I dont think DOM manipulation is available, i do know you can traverse all the tags.
reference
I'm searching a C# component or code snipped that does something like that:
I want to inform new users about the most important program functions if he opens a new window for example.
It should be a box showing text (formated if possible) that is of course not modal and has some mechanism to 'go out of the way' if the user enters the textbox area. So that he can access what's underneath it. Alternativly the window could also stick to the border of the window, but there needs to be a way that this also works if the window is maximized.
So I want to present him with a short introduction of what he can do in every corner of my app most painlessly.
Thank you!
I use a "bar" at the top of every window to display some information about the current window/dialog.
Use tooltips. They can be programmatically controlled, and you can have them appear at will. You'll need to add the functionality to your app to keep track of what tooltips have been shown to the user already.
You can add a "balloon" style by setting the IsBalloon property to true.
You can also replace them with smaller descriptions for when the user wants to hover over the control and have them displayed again.
I'm already using tooltips heavily. However, they aren't very practical when displaying bigger amounts of data and they are bound to specific user actions.
Have you considered having a contextual menu for each form / page which contains links to Adobe Captivate style presentations for each available task? That way the user can investigate an example of how to achieve a task relating to what they are trying to achieve from within the application / site.
This approach would require a good deal of maintenance and management if your code changes regularly but coordinating it with a training department can provide rich help features in your application.
See http://www.adobe.com/products/captivate/ for more information.