How to use existing Windows functionality to extract text from the UI - c#

I've done a bit of looking around and found various bits and pieces relating to this, but nothing concrete.
I need to find a method of extracting UI elements other than that of the Spy++ tool. I'm able to locate screen items and their underlying text captions based on HWND, however 3rd party apps such as Firefox offer further problems as they only have one large window for the display. If anyone has any ideas on how to natively get screen coordinates to do an OCR or control recognition of UI elements within, say, a web page I'd love to hear from you.

If what you want programmatic access a web page as Firefox sees it, there are much better ways.
Try Watir or selenium. These are for testing web applications, and they allow you to problematically control the web browser.

Related

Get the content of any active window

I am struggling to find a reliable way to get the content/text of the window that is currently in the foreground. It should be able to determine the text from every possible program that a user is currently using, if possible
What I tried:
Take a screenshot of the currently active window, apply some filters and run an OCR algorithm (tesseract .Net wrapper). This works, but takes a long time and is not very accurate.
Then I tried some Windows API functions (FindWindow and SendMessage), as described here. I could make it run for the standard Editor (notepad) for example, but not for most other programs
I also tried to make it work with AutoHotKey and the WinGetText function and again a .Net Wrapper. Here, I just get some info about the window, but in no way the text of it...
Unfortunately, now, I don't have any other idea what to do as I am stuck in every way... Does someone have experience with this or knows a way that works? Any suggestion is really much appreciated
It will be difficult to find a single solution to retrieve text from applications. Different methods for different programs will be required.
For AutoHotkey, AccViewer, which makes use of Acc.ahk is the best method of first resort. Acc works on a large variety of controls and also elements within controls, it can cover far more control types than AutoHotkey's ControlGet command.
Acc Library [AHK_L] (updated 09/27/2012) - Scripts and Functions - AutoHotkey Community
https://autohotkey.com/board/topic/77303-acc-library-ahk-l-updated-09272012/
Accessible Info Viewer - Alpha Release (2012-09-20) - Scripts and Functions - AutoHotkey Community
https://autohotkey.com/board/topic/77888-accessible-info-viewer-alpha-release-2012-09-20/
A link describing some further text retrieval methods:
AutoHotKey ControlGet
Note also:
COM (Component Object Model), is handled natively by AutoHotkey. It can be used to retrieve the text from web elements in Internet Explorer, and via VBA code, text can be retrieved from MS Office programs such as MS Excel and MS Word.

how to create a webbroswer in wpf that supports all the sites and scripts and play video?

I have to create a webbroswer in wpf that supports all the sites and scripts and play video , I have created but some site got crashed and not supporting .
Please provide solution.
What you ran into are probably error dialogs caused by JavaScript script errors. WPF WebBrowser control currently doesn't support disabling these dialogs and ignoring the errors. The WebBrowser control is based on Internet Explorer core and will have a hard time dealing with broken JavaScript, webpages will also sometimes report the web browser as old and inform the visitor about it, like FlipKart does.
One solution is to use WindowsFormsHost and host a Windows Forms equivalent - WebBrowser control - in it. The WF WebBrowser offers a property called SupressScriptErrors, which is not implemented by the WPF WebBrowser. That will hide the error dialogs for you, but you will still occasionaly receive a warning from the website saying they don't support the web browser used. That is out of your control unless you want to take a route of hacking the control to send a different User-Agent string and that will only fool broken UA detection, feature detection will still fail for things like HTML5 video, WebRTC and the like. More on this here.
You can also choose to embed Gecko (Firefox rendering engine) or Chromium. see here for more information on embedding alternative rendering cores as an alternative to IE COM wrapper offered by the framework.
Edit: this is also a possible solution, as it would seem. You will have to test it for yourself, though. I expect it to have less overhead than the WindowsFromHost element and although this uses reflection to develop against implementation, not interface, my opinion is it's safe to say the underlaying API in unlikely to change.

Changes required in existing Web Application to access in Smart phones/Tablet computer

I have a ASP.NET Web Application. I want to access this application in Smart phones and tablet computer. So please help me on this. What are the changes required?
The answer to your question could range from nothing to everything. At the end of the day, it all depends on what you existing app looks like in a target mobile device (iPad, Android, etc). If your existing app looks and functions properly, then you don't have to do anything. If not, well, you figure out what's wrong and make it work.
Your question really isn't suited for StackOverflow because it is way to broad and impossible to answer.
Asp.net applications can run in web browsers, including the ones that come with smart phones and tablets. The main concern would be the various screen sizes of the various devices. Because they vary so much, a good design concept for your asp.net application would be to layout content in a way it can adapt to the screen size.
The best technology for this within an asp.net application, especially for an application that was already built, would be to implement style sheets (.css)
Look at these styles as an example:
float, clear, max-width and max-height
Check these and other styles at: http://www.w3schools.com/css/
Now, there is also the concept of having your asp.net application detect which device is being used and then generate UI code (or load .ascx controls) accordingly to provide device specific displays.
Check:
Request.Browser.IsMobileDevice
Request.UserAgent (http://msdn.microsoft.com/en-us/library/system.web.httprequest.useragent.aspx)
However, I would only suggest using this to a minimum, because if you make a change to your UI, you won’t want to keep updating multiple instances of the UI for different devices.
A good compromise would be to build a style sheet for each group of screen sizes (smart phone, tablet, PC, etc.) and then detect which device is in use and include the respective style sheet.
NB: there are many open source projects, which could get you running more quickly with mobile development in mind. Check sourceforge.net and codeplex.com for examples.

Get caret position in non-GDI apps

So I'm writing an accesibility app that needs to know the location of the text entry caret. I tried GUIThreadInfo, but while that works in basic apps like Notepad, it fails in more complex ones like Chrome, iTunes, etc. that handle their own UI.
Is there even a way to get the caret position from apps like this?
Yes, doesn't work. The caret is an implementation detail of user32, associated with a window. Applications like browsers don't use window controls, far too expensive. And they don't have to, there's a separate API to allow such programs to provide an interface to accessibility apps like screen readers. Start reading here. Not easy to use from a C# app, this project can lessen the pain. No endorsement, never actually used it myself.

Create a WPF "control" that is run in an external process

I have a WPF app which contains a number of child controls.
One of these controls hosts a third party library which underneath the covers runs some native code which throws access violations and crashes the application. Unfortunately removing the library is not an option.
What I'd like to do is spin up a new windows process, host the third party library inside that, and somehow communicate with it. Much in the same way that Google Chrome and IE8 handle browser plugins.
The issue is that the third party library needs to draw to the screen, so I have to somehow have the equivalent of an HTML iframe inside my WPF app's main window.
I'm not sure how to get started on this, it's proving difficult to google for thus far. Any advice is greatly appreciated.
This is a tough one, but fortunately for you there is a little work being done in this space lately.
Have you heard of the System.Addin namespace in .NET 3.5? It could probably help in this case. It allows for controls to be loaded in a separate AppDomain, but be displayed in the same UI. I'd imagine you'd have to do a little bit of work to get everything communicating properly (never done this before), but it's possible.
Have a look at this early post from the Add-in team: http://blogs.msdn.com/clraddins/archive/2007/08/06/appdomain-isolated-wpf-add-ins-jesse-kaplan.aspx
Seems like they keep their samples and helper code on codeplex: http://clraddins.codeplex.com/
I'm very interested in this, so if you get this working, let us know how this went for you!
Coming in way late. If you still need this, have you seen this project: https://learn.microsoft.com/en-us/archive/blogs/changov/hosting-wpf-ui-cross-thread-and-cross-process
I'm running multiple plugins in multiple external processes in my LOB app. My framework grew out of the project above.
Probably not the easiest of tasks. Have you considered hosting your 3rd party stuff in a separate App-Domain? That way you will also get a good level of isolation while saving you the hassle of another project. Does it have to be refreshed constantly or could you refresh at predefined points in your application? Maybe some scheme where you basically do a screenshot of the 3rd party output and show it as image in your original app would then be possible...

Categories