We Keep Coding

HtmlAgilityPack table returns null when selecting nodes [duplicate]

I'm trying to scrape a particular webpage which works as follows.
First the page loads, then it runs some sort of javascript to fetch the data it needs to populate the page. I'm interested in that data.
If I Get the page with HtmlAgilityPack - the script doesn't run so I get what it essentially a mostly-blank page.
Is there a way to force it to run a script, so I can get the data?

You are getting what the server is returning - the same as a web browser. A web browser, of course, then runs the scripts. Html Agility Pack is an HTML parser only - it has no way to interpret the javascript or bind it to its internal representation of the document. If you wanted to run the script you would need a web browser. The perfect answer to your problem would be a complete "headless" web browser. That is something that incorporates an HTML parser, a javascript interpreter, and a model that simulates the browser DOM, all working together. Basically, that's a web browser, except without the rendering part of it. At this time there isn't such a thing that works entirely within the .NET environment.
Your best bet is to use a WebBrowser control and actually load and run the page in Internet Explorer under programmatic control. This won't be fast or pretty, but it will do what you need to do.
Also see my answer to a similar question: Load a DOM and Execute javascript, server side, with .Net which discusses the available technology in .NET to do this. Most of the pieces exist right now but just aren't quite there yet or haven't been integrated in the right way, unfortunately.

You can use Awesomium for this, http://www.awesomium.com/. It works fairly well but has no support for x64 and is not thread safe. I'm using it to scan some web sites 24x7 and it's running fine for at least a couple of days in a row but then it usually crashes.

Xpath works in Chorome but not in Selenium web-driver

I am working on a website where all other locator doesn't work expect using FindElements and take the 3rd a element. So I was curious to try xpath the first time.
I could get the xpath in chrome, but when I use in xpath, it says element not found.
I did a lot of search, still couldn't' find out what was wrong. So I tried in facebook page and use the login field as a try, the xpath is //*[#id="email"], it works perfectly in chrome, but same result in webdrive.
C# code: driver.findElement(By.xpath("//*[#id='email']"));
Please click for facebook picture and its location
Any advise?

I can give a complete solution on Python taking into account the features of React Native (used on Facebook)
But, you have C#. Therefore, it is possible to use a similar function driver.execute_script (execution of Javascript on Selenium)
driver.get("https://www.facebook.com/")
driver.execute_script('
document.getElementById("email").value = "lg#hosct.com";
document.getElementById("u_0_2").click();
')

I did another try with a more clear code:
driver.Url = "";
driver.findElement(By.xpath("//*[#id='email']"));
It works now, the only difference between this and my code before is: I was visiting some other pages before the facebook page. This seems to make difference. Anyway, above code works. If I encounter the issue again, I will post more detail code.

Results page works in IE only

I have a results page only working in IE. It is developed using C# and js in visual studio. So I select a search parameter from the drop down list and search. The results from the DB are displayed in a results page. Those results seem to only be displayed when I use IE. Chrome and fireFox allow for everything else to work except the results:/
Any ideas what could be occurring? Something i need to check with my web.config perhaps?
Thank you in advance=)
C

This is likely an html issue and unrelated to ASP.NET. You should examine the generated HTML. It will be especially easy to see if the data is in the DOM by using Chrome and Firebug.

In Chrome (since it's a place where it's not working) bring up the page and press CTRL+SHIFT+I - this will bring up the DEVELOPER TOOLS. Once up, attempt to use the page and watch the CONSOLE tab of the developer tools. You most likely have scripting errors and the Console will point them out. Many times, you can even click on the console-report to go directly to the offending code (but sometimes you cannot). Regardless, the developer console should help you find the trouble.
If it's a CSS issue, the first tab will be the most helpful to you instead - you can find the generated code in HTML and click on it, then all CSS styles will be on the right and you can review them (and even change them if you need to for testing purposes) to find and eliminate trouble items.

Custom functions unavailable from BrowserInteropHelper.HostScript in an XBAP

I'm attempting to use an XBAP to acquire TWAIN Images but I'm not even getting that far. I can't seem to get the BrowserInteropHelper.HostScript to allow me to talk back to the javascript on the host page.
I am running the XBAP in an iframe.
I tried full trust (even though that should be a requirement).
I'm testing on IE9, .NET Framework 4.0
The BrowserInteropHelper.HostScript is not null, I can use the normal window methods, like .Close().
My code looks like this:
Index.html:
<p id="someP">My cat's breath smells like cat food.</p>
<script type="text/javascript">
function WorkDamnit() {
$('#someP').hide();
}
</script>
<iframe src="#Url.Content("~/XBAPs/WPFBrowserApplication1.xbap")" style="border: none;" width="500" height="500" />
Page1.xaml.cs:
private void Button_Click(object sender, RoutedEventArgs e)
{
if (BrowserInteropHelper.HostScript == null)
throw new ApplicationException("hostscript is null");
else
BrowserInteropHelper.HostScript.WorkDamnit();
}
I get:
System.MissingMethodException: Method '[object Window].WorkDamnit' not found.
New Info:
This scenario works properly from other PCs on the network and I found out why. IE9 has a setting turned on by default called "Display intranet sites in Compatibility View". This causes the page to render in "IE7 mode" and the javascript function is available from the XBAP. If I click Document Mode: (under the F12 developer tools console) and switch to IE9 mode then it no longer works (as above). On my own PC it uses IE9 mode by default (As it should) and it does not work unless I manually switch to IE7 mode.

In IE9, window methods are available to you, so you could try setTimeout
BrowserInteropHelper.HostScript.setTimeout("WorkDamnit()",0);

Its due to IE security settings.
IE by default blocks the scripts if page is opened from local drive.
IE should display the warning message as "'To help protect your security, Internet Explorer has restricted...."
However you can change the settings
Check the checkbox "Allow active content to run in files on My Computer"

IE9 does not expose BrowserInteropHelper.HostScript unless it is in compatibility mode.

This is pretty far after the fact but I wanted to chime in because I've dealt with this problem and it's an ugly one.
As you have observed, the machines it runs on work because they are running in compatibility mode. You can instruct IE9 to render your page using compatibility mode by adding the following tag to your html documents:
<meta http-equiv="X-UA-Compatible" content="IE=8"/>
I'm confident in saying, adding this tag is your only 'solution.' I've spent weeks looking for a better one and this is the one I'm still stuck with.
For whatever reason the ScriptInteropHelper is rife with problems in IE9. It hasn't gotten a lot of attention from Microsoft, likely because it's obscure functionality. That said, particularly because IE7 and IE8 have all sorts of quirks, it's extraordinarily annoying you have to run in compatibility mode. There are a number of MSDN articles, for example this one and enter link description here that discuss the issue further.

If someone is still wondering about this problem, I got found out a solution. Basically from document object you can communicate properly without any hacks in latest browsers and with HTML5 doctype.
I wrote a blog post that contains the codes and examples:
XBAP and Javascript with IE9 or newer - Solution for MissingMethodException problems
Quickly explained this works properly:
Javascript
document.ResponseData = function (responseData) {
alert(responseData);
}
C# XBAP
var hostScript = BrowserInteropHelper.HostScript;
hostScript.document.ResponseData("Hello World");
It can be then used to pass C# object to Javascript and used as "proxy" object. Check that blog post for details how that can be used.

BrowserInteropHelper.HostScript.setTimeout("WorkDamnit()",0);
its working but if i want to send callback functions to js how can i send object with setTimeout ?
when using for ie8
its working good
dynamic host = BrowserInteropHelper.HostScript;
host.sampleJSFunction(new CallbackObject(this));

display google-map in ASP.NET website without using map key

I have to display a google map in my website with certain specifications. However i am not even able to display the map itself.
script src="http://maps.google.com/maps?file=api&v=2&key=ABQIAAAAxTQuI4cSXZdKpxGMrX3fvRQUIgdt2KQbt-zRShjtqzhqRcrL_RSS0tVWaCYz7rT9XNT1ohGSL_WXBA" type="text/javascript">/script
script type="text/javascript"
i have tried using alert within the function but it seems the function is not called at all from the code behind page.i have written the function for map display in managerview.aspx page while i am calling it from the code behind page that is the managerview.aspx.cs page.i believe there is something wrong with the line i have posted.its bcoz it was displaying perfectly fine before. but not working now. :( please help
is there a way to call google-maps without using any kind of key???

you dont need to use java use iframe html code and in google it self it is explain how:
1-search your place in google map
2-click link button on the top right of the page
3-copy the code to your page
its done i do it my self and its work