We Keep Coding

Scraping from a div

I am experimenting with web scraping and I am having trouble scraping a particular value out of some nested div classes. I am using the .NET HtmlAgilityPack class library in a .NET Framework C# Console App. Here is the div code:
<div class="ds-nearby-schools-list">
<div class="ds-school-row">
<div class="ds-school-rating">
<div class="ds-gs-rating-8">
<span class="ds-hero-headline ds-schools-display-rating">8</span>
<span class="ds-rating-denominator ds-legal">/10</span>
</div>
</div>
<div class="ds-nearby-schools-info-section">
<a class="ds-school-name ds-standard-label notranslate" href="https://www.greatschools.org/school?id=00870&state=MD" rel="nofollow noopener noreferrer" target="_blank">Candlewood Elementary School</a>
<ul class="ds-school-info-section">
<li class="ds-school-info">
<span class="ds-school-key ds-body-small">Grades:</span>
<span class="ds-school-value ds-body-small">K-5</span>
</li>
<li class="ds-school-info">
<span class="ds-school-key ds-body-small">Distance:</span>
<span class="ds-school-value ds-body-small">0.8 mi</span>
</li>
</ul>
</div>
</div>
</div>
I want to scrape the "8" from the ds-hero-headline ds-schools-display-rating class. I am having trouble formulating the selector for the SelectNodes method on the DocumentNode object of the HtmlNode.HtmlDocument class.

I guess you might be having a trouble to write XPath to select the node. Try //*[contains(#class, 'ds-hero-headline') and contains(#class, 'ds-schools-display-rating')] with SelectNodes method.
However, this XPath could have a problem if the page your targeting would also have class name like ds-hero-headline-content, which ds-hero-headline can partially match. In that case, see the solution in How can I find an element by CSS class with XPath?

I would use this to extract 0.8 mi
//div[#class='ds-nearby-schools-list']/div[#class='ds-school-row']/div[#class='ds-nearby-schools-info-section']/ul[#class='ds-school-info-section']/li[#class='ds-school-info']/span[#class='ds-school-value ds-body-small' and preceding-sibling::span[#class='ds-school-key ds-body-small' and text()='Distance:']]/text()
Then this regex to group data:
^[0-9\.]+ (.*)$
At the end you can use some kind of conversion to save distance to an object.

Have you tried the following to get the 8. You can search for a specific span element with the class name to get the inner text.
Note: I used text file to load the html from your question.
string htmlFile = File.ReadAllText(#"TempFile.html");
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(htmlFile);
HtmlNode htmlDoc = doc.DocumentNode;
HtmlNode node = htmlDoc.SelectSingleNode("//span[#class='ds-hero-headline ds-schools-display-rating']");
Console.WriteLine(node.InnerText);
// output: 8
Alternate:
Another way is to specify the path that you want the value from, starting from the div element.
HtmlNode node2 = htmlDoc.SelectSingleNode("//div[#class='ds-gs-rating-8']//span[#class='ds-hero-headline ds-schools-display-rating']");
Console.WriteLine(subNode.InnerText);
output
8

c# selenium chrome-webdrive clicking button using class and title

i just want to click button in my page.
The html code lookls like :
<tr ng-repeat="row in rowCollection" ng-class="{ "error-row": row.errorMessage }" ng-style="vm.getColor(row)" class="ng-scope" style="background: rgb(255, 242, 255) none repeat scroll 0% 0%;">
<td class="ng-binding">Wylaczenie nadan RDF</td><td class="ng-binding">WAITING_FOR_NOTIFICATION</td>
<td>
rfsSendingExecutor
</td>
<td class="ng-binding">2017-09-06 11:14:12</td><td class="ng-binding">2017-09-06 11:14:13</td>
<td has-role="REQUEST" class="text-center">
<!-- ngIf: row.inXml || row.outXml -->
<button ng-if="row.inXml || row.outXml" ng-click="vm.showXml(row)" title="Show" class="btn btn-xs ng-scope"><span class="fa fa-code"></span></button>
<!-- end ngIf: row.inXml || row.outXml -->
</td>
<td has-role="ERROR" class="text-center"><button ng-show="row.errorMessage" ng-click="vm.showError(row.errorMessage)" title="Show" class="btn btn-xs ng-hide"><span class="fa fa-search"></span></button></td>
<td class="text-center">
<button ng-show="vm.enableCancel(row)" ng-click="vm.cancelTask(row.workItemId)" title="Cancel" class="btn btn-xs ng-hide">
<span class="fa fa-ban text-warning"></span>
</button>
<button ng-show="vm.enableRepeat(row)" ng-click="vm.repeatTask(row.id)" title="Repeat" class="btn btn-xs ng-hide">
<span class="fa fa-refresh text-success"></span>
</button>
<button ng-show="vm.enableRepeat(row)" ng-click="vm.repeatTaskWithParams(row.id)" title="Repeat with parameters" class="btn btn-xs ng-hide">
<span class="fa fa-refresh text-warning"></span>
</button>
<button ng-show="vm.enableSkip(row)" ng-click="vm.skipTask(row.workItemId)" title="Skip" class="btn btn-xs">
<span class="fa fa-angle-double-right text-success"></span>
</button>
</td>
</tr>
All i want to do is click this button :
<button ng-show="vm.enableSkip(row)" ng-click="vm.skipTask(row.workItemId)" title="Skip" class="btn btn-xs">
<span class="fa fa-angle-double-right text-success"></span>
I've been through the xpath tutorials and checked many other posts nad forums. I'm not sure what I'm missing. I'm simply trying to find the following element by xpath like this :
button_to_click= findElement(By.xpath("//button[#title='Skip']"));
but it doesn't work. QUESTION : Why it don't work only by title?
I try another way and do like that :
button_to_click= findElement(By.xpath("//button[#class='btn btn-xs']"));
And it works well , but when i have 3 or 4 elements in this class it just press wrong button.
How can i press exacly this button can someone help me?
Maybe shouold i try something like this?
button_to_click= findElement(By.xpath("//button[#class='btn btn-xs']//button[#title='Skip']"));
Why it don't work only by title? And how can i do that better? Please be patient for newbies.
EDIT 1
I add more code as you want to know what I'm doing. :
This code works well :
driver = new ChromeDriver();
driver.url ="http://mypage.com"
button_to_click= findElement(By.xpath("//button[#class='btn btn-xs']")).Click();
And this code doesn't work :
driver = new ChromeDriver();
driver.url ="http://mypage.com"
button_to_click= findElement(By.xpath("//button[#title='Skip']")).Click();
EDIT 2
I will give you an example page for testing. You just have to download the html file and open it in your browser.Html page file
What we now want to do?
If you run this html file you will see all page.
And now we want to make a Click on exacly this button on screen :
After when you click on this button you will see click counter below : like this :
Have anyone idea how to click it? I try few ways and can't find solution still. Please help.
EDIT 3
I try also : - but it too doesn't work
drive.FindElement(By.XPath("//tr[class='ng-scope']/td[text()='Wylaczenie nadan RDF'] and button[#title='Skip'']]")).Click();

As per your Question, this line of code works :
button_to_click= findElement(By.xpath("//button[#class='btn btn-xs']")).Click();
This line of code does't works :
button_to_click= findElement(By.xpath("//button[#title='Skip']")).Click();
Explanation:
Looking at the HTML DOM it's clear the WebApplication uses a lot of JavaScript & Ajax Calls. Hence are the attributes e.g. ng-repeat, ng-class etc with dynamic values e.g. { "error-row": row.errorMessage }, vm.showError(row.errorMessage) etc. So it will be tough to use these values/attributes to construct an xpath or CSSselector
Using xpath as //button[#title='Skip'] should have worked provided the xpath uniquely identified the specific element of our interest. But as it is not happening I suspect there may be multiple elements matching this xpath where some of them may be disabled/hidden. So, the xpath using the title attribute as Skip FAILED.
Using xpath as //button[#class='btn btn-xs'] works without failure because here we have considered the class attribute which is extensively used within CSSselector as well as within xpath which maps down to querySelector/querySelectorAll. Hence, this option is more reliable and works perfect.
Update :
Though using xpath as //button[#class='btn btn-xs'] works for you without any failure I am not sure why you want to avoid it. About the xpath you mentioned in your comment, as you have got much granular in your search using the <button> tag it seems unnecessary to reference any parent node e.g. tr[text()='Wylaczenie nadan RDF']. Incase xpath as //button[#class='btn btn-xs'] doesn't identifies the element uniquely you can consider to club up the class and title attribute as follows:
button_to_click= findElement(By.xpath("//button[#class='btn btn-xs' and #title='Skip']")).Click();

I'm curious if you have an element higher in the dom with the same attributes and html elements as the XPath that you're attempting to select. Let's get super specific with our XPath and throw in some and conditions. Give this guy a shot:
//button[#ng-show='vm.enableSkip(row)' and #title='Skip' and #class='btn btn-xs']
If you need to add even more identifiers keep throwing them in there. There is nothing wrong with having very concrete XPaths.
It also looks like your button is in a table row. If there is anything unique in the tr element that contains the button you should definitely throw that in the xpath before the button and you wouldn't have to be concerned about it clicking a button in a previous tr, and in doing so you know you wouldn't need anything more than #title='Skip' for your uniqueness for the button portion of the XPath.
For example...
//tr[#attribute='uniqueTRValue']/button[#title='Skip']

Webdriver Selenium c# - cannot find element

I have a problem, I write test automation c#. In my apps i have in menu position:
<a href="/op/va">
<i class="fa fa-fw fa-child"></i>
<div class="sidebar__label">I read</div> </a>
I want to click on this position "I read" but i don't know how i can search this element.

You could take your element in multiple ways:
//class name
driver.FindElement(By.ClassName("sidebar__label"));
//xpath using class name
driver.FindElement(By.Xpath(".//*[#class='sidebar__label']"));
//xpath using text
driver.FindElement(By.Xpath(".//*[text()='I read']"));
You can read more about it here.

Find element in Selenium using XPATH or CSS Selector

I m trying to find the element "import" using Selenium Webdriver in C#. Have tried the following codes but nothing find it.
driver.FindElement(By.XPath("//*[#class='menu_bg']/ul/li[3]")).Click();
driver.FindElement(By.XPath("//*[#id='import']/a")).Click();
driver.FindElement(By.CssSelector("#import>a")).Click();
driver.FindElement(By.XPath("//*[#class='menu_bg']/ul/li[3]/a")).Click();
driver.FindElement(By.CssSelector("ul[#class='menu_bg']>li[value='3']")).Click();
Please help me out. Design page looks like below:
<body>
<div class="header_bg"></div>
<div class="menu_bg">
<ul class="menu">
<li id="retrieve"></li>
<li id="scan" class="test"></li>
<li id="import">
<a target="main" href="import/import.aspx" onclick="clickme(this,'import')">Import</a>
</li>
<li id="admin"></li>
<li id="help"></li>
<li style="float: right;"></li>
</ul>
</div>
</body>
All the time I got the error as below:
unable to find the element

XPath indexers are 1-based, as opposed to most other languages whereby they are 0-based.
This means you are actually targetting the 2nd li element, which has no anchor element.
So:
//*[#class='menu_bg']/ul/li[3]/a
However, this XPath query is not great and is too strict on position - thus although this newly fixed XPath above should work, I'd advise you to think of something else.

By reviewing this link(Thanks to #Arran), the above issue was fixed. 'switching' to the current IFrame directs Selenium to show any requests to that frame instead.
driver.SwitchTo().Frame()

You can do this by chaining Selenium 'FindElement' like so;
driver.FindElement(By.Id("import")).FindElement(By.TagName("a"));
which will give you the child of the element with ID that has a tag of 'a'.
Another way you could do this is by casting your Driver to an IJavascriptExecutor and executing javascript directly in the browser using a JQuery selector. I find this better for more complex Selenium lookups;
((IJavascriptExecutor)Driver).ExecuteScript("$("a[target='main'][href='import/import.aspx'])").click();

How to find div Element without an id attribute with Selenium Webdriver

I'm using C# and Selenium Webdriver and I'm trying to find a div Element in my html code which looks like this:
<div class="x-grid-cell-inner" style="text-align: left;" unselectable="on">
phys_tag_desc
</div>
I cant find a method to search for the value of the div Element with Selenium Webdriver. I already searched this site and checked the Selenium Webdriver Documentation, but couldn't find anything.

Well if text value is unique, then solution is simple. Try the xpath below:
//div[text()='phys_tag_desc']
If the text is not exact match. Try following:
//div[contains(text(),'phys_tag_desc')]

I have two ways.
Way 2 is more complex but more effective.
Way 1;
You can loop in all divs and look for some equals
Example:
foreach(HtmlElement o in webbrowser.Document.GetElementByTagName("div"))
{
HtmlElement yourElement;
if(o.GetAttribute("class")=="x-grid-cell-inner"&&o.GetAttribute("style")=="text-align: left;")
{
yourElement=o;
break;
}
DoSomethingWith(yourElememt);
}
The other way is follow elements path;
You can find the closer element that has a ID
Example:
<div id="element">
<div>content..</div>
<div>
<div class="x-grid-cell-inner" style="text-align: left;" unselectable="on">
phys_tag_desc
</div>
</div>
</div>
The closest element that have id on this example is
<div id="element">
your element's parent is 2. children of id="element" div
You can get it and follow path like this
yourElement = webbrowser.Document.GetElementById("element").Children[1].Children[0];

In other situation you can use the XPATH Boolean operators.
Try the xpath below:
By.XPath("//div[contains(#class,'*x-grid-cell-inner*') and contains(#unselectable, '*on*') and contains(text(),'*phys_tag_desc*')]")
Bye