Having an html string like below,
...
<ul class="not-this-class" ...>
... <!-- some "<li> <a href.. > </a> </li>" here -->
</ul>
<ul class="yes-this-class" ...>
<li>
some text 1
</li>
<li>
some text 2
</li>
<li>
some text 3
</li>
...
</ul>
<ul class="not-this-class-either" ...>
... <!-- some "<li> <a href.. > </a> </li>" here -->
</ul>
I can extract the class named yes-this-class using the following code:
HtmlAgilityPack.HtmlDocument document = new HtmlAgilityPack.HtmlDocument();
document.LoadHtml(responseString.ToString());
HtmlNode htmlNode =
document.DocumentNode.SelectSingleNode("//*[#class='yes-this-class']");
then, I do some string manipulation (regular expression) to extract the text below:
some text 1
some text 2
some text 3
How can I extract the result just above, using only HtmlAgilityPack and without using regular expression? I tried something like below but it didn't work.
HtmlNodeCollection htmlNodes =
document.DocumentNode.SelectNodes("//*[#class='yes-this-class']/[#a='href']");
Use the following XPath query:
//ul[#class='yes-this-class']/li//text()
Then run Trim() on each result to remove any leading and trailing whitespace around the string.
Related
<div class="accordion_browse">
<a target="_blank" href="https://www.silverpages.sg/tools/e-care-locator"
aria-expanded="false" aria-controls="collapseExample">
<span class="browse_img">
<img src="/sites/assets/assets/directory/icons/eldercare.png"/>
</span>
Eldercare Services
<i class="indicator fa fa-chevron-right pull-right" aria-hidden="true"/>
</a>
</div>
Note: the text required is 'Eldercare Services'
Code:
string Eldercare = driver.FindElement(By.XPath("//a[contains(.,'Eldercare Services')")).Text;
Console.WriteLine(Eldercare + " found");
Assert.IsTrue(Eldercare.Contains("Elder"), Eldercare + " not found");
A text node is retrieved using the text node text()
XPath: "/body/div/text()"
That will retrieve all children text nodes of /body/div.
"/body/div/text()[1]" will retrieve the first text node inside /body/div
Additional Reading on XPath:
https://www.w3schools.com/xml/xpath_examples.asp
The given xpath is looking good, but By.XPath("//a[contains(.,'Eldercare Services')") example code missing a closing square bracket: By.XPath("//a[contains(.,'Eldercare Services')]")
With this fix, the xpath will give the a element
<a target="_blank" href="https://www.silverpages.sg/tools/e:care:locator" aria:expanded="false" aria:controls="collapseExample">
<span class="browse_img">
<img src="/sites/assets/assets/directory/icons/eldercare.png"/>
</span>
Eldercare Services longtext
<i class="indicator fa fa:chevron:right pull:right" aria:hidden="true"/>
</a>
The missing bracket should fix the code.
If Selenium would have work with text elements it could be
//a[contains(.,'Eldercare Services')]/text()
results would be a string array with 3 strings. the middle one has the expected string, because before the span and after i elements there will probably be two empty strings.
Text =
Text =
Eldercare Services longtext
Text =
If use a second filter for the text, it would result the expected string only
//a[contains(.,'Eldercare Services')]/text()[contains(.,'Elder')]
How can I use this xPath with Html Agility Pack?
xPath:
//div[#class='test']/(text())[last()]
I've tried this code:
foreach (HtmlNode node in doc.DocumentNode.SelectNodes("//div[#class='test']/(text())[last()]"))
{
test = node.InnerText();
}
Html:
<div class="test">
<ul>
<li><b>Test1</b>Test1 Text</li>
<li><b>Test2</b>Test2 Text</li>
</ul>
</div>
I need to extract "Test2 Text" without specific the ul tag in the xPath.
You can try using this XPath :
(//div[#class='test']//text()[normalize-space()])[last()]
//div[#class='test']//text()[normalize-space()] finds all non-empty text nodes within the div. And then, [last()] return only the last node from all found text nodes.
Working demo example (see it online here) :
var html = #"<div class='test'>
<ul>
<li><b>Test1</b>Test1 Text</li>
<li><b>Test2</b>Test2 Text</li>
</ul>
";
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(html);
HtmlNode node = doc.DocumentNode.SelectSingleNode("(//div[#class='test']//text()[normalize-space()])[last()]");
Console.WriteLine(node.InnerText);
output :
Test2 Text
I have following xpath fetched using firefox xpath plugin
id('some_id')/x:ul/x:li[4]/x:span
using html agility pack I'm able to fetch id('some_id')/x:ul/x:li[4]
htmlDoc.DocumentNode.SelectNodes(#"//div[#id='some_id']/ul/li[4]").FirstOrDefault();
but I dont know how to get this span value.
update
<div id="some_id">
<ul>
<li><li>
<li><li>
<li><li>
<li>
Some text
<span>text I want to grab</span>
</li>
</ul>
</div>
You don't need parse HTML with LINQ2XML, HTMLAgilityPack it's for it and it's more easy to obtain the node in the following way :
var html = #" <div id=""some_id"">
<ul>
<li></li>
<li></li>
<li></li>
<li>
Some text
<span>text I want to grab</span>
</li>
</ul>
</div>";
var doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(html);
var value = doc.DocumentNode.SelectSingleNode("div[#id='some_id']/ul/li/span").InnerText;
Console.WriteLine(value);
An alternative approach (without html-agility-pack) would be to use LINQ2XML. You can use the XDocument.Descendants method to take the span element and take it's value:
var xml = #" <div id=""some_id"">
<ul>
<li></li>
<li></li>
<li></li>
<li>
Some text
<span>text I want to grab</span>
</li>
</ul>
</div>";
var doc = XDocument.Parse(xml);
Console.WriteLine(doc.Root.Descendants("span").FirstOrDefault().Value);
The code can be extended to check if the div element has the matching id, using the XElement.Attribute property:
var doc = XDocument.Parse(xml);
Console.WriteLine(doc.Elements("div").Where (e => e.Attribute("id").Value == "some_id").Descendants("span").FirstOrDefault().Value);
One drawback of this solution is that the XML structure (HTML, XHTML) needs to be properly closed or else the parsing will fail.
My HTML string is like this , stored in a variable named sourceCode
<ul class="yom-list col first" style="width:33.333333333333%">
<li class="first">
<a href="/india/andaman-and-nicobar-islands/">
<span>Andaman and Nicobar Islands</span>
</a>
</li>
<li>
<a href="/india/jammu-and-kashmir/">
<span>Jammu and Kashmir</span>
</a>
</li>
<li class="last">
<a href="/india/andhra-pradesh/">
<span>Andhra Pradesh</span>
</a>
</li>
<li>
<a href="/india/jammu-and-kashmir/">
<span>Jammu and Kashmir</span>
</a>
</li>
</ul>
I want to convert it in to a generic List
So that I can access the data inside it in my code like href, name etc..
I have tried something like this
foreach (Match match in Regex.Matches(sourceCode, #"<li><a href=""(?<url>[^""])</a></li>"))
items.Add(new Item()
{
name = match.Groups["span"].Value, // i don't know how to get value inside that span
url = match.Groups["url"].Value,
});
But it does not work, Probably the regex is wrong. Can any one tell me what I am doing wrong?
Note: I can't use HTMLAgilityPack in this project
Try the below regex to get the values between <a href> tag and <span> tag only if it is present inside <li> tag.
/<li>\s*<a href=\"(?<url>[^"]*)\">\s*<span>(?<span>[^<]*)<\/span>/m
DEMO
Your c# code would be,
Regex rgx = new Regex(#"<li>\s*<a href=""(?<url>[^""]*)"">\s*<span>(?<span>[^<]*)</span>");
foreach (Match m in rgx.Matches(input))
{
Console.WriteLine(m.Groups["url"].Value);
Console.WriteLine(m.Groups["span"].Value);
}
IDEONE
I'm developing a program in C# and I require some help. I'm trying to create an array or a list of items, that display on a certain website. What I'm trying to do is read the anchor text and it's href. So for example, this is the HTML:
<div class="menu-1">
<div class="items">
<div class="minor">
<ul>
<li class="menu-item">
<a class="menu-link" title="Item-1" id="menu-item-1"
href="/?item=1">Item 1</a>
</li>
<li class="menu-item">
<a class="menu-link" title="Item-1" id="menu-item-2"
href="/?item=2">Item 2</a>
</li>
<li class="menu-item">
<a class="menu-link" title="Item-1" id="menu-item-3"
href="/?item=3">Item 3</a>
</li>
<li class="menu-item">
<a class="menu-link" title="Item-1" id="menu-item-4"
href="/?item=4">Item 4</a>
</li>
<li class="menu-item">
<a class="menu-link" title="Item-1" id="menu-item-5"
href="/?item=5">Item 5</a>
</li>
</ul>
</div>
</div>
</div>
So from that HTML I would like to read this:
string[,] array = {{"Item 1", "/?item=1"}, {"Item 2", "/?item=2"},
{"Item 3", "/?item=3"}, {"Item 4", "/?item=4"}, {"Item 5", "/?item=5"}};
The HTML is an example I had written, the actual site does not look like that.
As others said HtmlAgilityPack is the best for html parsing, also be sure to download HAP Explorer from HtmlAgilityPack site, use it to test your selects, anyway this SelectNode command will get all anchors that have ID and it start with menu-item :
HtmlDocument doc = new HtmlDocument();
doc.Load(htmlFile);
var myNodes = doc.DocumentNode.SelectNodes("//a[starts-with(#id,'menu-item-')]");
foreach (HtmlNode node in myNodes)
{
Console.WriteLine(node.Id);
}
If the HTML is valid XML you can load it using the XmlDocument class and then access the pieces you want using XPaths, or you can use and XmlReader as Adriano suggests (a bit more work).
If the HTML is not valid XML I'd suggest to use some existing HTML parsers - see for example this - that worked OK for us.
You can also use the HtmlAgility pack
I think this case is simple enough to use a regular expression, like <a.*title="([^"]*)".*href="([^"]*)":
string strRegex = #"<a.*title=""([^""]*)"".*href=""([^""]*)""";
RegexOptions myRegexOptions = RegexOptions.None;
Regex myRegex = new Regex(strRegex, myRegexOptions);
string strTargetString = ...;
foreach (Match myMatch in myRegex.Matches(strTargetString))
{
if (myMatch.Success)
{
// Use the groups matched
}
}