We Keep Coding

How can I extract the headers from multiple HTML tables using C# HTMLAgilityPack?

I'm trying to iterate through each table, and extract the headers of each table separately. This is what I've got so far, but whenever I run this it seems to be extracting the header of all tables per loop (headerCount goes up to 61 on each iteration).
namespace DataCollection
{
internal class Program
{
static void Main(string[] args)
{
int headerCount;
HtmlWeb web = new HtmlWeb();
HtmlDocument doc = web.Load("https://en.wikipedia.org/wiki/List_of_actors_who_have_played_the_Doctor");
//Extracting the tables from the HTML
foreach (HtmlNode table in doc.DocumentNode.SelectNodes("//table"))
{
headerCount = 0;
//Extracting the header cells from each table
foreach (HtmlNode headerCol in table.SelectNodes("//th"))
{
headerCount++;
Console.WriteLine(headerCount);
}
};
Console.ReadLine();
}
}
}
What am I doing wrong? Thanks in advance!

I went through the same problem and its intuitive.
using SelectNodes("//th") will search through the entire web document again instead of searching through the selected htmlnode. and its weird.
Try using ".//th"
Placing a dot at the start will tell it to search trough the active node and not the entire htmldocument again.
Hope it helps.

Inserting Custom Element with AngleSharp

I'm trying to update a site that uses an sanitizer based on AngleSharp to process user-generated HTML content. The site users need to be able to embed iframes, and I am trying to use a whitelist to control what domains the frame can load. I'd like to rewrite the 'blocked' iframes to a new custom element "blocked-iframe" that will then be stripped out by the sanitizer, so we can review if other domains need to be added to the whitelist.
I'm trying to use a solution based on this answer: https://stackoverflow.com/a/55276825/794
It looks like so:
string BlockIFrames(string content)
{
var parser = new HtmlParser(new HtmlParserOptions { });
var doc = parser.Parse(content);
foreach (var element in doc.QuerySelectorAll("iframe"))
{
var src = element.GetAttribute("src");
if (string.IsNullOrEmpty(src) || !Settings.Sanitization.IFrameWhitelist.Any(wls => src.StartsWith(wls)))
{
var newElement = doc.CreateElement("blocked-iframe");
foreach (var attr in element.Attributes)
{
newElement.SetAttribute(attr.Name, attr.Value);
}
element.Insert(AdjacentPosition.BeforeBegin, newElement.OuterHtml);
element.Remove();
}
}
return doc.FirstElementChild.OuterHtml;
}
It ostensibly works but I notice that the angle brackets in the new element's tag are being escaped on insertion, so the result just gets written into the page as text. I think I could build a map of replacements and just execute them against the string before sending back but I'm wondering if theres a way to do it using AngleSharp's API. The site is using 0.9.9 currently and I'm not sure how far ahead we'll be able to update considering some of the other dependencies in play.

Digging around in the source I found the ReplaceChild method in INode, which works if called from the parent of element
string BlockIFrames(string content)
{
var parser = new HtmlParser(new HtmlParserOptions { });
var doc = parser.Parse(content);
foreach (var element in doc.QuerySelectorAll("iframe"))
{
var src = element.GetAttribute("src");
if (string.IsNullOrEmpty(src) ||
!Settings.Sanitization.IFrameWhitelist.Any(wls => src.StartsWith(wls)))
{
var newElement = doc.CreateElement("blocked-iframe");
foreach (var attr in element.Attributes)
{
newElement.SetAttribute(attr.Name, attr.Value);
}
element.Parent.ReplaceChild(newElement, element);
}
}
return doc.FirstElementChild.OuterHtml;
}
I will keep testing but this seems decent enough to me, if there is a better way I'd love to hear it.

System.NullReferenceException in LINQ

Based upon a previous written code snippet I'm now trying to store multiple images at once from a certain subreddit into a local directory. My Problem is that I can't get my LINQ statement working properly. I also don't want to download the thumbnail pictures which was why I took a look at the HTML-page and found out that the links I aim to retrieve are hidden in level 5 within the href attribute:
(...)
Level 1: <div class="content">...</div>
Level 2: <div class="spacer">...</div>
Level 3: <div class="siteTable">...</div>
Level 4: <div class=" thing id-t3_6dj7qp odd link ">...</div>
Level 5: <a class="thumbnail may-blank outbound" href="href="http://i.imgur.com/jZ2ZAyk.jpg"">...</a>
That was my best bet in line '???':
.Where(link => Directory.GetParent(link).Equals(#"http://i.imgur.com"))
Sadly enough it throws out an error stating that
Object reference not set to an instance of an object
Well now I know why it's not working but I've still got no clue how to rewrite this line since I'm still fairly new to Lambda Expressions. To be honest, I don't really know why I got a System.NullReferenceException in the first place but not in the next line. What's the difference? Maybe my approach on this problem isn't even good practice at all so please let me know how I could proceed further.
using System;
using System.Collections.Generic;
using System.Linq;
using System.IO;
using System.Net;
using HtmlAgilityPack;
namespace GetAllImages
{
class Program
{
static void Main(string[] args)
{
List<string> imageLinks = new List<string>();
// Specify Directory manually
string dirName = "Jessica Clements";
string rootPath = #"C:\Users\Stefan\Desktop";
string dirPath = Path.Combine(rootPath, dirName);
// Specify the subReddit manually
string subReddit = "r/Jessica_Clements";
string url = #"https://www.reddit.com/" + subReddit;
try
{
DirectoryInfo imageFolder = Directory.CreateDirectory(dirPath);
HtmlDocument document = new HtmlWeb().Load(url);
imageLinks = document.DocumentNode.Descendants("a")
.Select(element => element.GetAttributeValue("href", null))
.Where(???)
.Where(stringLink => !String.IsNullOrEmpty(stringLink))
.ToList();
foreach(string link in imageLinks)
{
using (WebClient _wc = new WebClient())
{
_wc.DownloadFileAsync(new Uri(link), Path.Combine(dirPath, Path.GetFileName(link)));
}
}
Console.WriteLine($"Files successfully saved in '{Path.GetFileName(dirPath)}'.");
}
catch(Exception e)
{
while(e != null)
{
Console.WriteLine(e.Message);
e = e.InnerException;
}
}
if(System.Diagnostics.Debugger.IsAttached)
{
Console.WriteLine("Press any key to continue . . .");
Console.ReadKey(true);
}
}
}
}
Edit: Just in case someone is interested in this solution that's how I made it work in the end using the answers below:
HtmlDocument document = new HtmlWeb().Load(url);
imageLinks = document.DocumentNode.Descendants("a")
.Select(element => element.GetAttributeValue("href", null))
.Where(link => (link?.Contains(#"http://i.imgur.com") == true))
.Distinct()
.ToList();

Given that this line throws the exception:
.Where(link => Directory.GetParent(link).Equals(#"http://i.imgur.com"))
I'd make sure that link is not null and that the result of GetParent(link) is not null either. So you could do:
.Where(link => link != null && (Directory.GetParent(link)?.Equals(#"http://i.imgur.com") ?? false))
Notice the null check and the ?. after GetParent(). This one stops the execution of the term if null is returned from GetParent(). It is called the Null Conditional Operator or "Elvis Operator" because it can be seen as two eyes with twirly hair. The ?? false gives the default value in case the execution was stopped because of a null value.
However, if you plan to parse HTML code you should definitely have a look at the Html Agility Pack (HAP).

if you are trying to get all links pointing to the http://i.imgur.com, you need something like this
imageLinks = document.DocumentNode.Descendants("a")
.Select(element => element.GetAttributeValue("href", null))
.Where(link => link?.Contains(#"http://i.imgur.com") == true)
.ToList();

Parsing a website data c# - HTML agility pack

I tried to parse the results from this website.
http://www.nokia.com/in-en/store-locator/?action=storeSearch&qt=madurai&tags=Nokia_Recycling_Point&country=IN
I need specifically the contents of the div class 'result-wrapper'. That is, all the 'h4', 'category' and 'description' span classes. The following is the code I could reach upto, later on, I do not know to parse that particular div. I need help to get all the contents of that div class.
protected async override void OnNavigatedTo(NavigationEventArgs e)
{
base.OnNavigatedTo(e);
string htmlPage = "";
using (var client = new HttpClient())
{
try
{
htmlPage = await client.GetStringAsync("http://www.nokia.com/in-en/store-locator/?action=storeSearch&qt=madurai&tags=Nokia_Recycling_Point&country=IN");
}
catch (HttpRequestException exc) { }
}
HtmlDocument htmlDocument = new HtmlDocument();
htmlDocument.LoadHtml(htmlPage);

Well, you can try:
var resultWrapperDivs = htmlDocument.DocumentNode.SelectNodes("//div[#class='result-wrapper']");
foreach (var resultWrapperDiv in resultWrapperDivs)
{
// Do stuff with each div.
}
Also, to get a specific content/"html tag" you can take each resultWrapperDiv alone and get also its children nodes (resultWrapperDiv.SelectSingleNode or resultWrapperDiv.SelectNodes)

Select elements added to the DOM by a script

I've been trying to get either an <object> or an <embed> tag using:
HtmlNode videoObjectNode = doc.DocumentNode.SelectSingleNode("//object");
HtmlNode videoEmbedNode = doc.DocumentNode.SelectSingleNode("//embed");
This doesn't seem to work.
Can anyone please tell me how to get these tags and their InnerHtml?
A YouTube embedded video looks like this:
<embed height="385" width="640" type="application/x-shockwave-flash"
src="http://s.ytimg.com/yt/swf/watch-vfl184368.swf" id="movie_player" flashvars="..."
allowscriptaccess="always" allowfullscreen="true" bgcolor="#000000">
I got a feeling the JavaScript might stop the swf player from working, hope not...
Cheers

Update 2010-08-26 (in response to OP's comment):
I think you're thinking about it the wrong way, Alex. Suppose I wrote some C# code that looked like this:
string codeBlock = "if (x == 1) Console.WriteLine(\"Hello, World!\");";
Now, if I wrote a C# parser, should it recognize the contents of the string literal above as C# code and highlight it (or whatever) as such? No, because in the context of a well-formed C# file, that text represents a string to which the codeBlock variable is being assigned.
Similarly, in the HTML on YouTube's pages, the <object> and <embed> elements are not really elements at all in the context of the current HTML document. They are the contents of string values residing within JavaScript code.
In fact, if HtmlAgilityPack did ignore this fact and attempted to recognize all portions of text that could be HTML, it still wouldn't succeed with these elements because, being inside JavaScript, they're heavily escaped with \ characters (notice the precarious Unescape method in the code I posted to get around this issue).
I'm not saying my hacky solution below is the right way to approach this problem; I'm just explaining why obtaining these elements isn't as straightforward as grabbing them with HtmlAgilityPack.
YouTubeScraper
OK, Alex: you asked for it, so here it is. Some truly hacky code to extract your precious <object> and <embed> elements out from that sea of JavaScript.
class YouTubeScraper
{
public HtmlNode FindObjectElement(string url)
{
HtmlNodeCollection scriptNodes = FindScriptNodes(url);
for (int i = 0; i < scriptNodes.Count; ++i)
{
HtmlNode scriptNode = scriptNodes[i];
string javascript = scriptNode.InnerHtml;
int objectNodeLocation = javascript.IndexOf("<object");
if (objectNodeLocation != -1)
{
string htmlStart = javascript.Substring(objectNodeLocation);
int objectNodeEndLocation = htmlStart.IndexOf(">\" :");
if (objectNodeEndLocation != -1)
{
string finalEscapedHtml = htmlStart.Substring(0, objectNodeEndLocation + 1);
string unescaped = Unescape(finalEscapedHtml);
var objectDoc = new HtmlDocument();
objectDoc.LoadHtml(unescaped);
HtmlNode objectNode = objectDoc.GetElementbyId("movie_player");
return objectNode;
}
}
}
return null;
}
public HtmlNode FindEmbedElement(string url)
{
HtmlNodeCollection scriptNodes = FindScriptNodes(url);
for (int i = 0; i < scriptNodes.Count; ++i)
{
HtmlNode scriptNode = scriptNodes[i];
string javascript = scriptNode.InnerHtml;
int approxEmbedNodeLocation = javascript.IndexOf("<\\/object>\" : \"<embed");
if (approxEmbedNodeLocation != -1)
{
string htmlStart = javascript.Substring(approxEmbedNodeLocation + 15);
int embedNodeEndLocation = htmlStart.IndexOf(">\";");
if (embedNodeEndLocation != -1)
{
string finalEscapedHtml = htmlStart.Substring(0, embedNodeEndLocation + 1);
string unescaped = Unescape(finalEscapedHtml);
var embedDoc = new HtmlDocument();
embedDoc.LoadHtml(unescaped);
HtmlNode videoEmbedNode = embedDoc.GetElementbyId("movie_player");
return videoEmbedNode;
}
}
}
return null;
}
protected HtmlNodeCollection FindScriptNodes(string url)
{
var doc = new HtmlDocument();
WebRequest request = WebRequest.Create(url);
using (var response = request.GetResponse())
using (var stream = response.GetResponseStream())
{
doc.Load(stream);
}
HtmlNode root = doc.DocumentNode;
HtmlNodeCollection scriptNodes = root.SelectNodes("//script");
return scriptNodes;
}
static string Unescape(string htmlFromJavascript)
{
// The JavaScript has escaped all of its HTML using backslashes. We need
// to reverse this.
// DISCLAIMER: I am a TOTAL Regex n00b; I make no claims as to the robustness
// of this code. If you could improve it, please, I beg of you to do so. Personally,
// I tested it on a grand total of three inputs. It worked for those, at least.
return Regex.Replace(htmlFromJavascript, #"\\(.)", UnescapeFromBeginning);
}
static string UnescapeFromBeginning(Match match)
{
string text = match.ToString();
if (text.StartsWith("\\"))
{
return text.Substring(1);
}
return text;
}
}
And in case you're interested, here's a little demo I threw together (super fancy, I know):
class Program
{
static void Main(string[] args)
{
var scraper = new YouTubeScraper();
HtmlNode davidAfterDentistEmbedNode = scraper.FindEmbedElement("http://www.youtube.com/watch?v=txqiwrbYGrs");
Console.WriteLine("David After Dentist:");
Console.WriteLine(davidAfterDentistEmbedNode.OuterHtml);
Console.WriteLine();
HtmlNode drunkHistoryObjectNode = scraper.FindObjectElement("http://www.youtube.com/watch?v=jL68NyCSi8o");
Console.WriteLine("Drunk History:");
Console.WriteLine(drunkHistoryObjectNode.OuterHtml);
Console.WriteLine();
HtmlNode jessicaDailyAffirmationEmbedNode = scraper.FindEmbedElement("http://www.youtube.com/watch?v=qR3rK0kZFkg");
Console.WriteLine("Jessica's Daily Affirmation:");
Console.WriteLine(jessicaDailyAffirmationEmbedNode.OuterHtml);
Console.WriteLine();
HtmlNode jazzerciseObjectNode = scraper.FindObjectElement("http://www.youtube.com/watch?v=VGOO8ZhWFR4");
Console.WriteLine("Jazzercise - Move your Boogie Body:");
Console.WriteLine(jazzerciseObjectNode.OuterHtml);
Console.WriteLine();
Console.Write("Finished! Hit Enter to quit.");
Console.ReadLine();
}
}
Original Answer
Why not try using the element's Id instead?
HtmlNode videoEmbedNode = doc.GetElementbyId("movie_player");
Update: Oh man, you're searching for HTML tags that are themselves within JavaScript? That's definitely why this isn't working. (They aren't really tags to be parsed from the perspective of HtmlAgilityPack; all of that JavaScript is really one big string inside a <script> tag.) Maybe there's some way you can parse the <script> tag's inner text itself as HTML and go from there.