We Keep Coding

Is there a way of scraping a list of HTML elements based on key words where the document structure is indeterminate?

I am building a scraper to be used on many sites (too many to scrape manually using a web scraping tool such as Octoparse).
Each site will probably be different in structure. Some sites may have data that I wish to be scraped; some may not. This is to be determined using a list of keywords/keyphrases. Of sites that I wish data to be parsed, these are likely to be presented in a list of some way. However, the HTML elements used to present the list is indeterminate (i.e. could be a ul list, li list, a div list, a table, etc).
If a keyword/keyphrase is found, I wish for not only that element to be parsed, but all others that may be part of the same list/group.
Example 1
<div>
<h1>Random content I am not interested in</h1>
</div>
<div>
<h1>Some more random content I am not interested in</h1>
</div>
<div>
<ul>
<li>Dogs</li>
<li>Cats</li>
<li>Birds</li>
</ul>
</div>
Example 2
<div>
<h1>Random content I am not interested in</h1>
</div>
<div>
<h1>Some more random content I am not interested in</h1>
</div>
<div>
<div>
<div>
<div>
<h1>Bob</h1>
<p>A description of Bob</p>
</div>
<div>
<h1>Ben</h1>
<p>A description of Ben</p>
</div>
<div>
<h1>Bill</h1>
<p>A description of Bill</p>
</div>
</div>
</div>
</div>
From example one, if I had identified the element Dogs, I would like the result to be Dogs, Cats, Birds.
From example two, if I had identified Ben, I would like the result to be 3 div elements, each of which contains the heading and paragraph; the key is that all results are to include HTML, not just text.
Any help/guidance would be much appreciated.

I managed something like this:
static IEnumerable<string> FindSimilarItems(string html, string[] values, int maxDepth)
{
var doc = new HtmlDocument();
doc.LoadHtml(html);
var output = new List<string>();
foreach (var value in values)
{
var rootElement = doc.DocumentNode.SelectSingleNode($"//*[text()='{value}']");
if (rootElement == null) continue;
for (int i = 0; i < maxDepth; i++)
{
var newXpath = RemoveXpathGroupIndex(rootElement.XPath, i);
var newElements = doc.DocumentNode.SelectNodes(newXpath);
if (newElements.Count <= 1) continue;
output.AddRange(newElements.Select(x => x.InnerText));
}
}
return output.GroupBy(x => x).Select(x => x.First()).ToList();
}
static string RemoveXpathGroupIndex(string xpath, int groupElement)
{
var splited = xpath.Split('/');
var pickedElement = splited.Length - 1 - groupElement;
splited[pickedElement] = splited[pickedElement].Substring(0, splited[pickedElement].IndexOf('['));
return string.Join("/", splited);
}
This code:
var similarItems = FindSimilarItems(input1, new string[] { "Dogs" }, 3);
Will return
["Dogs", "Cats", "Birds"]

how to select by text using selenium and c#

I need to select the value (hours) related to an specific date. For example in the html below I need to read the number 24:20 based on the number 6;
this is the html:
<div class="day-ofmonth">
<div class="day-ofmonth">
<span class="day-num">6</span>
<span class="available-time">24:20</span>
</div>
<div class="day-ofmonth">
<span class="day-num">7</span>
<span class="available-time">133:50</span>
</div>
<div class="day-ofmonth">
<div class="day-ofmonth">
if I use:
IWebElement t_value = d.FindElement(By.XPath(".//* [#id='calinfo']/div[9]/span[2]"));
var t_again2 = t_value.GetAttribute("textContent");
i will get 24:20; but i need to get the value 24:20(in this case) based on number 6 (6 refers to day of the month) and not the Xpath (everyday will be a different date). If anyone can point me in the right direction, Thanks

string availableTime = null;
// Find all elements with class = 'day-num'
var dayNums = d.FindElements(By.XPath("//span[#class='day-num']"));
foreach (IWebElement dayNum in dayNums)
{
// check if text is equal to 6
if (dayNum.Text == "6")
{
// get the following sibling with class = 'available-time', then get the text
availableTime = dayNum.FindElement(By.XPath("following-sibling::span[#class='available-time']")).Text;
break;
}
}

A one liner solution:
string availableTime = d.FindElement(By.XPath("//span[#class='day-num' and text()='6']/following-sibling::span[#class='available-time']")).Text;

xpath=//span[text()='6')]/following-sibling::span[1]

Razor - using foreach, insert html every nth item

I'm outputting a large list of items on my page using Razor and MVC 5. Here's what I have currently:
#foreach (var item in Model.Items)
{
<a>#item.Name</a>
}
What I'm trying to do is something that would output this:
<div class="tab-0">
<a>Item 1</a>
<a>Item 2</a>
<a>Item 3</a>
<a>Item 4</a>
</div>
<div class="tab-1">
<a>Item 5</a>
<a>Item 6</a>
<a>Item 7</a>
<a>Item 8</a>
</div>
<div class="tab-2">
<a>Item 9</a>
<a>Item 10</a>
<a>Item 11/a>
<a>Item 12</a>
</div>
I need to group every 4 items within a div tag. How can I do that in Razor?

Not sure if your wanting to increment the Item number (or if #item.Name actually contains the incremented number), but the following code will increment both the class name (a new div every 4th iteration) and the item number.
#{ var t = 0; var i = 1; }
<div class="tab-#t">
#foreach (var item in Model.Items)
{
<a>Item #i</a> // this may be just #item.Name?
if (i % 4 == 0)
{
t++;
#:</div><div class="tab-"#t>
}
i++;
}
</div>

Here's my take on the solution. Bear in mind that I haven't compiled the code myself, so please check it:
foreach(var item in Model.Items.Select((x, i) => new {Index = i, Value = x}))
{
if (item.Index % 4 == 0)
{
<div class="tab-#(item.Index/4)">
}
<a>Item #(item.Index + 1)</a>
if (item.Index % 4 == 0)
{
</div>
}
}
// If the number of items has a remainder of 4, close the div tag
if(Model.Items.Count %4 != 0)
{
#:</div>
}
I have added the Value in Linq Select in case you need the information inside the loop.

foreach doesn't give you the index. That's C#.
I suggest you change it to for.
#for (var index = 0; index < Model.Items.Count(); index++)
{
var item = Model.Items[index];
<a>#item.Name</a>
#* Use `item` or `index` as you wish here *#
}
Maybe you'll use Length instead of Count. If Model.Items is just IEnumerable not array or List<>, you might want to call Model.Items.ToList(), etc. You probably get the idea.

Wrap each 3 items with HTML Wrapper code in ForEach loop C#

I need to build an HTML code structure in C# code behind. The code inserted an HTML item that occupies 3 columns in a 12 columns Row (I'm using Zurb Foundation).
I iterate over over a collection of items in a foreach loop.
I want to add a <div class='row'>[item html code here]</div> code that will wrap the 3 column items.
Item code is something like this:
<div class='column large-4 medium-4- small-12' >some content</div>
What logic should I use (C#) in order to be able to inject the item HTML code inside the row code every three items?
My complications start when I need to separate the opening tags (<div class='row'>) with the closing tag (</div>) and put the items codes (the column divs) inside.
Assume large number of items that needs to iterate over.
The results html should look something like this if for example I have 7 items:
<div class='row'>
<div class='column large-4'>item 1</div>
<div class='column large-4'>item 2</div>
<div class='column large-4'>item 3</div>
</div>
<div class='row'>
<div class='column large-4'>item 4</div>
<div class='column large-4'>item 5</div>
<div class='column large-4'>item 6</div>
</div>
<div class='row'>
<div class='column large-4'>item 7</div>
</div>

How about using Zurb Foundation block-grid?

string BuildItems(int itemCount, int colsPerRow) {
StringBuilder sbItemHTML = new StringBuilder();
bool divBegin = true, divClosed = false;
for ( int i = 0; i < itemCount; i++ ) {
if ( divBegin ) {
sbItemHTML.AppendLine("<div class='row'>");
divBegin = false;
divClosed = false;
}
sbItemHTML.AppendLine("<div class='column large-4'>item " + i + "</div>");
if ( (i % colsPerRow == 0) && i > 0 ) {
sbItemHTML.AppendLine("</div>");
divBegin = true;
divClosed = true;
}
}
if ( !divClosed )
sbItemHTML.AppendLine("</div>");
return sbItemHTML.ToString();
}
how about you to try this code??
Example for use: BuildItems(7, 3);

If you use a counter in for loop; it would help to check by dividing the counter with 3 and check if the returned value is a complete number. If yes, then add the div closing tag (and next div opening tag). example below.
if (Math.Abs(i / 3) == (i / 3)) {
str = str + "<div class='column large-4'>" + value + "</div>" + "</div><div class='row'>";
}
else{
str = str + "<div class='column large-4'>" + value + "</div>";
}

C# Contains Method

My site has a column that displays the news. The DIV-s with these news which contain the word "prize" must be painted in the green color. (If a person has created a record (a news) without specifying the Prize (without word "Prize"), the green is not necessary. But if a person filled out the field model.Prize (so in the text we have a word "Prize") the div must be painted in green color.
In a view for creating news there is a field model.Prize
<div class="editor-field">
#Html.TextAreaFor(model => model.Prize,4,55,null)
#Html.ValidationMessageFor(model => model.Prize)
</div>
The value of model.Prize takes the Controller which create a new news record.
public ActionResult Create(Project project)
{
if (ModelState.IsValid)
{(some code...)
News n = new News();
n.Date = DateTime.UtcNow;
n.Description = project.Shortdescription+"\r\n\Prize:\r\n"+project.Prize;
(some code…)
NewsController.Add(db,n);
db.SaveChanges();
return RedirectToAction("Main");
}
In the another method Block of News Controller I display the news:
public PartialViewResult Block()
{
List<News> news = new List<News>();
Int32 count = 0;
foreach (News n in db.News.ToList().OrderByDescending(n => n.Date))
{
if (count++ < 13) news.Add(n);
}
return PartialView(news);
For each entry in the View Block creates <div class ="newsWrapper"> in which the news record insert.
#foreach (var item in Model){
<div class ="newsWrapper">
<p class="newsDate">#item.Date.AddHours(4).ToString("dd.MM.yyyy HH:mm")</p>
#item.Title
<p>#Html.Raw(Html.Encode(item.Description).Replace("\n", "<br />"))</p>
</div>
}
I tried to solve the problem
I added the new div in the Block View:
#foreach (var item in Model)
{
<div class ="newsWrapper">
<div class="#(ViewBag.IsPrize == true ? "GreenNewsClass" : "")">
<p class="newsDate">#item.Date.AddHours(4).ToString("dd.MM.yyyy HH:mm")</p>
#item.Title
<p>#Html.Raw(Html.Encode(item.Description).Replace("\n", "<br />"))</p>
</div>
</div>
}
The GreenNewsClass will paint this div in green color.
But how can I get ViewBag.IsPrize == true if n.Description contains the word Prize,
and ViewBag.IsPrize == false if it's not?
I tried to change the method Block:
public PartialViewResult Block()
{
List<News> news = new List<News>();
Int32 count = 0;
foreach (News n in db.News.ToList().OrderByDescending(n => n.Date))
{
if (count++ < 13) news.Add(n);
if (n.Description.Contains("Призы"))
{
ViewBag.IsPrize = true;
}
else { ViewBag.IsPrize = false; }
}
return PartialView(news);
but it paints all news in green color, not only those which contain the word Prize.

It sounds like you want to do this:
#foreach (var item in Model)
{
<div class ="newsWrapper">
<div class="#(item.Description.Contains("Призы") ? "GreenNewsClass" : "")">
<p class="newsDate">#item.Date.AddHours(4).ToString("dd.MM.yyyy HH:mm")</p>
#item.Title
<p>#Html.Raw(Html.Encode(item.Description).Replace("\n", "<br />"))</p>
</div>
</div>
}

First try to add a property to your model instead to your ViewBag, it seems like you only have a single value in your ViewBag.
Remove the true condition because it's redundant, move the class definition inside the condition that way the div will be empty when the condition is false
and try the following:
#foreach (var item in Model)
{
<div class ="newsWrapper">
<div #(item.IsPrize? "class=GreenNewsClass" : "")>
<p class="newsDate">#item.Date.AddHours(4).ToString("dd.MM.yyyy HH:mm")</p>
#item.Title
<p>#Html.Raw(Html.Encode(item.Description).Replace("\n", "<br />"))</p>
</div>
</div>
}
I have not verified the code but try it out.