We Keep Coding

Load multiple pages with Htmlagilitypack - C#

I've scraped a website with HTMLAgilityPack in C#, and I'm trying to open all link inside it and scrape them with same method.
But when I try to call this method bottom, page is downloaded from library as I have AdBlock active. In fact, I can't find any tables and HTML code downloaded says "ADblock detected".
This is strange because I've filter oddsmath website on my Google Chrome and I can download the master page withouth any problem. Anyone has faced with this problem?
This is the function and the "Console.WriteLine" is just for testing and see full HTML code.
public void GetMatchesDetails()
{
List<String> matchDetails = new List<string>();
foreach (Oddsmath om in oddsmathGoodMatches)
{
matchDetails.Add("http://www.oddsmath.com" + om.matchUrl);
}
foreach (String om in matchDetails)
{
HtmlDocument doc = new HtmlWeb().Load(om);
foreach (HtmlNode table in doc.DocumentNode.SelectNodes("html"))
{
Console.WriteLine("Found: " + table.OuterHtml);
foreach (HtmlNode row in table.SelectNodes("tr"))
{
Console.WriteLine("row");
foreach (HtmlNode cell in row.SelectNodes("th|td"))
{
Console.WriteLine("cell: " + cell.InnerText);
}
}
}
}
}
EDIT
Going little deeper, I've noticed that maybe is not a problem of my application or something related to Adblock, but seems connected to website i'm trying to scrape... In fact, if you see a page like this: oddsmath.com/football/international/afc-champions-league-1053/… you can see that content are correctly loaded in browser, but tables are empty inside source code. Why? It's Javascript that prevents loading of page?

First: Use whatever you are most comfortable with HAP vs AngleSharp unless time is really a factor in your application. And in this case it is not.
Second: Use a Web Debugger like Fiddler or Charles to understand what it is that you are actually getting from the when you make a request. Since you are not actually getting any html created with javascript or api calls. You only get the page source. Which is why the tables are empty. They are generated with either javascript.
For instance. I just used a web debugger to see that the site makes an api call to:
http://www.oddsmath.com/api/v1/dropping-odds.json/?sport_type=soccer&provider_id=7&cat_id=0&interval=60&sortBy=1&limit=30&language=en
Then javascript will use this json object to create the rest of page.
And this returns a nice json object that is easier to navigate than with eithr HAP or AngleSharp. I recommend using NewtonSoft JSON.
If you are adamant on using HtmlAgilityPack then you need to combine it with Selenium. Because then you can wait until the page is fully loaded before parsing the HTML.
[Edit]
Further digging:
Api-request to get all the leagues and their ids:
http://www.oddsmath.com/api/v1/menu-leagues.json/?language=en
Api-request for just the asian champions league:
http://www.oddsmath.com/api/v1/events-by-league.json/?language=en&country_code=GB&league_id=1053
Other solution with Selenium with Firefox driver.
Eventhough I highly recommend that you use API and NewtonSoft-JSON to your solution I will provide how it can be done with Selenium.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using HtmlAgilityPack;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium;
using System.Threading;
namespace SeleniumHap {
class Program {
static void Main(string[] args)
{
HtmlDocument doc = new HtmlDocument();
string url = "http://www.oddsmath.com/football/sweden/division-1-1195/2019-04-26/if-sylvia-vs-nykopings-bis-2858046/";
//string url = "http://www.oddsmath.com/";
FirefoxOptions options = new FirefoxOptions();
//options.AddArguments("--headless");
IWebDriver driver = new FirefoxDriver(options);
driver.Navigate().GoToUrl(url);
while (true) {
doc.LoadHtml(driver.PageSource);
HtmlNode n = doc.DocumentNode.SelectSingleNode("//table[#id='table-odds-cat-0']//*[self::th or self::td]");
if (n != null) {
n = n.SelectSingleNode(".//div[#class='live-odds-loading']");
if (n == null) {
break;
}
}
Thread.Sleep(1000);
}
Console.WriteLine("Exited loop. Meaning the page is done loading since we could get a td. A Crude method but it works");
HtmlNodeCollection tables = doc.DocumentNode.SelectNodes("//table");
foreach(HtmlNode table in tables) {
Console.WriteLine(table.GetAttributeValue("id", "No id"));
HtmlNodeCollection tableContent = table.SelectNodes(".//*[self::th or self::td]");
foreach(HtmlNode n in tableContent) {
Console.WriteLine(n.InnerHtml);
}
break;
}
Console.ReadKey();
}
}
}
As you can see I use Firefox as my driver instead of chrome. When using either you might have to edit the options where you edit the variable 'BrowserExecutableLocation' to tell where the browser's executable is.
As you can see I am using a while loop in a crude way to make sure that the browser fully loads page before continuing on reading html.

C# Retrieve Document List From SharePoint 2013 Document Library

I want to make this really simple. I have a fresh, brand new asp.net C# web form with the code behind showing as below.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
public partial class _Default : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
}
}
I have a SharePoint 2013 site with a document library that has a few documents in it with a few columns of metadata.
How do I make it show on the web page, a link to each document and the metadata from the columns for each document from the library. I'm super new to any work with integrating SharePoint and ASP.Net.
Please help.
Andy

Sharepoint has 3 APIs that you could use. Have a look here: https://msdn.microsoft.com/en-us/library/office/jj164060.aspx
You probably want to use the client.svc service via the CSOM Library (Microsoft.SharePoint.Client) just because it is the easiest to get up and running on. Don't use the older asmx api because it is deprecated. There's a third option - REST - but it doesn't provide all the functionality that CSOM does.
Here's some rough code showing the basics. There are a lot of nuances that aren't covered in the code (SharePoint is complicated) so you you'll also want to find some additional information online.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using Microsoft.SharePoint.Client;
public partial class _Default : System.Web.UI.Page
{
protected string SiteUrl = "http://mysite.mydomain.com/site";
protected string LibraryName = "MyList";
protected void Page_Load(object sender, EventArgs e)
{
var context = new ClientContext(SiteUrl);
context.Load(context.Site);
context.ExecuteQuery();
var list = context.Web.Lists.GetByTitle(LibraryName);
if (list == null)
{
throw new ArgumentException(string.Format("List with name '{0}' not found on site '{1}'", LibraryName, SiteUrl));
}
context.Load(list, l => l.RootFolder.ServerRelativeUrl);
context.ExecuteQuery();
// Empty query. You probably want to filter on something so
// do a search on "CAML Query". Also watch out for SharePoint
// List View Threshold which limits # of items that can be retrieved
var camlQuery = #"<View Scope='All'><Query></Query></View>";
var items = list.GetItems(camlQuery);
context.Load(items, l => l.IncludeWithDefaultProperties(i => i.Folder, i => i.File, i => i.DisplayName));
context.ExecuteQuery();
// Url for first item
var url = SiteUrl + "/" + LibraryName + "/" + items[0]["Title"]
}
}

Can't get Request.QueryString or similar methods to work

To my shame, because I can barely C#, I cannot manage to read the parameters from the URL.
I run a C# cgi executable on my IIS 7 as an application. The url that calls the executable is as below:
https://server/cgi/showEmail/showEmail.exe?email=john#gmail.com
The code starts as below:
using System;
using System.Web; // <---- isn't this for Request.QueryString ?
using System.Web.UI;
using System.Collections;
using System.Collections.Specialized;
using System.Data.SqlClient;
class showEmail
{
static void Main(string[] args)
{
Console.WriteLine("\r\n\r\n");
Console.WriteLine("<h1>Test</h1>");
try
{
Now, if I use the code below, the program compiles, but gives this a null exception when executed in the browser:
string email = HttpContext.Current.Request.QueryString["email"];
System.NullReferenceException: Object reference not set to an instance of an object. at showEmail.Main(String[] args)
and if I use this code below, the fun stops already at the compiler, who gives a "current context" exception:
string email = Request.QueryString["email"];
error CS0103: The name 'Request' does not exist in the current context
...
Am I missing something elementary that is required for the executable to see the url parameters?
Edit: I have looked through sof and many other places, but so far have not been able to connect the dots on this issue.

HttpContext.Current.Request is not available in console appliaction.
Use args parameter to receive query string parameter, like I am doing below.
for (int i = 0; i < args.Length; i++)
{
Console.WriteLine("parameter[{0}] is [{1}]", i, args[i]);
}
You might need to use below code to extract parameters from url received in args.
var url=args[0];
var queryString = url.Substring(url.IndexOf('?')).Split('#')[0]
System.Web.HttpUtility.ParseQueryString(queryString)

This is how I got it to work finally, please keep in mind this is not professionally validated code, more of try and try and try...
using System;
using System.Data;
using System.Configuration;
public partial class _Default : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
string email = Request.QueryString["email"] ?? "null";
if (email != "null") {

How to get output of http request in JSON file?

I am making a http get request to a url and I am able to get data from that web page but I am not able to store it in JSON format and also not able to interpret data and get the required data.I am using ASP.NET and C# for it.
This is code for my CS file:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Net;
using System.IO;
using Newtonsoft.Json;
namespace httprequest_web
{
public partial class req : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
}
protected void Button1_Click(object sender, EventArgs e)
{
try
{
string baseu = "http://railenquiry.in/runningstatus/";
string url = string.Concat(baseu, TextBox1.Text);
var request = WebRequest.Create(url);
string text;
var response = (HttpWebResponse)request.GetResponse();
using (var sr = new StreamReader(response.GetResponseStream()))
{
request.ContentType = "application/json; charset=utf-8";
text = sr.ReadToEnd();
}
Label1.Text = text;
}
catch
{
Label1.Text = "No Data Found";
}
}
}
}
and screenshot of output I am receiving is:
I want to take output in a well structured JSON file and only want name of station, time of arrival and time of departure in it. Please tell me how to do it?

Take a look at this project : HtmlAgilityPack But you need to ask the owner of the website before using his data

The URL you're requesting is just a normal web page, it doesn't return JSON.
You'll need to scrape the response, in order to do what you want.
Take a look at this question for examples using HtmlAgilityPack:
Parsing HTML Table in C#

How to write generic methods in coded ui which can used with any type of control irrespective of its property same or changed?

I am using coded ui automation in Visual Studio 2012. So I have a requirement where I have to write some generic/common methods which can be used with any type of UI control.
I an new to coded ui. How to write such methods so that i can reuse it in code.
Basically, for example I want common methods for verifying certain control/Tiles/TAB/Objects is visible or not on UI.

NOTE:- I found below solution from some blog, might be this could be helpful.
To replace the UIMap, we are going to use the power of Generics in the .Net Framework. For ease of use in our testing, I built a static extension so that I could attach it to any UITestControl and use a SearchFor method to find things. As you can see in my code below, it is pretty straight forward. I use dynamics to pass in variables that I can use for both the SearchProperties and FilterProperties to go find items in the application. If you need to find more information on the controls you can use the “Coded UI Test Builder” to get more detailed information.
using Microsoft.VisualStudio.TestTools.UITesting;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace Blog.Example
{
public static class CodedUIExtension
{
public static T SearchFor<T>(this UITestControl _this, dynamic searchProperties, dynamic filterProperties = null) where T : UITestControl, new()
{
T ctrl = new T();
ctrl.Container = _this;
IEnumerable<string> propNames = ((object)searchProperties).GetPropertiesForObject();
foreach (var item in propNames)
{
ctrl.SearchProperties.Add(item, ((object)searchProperties).GetPropertyValue(item).ToString());
}
if (filterProperties != null)
{
propNames = ((object)filterProperties).GetPropertiesForObject();
foreach (var item in propNames)
{
ctrl.FilterProperties.Add(item, ((object)filterProperties).GetPropertyValue(item).ToString());
}
}
return ctrl as T;
}
private static IEnumerable<string> GetPropertiesForObject(this object _this)
{
return (from x in _this.GetType().GetProperties() select x.Name).ToList();
}
private static object GetPropertyValue(this object _this, string propName)
{
var prop = (from x in _this.GetType().GetProperties() where x.Name == propName select x).FirstOrDefault();
return prop.GetValue(_this);
}
}
}
Next, let’s see how to use this in a test. You can just start out with a basic class template using the File –> New in Visual Studio. You will see in my example below that I have added some “using” statements to bring in the testing framework while also adding some attributes for CodedUITest and TestMethod. Once you have these in place you can get started writing your test. Since we are not using the Coded UI Test Generator here, we will go ahead and add the items we need on our own. Not only are we going to remove the brittleness here but we are also going to be efficient.
The first thing I do in my test is to open a Browser and go to a url. In my case I am going to our blog. This will open up the browser and navigate all in one line (nifty). Next we are going to start to find the items we need to interact with on the screen. As you can see the first item I am going to go get n my test is the header section. This is a custom html control with the tagname “HEADER.” Once I find this control, I am going to use it in the next section so that I have a container to search within. To be honest, I really don’t have to do this, but I did want to show you that you can search for items using a container instead of having to search the entire DOM every time. Next I will get the search button control and use the “Mouse.Click” operation to click and open the search text box. Once the Coded UI Test finds the search box, it automates the text I want and use and then uses keyboard controls to send the ENTER key for the search.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Input;
using Microsoft.VisualStudio.TestTools.UITest.Extension;
using Microsoft.VisualStudio.TestTools.UITesting;
using Microsoft.VisualStudio.TestTools.UITesting.WinControls;
using Microsoft.VisualStudio.TestTools.UITesting.HtmlControls;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Keyboard = Microsoft.VisualStudio.TestTools.UITesting.Keyboard;
using Mouse = Microsoft.VisualStudio.TestTools.UITesting.Mouse;
using MouseButtons = System.Windows.Forms.MouseButtons;
namespace Blog.Example
{
[CodedUITest]
public class SimpleUITestWithoutCodedUI
{
[TestMethod]
public void SimpleWebTest()
{
//open the default browser and navigate to a web page - blog.falafel.com
BrowserWindow browser = BrowserWindow.Launch(new Uri("http://blog.falafel.com"));
//we can use controls as a way to narrow down the search for other controls
var headerSection = browser.SearchFor<HtmlCustom>(new { TagName = "HEADER" });
//let's search for something
var searchIcon = headerSection.SearchFor<HtmlHyperlink>(new { href = "http://blog.falafel.com/#searchbox" });
Mouse.Click(searchIcon);
var searchInput = browser.SearchFor<HtmlEdit>(new { name = "s" }, new { type = "INPUT" });
searchInput.Text = "treats and tricks";
Keyboard.SendKeys("{ENTER}");
//find our searched for item
var postToLookFor = browser.SearchFor<HtmlCustom>(new { TagName = "ARTICLE" }, new { InnerText = "31 Days of Visual Studio 2015 Tricks and Treats Blog Post" });
//validate our search
Assert.IsTrue(postToLookFor.Exists);
}
}
}
The next thing I do on the search result screen is find the item that I am interested in evaluating. I made that sound really easy, but having dynamic controls show up and find them is one of the hardest things to do in any Testing Framework. With the power of C# and the flexibility of the Coded UI Framework, I am able to dynamically find my items using search and filter properties so that I can evaluate them. The rest of the test, including the assertion, are basic unit testing techniques.

Like that?
public bool IsVisible(Control control)
{
return control.Visible;
}

I implemented a generic search function as such:
public static ControlType IdentifyControlByIdentifierEqualsValue<ControlType>(UITestControl parent, ControlIdentifier identifierTypeAndValue) where ControlType : UITestControl
{
var control = (ControlType)Activator.CreateInstance(typeof(ControlType), new UITestControl[] { parent });
control.SearchProperties.Add(identifierTypeAndValue.Type, identifierTypeAndValue.Value);
var wasFound = control.TryFind();
if (!wasFound)
throw new UITestControlNotFoundException("The control of type " + control.GetType().ToString() + " with the identifier type of " + identifierTypeAndValue.Type + " and the identifying attribute of " + identifierTypeAndValue.Value + " was not able to be found");
return control;
}
With the supporting class
public class ControlIdentifier
{
public ControlIdentifier(string type, string id)
{
Type = type;
Value = id;
}
public string Type { get; set; }
public string Value { get; set; }
}
Called as such:
IdentifyControlByIdentifierEqualsValue<WpfTabList>(workWindow,
new ControlIdentifer(WpfButton.PropertyNames.Name, "OK");