We Keep Coding

How to access the elements wrapped inside a ".lazyload-wrapper" using Selenium and C#?

I am building a testing tool for a popular hardware shop website in Australia. I want my test to go to a page that contains, let's say for example, 36 results (The page actually says "Showing 36 of 151 results") and click in the "add to cart" button in one randomly chosen product. To do this I am trying with driver.FindElements to capture all the 36 buttons in the page using a CssSelector.
The problem I am facing is that i can only get the first 12 items because the rest 24 are wrapped in a class=lazyload-wrapper and I just can't get the driver to capture them.
I have tried scrolling the page to the bottom, I have tried inserting Thread.Sleep everywhere, I have tried with WebDriverWait, I have also recreated some deprecated functions from ExpectedConditions to make it work but I am still only getting 12 elements.
I have read pretty much all of the posts about similar issues here and in other forums but I am still stuck with the same 12 items. Perhaps Selenium is not the best choice to do this. If anyone can throw some light with on this challenge it will be greatly appreciated.
This is the class I am using to interact with the page, please forgive me for leaving all that commented out code, it's just to show all the failed attempts:
using System;
using System.Threading;
using OpenQA.Selenium;
using OpenQA.Selenium.Support.UI;
using System.Text.RegularExpressions;
using System.Collections.ObjectModel;
using System.Linq;
namespace Model.Pages
{
public class CarAccessoriesPage : BasePage
{
public CarAccessoriesPage(IWebDriver driver) : base(driver)
{
}
/// <summary>
/// Add a random product to the shopping cart.
/// </summary>
public CarAccessoriesPage AddRandomItemToCart()
{
// smooth scrolling down to force loading did not work
// IJavaScriptExecutor jsex = (IJavaScriptExecutor)driver;
// jsex.ExecuteScript("window.scrollTo({left:0, top:document.body.scrollHeight, behavior:'smooth'});");
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(5));
// Scroll
IJavaScriptExecutor jsex = (IJavaScriptExecutor)driver;
jsex.ExecuteScript("window.scrollTo(0, document.body.scrollHeight);");
// maybe wait for the page to load? Did not work.
wait.Until(d => ((IJavaScriptExecutor)d).ExecuteScript("return document.readyState").Equals("complete"));
// Maybe inserting an ugly pause... no bueno
Thread.Sleep(3000);
// driver.FindElement(By.CssSelector("[data-index='35']")); // this element is not found
Random r = new Random();
int i = r.Next(0, GetPartialResultsCount());
// First approach throws OutOfRangeException because only 12 elements are found
// IWebElement itemElement = new WebDriverWait(driver, TimeSpan.FromSeconds(5))
// .Until(d => d.FindElements(By.CssSelector("[data-locator='atcButton']")))[i];
// Implementing a function from ExpectedConditions deprecated in C#, no good enough
var elements = wait
.Until(PresenceOfAllElementsLocatedBy(By.CssSelector("[data-locator='atcButton']")));
// maybe trying access the article elements inside the wrapper... fail
// var elements = driver.FindElements(By.CssSelector("article.lazyload-wrapper "));
Console.WriteLine(elements.Count); // Always 12!!! I know there are better ways to log
IWebElement ranItem = elements[i]; // Throws an OutOfRangeException if i > 11
Thread.Sleep(1000);
// center the element so the tob and bottom banners don intercept the click
jsex.ExecuteScript("arguments[0].scrollIntoView({block: 'center'});", ranItem);
Thread.Sleep(1000);
ranItem.Click();
Thread.Sleep(1000); // important or last function in test does not work (assertion fails: expected != actual)
return this;
}
/// <summary>
/// Get the number of items in the shopping cart.
/// </summary>
public int GetCartCount()
{
return Int32.Parse(driver.FindElement(By.ClassName("cartItemCount")).Text);
}
private Func<IWebDriver, ReadOnlyCollection<IWebElement>> PresenceOfAllElementsLocatedBy(By locator)
{
return (driver) =>
{
try
{
var elements = driver.FindElements(locator);
return elements.Any() ? elements : null;
}
catch (StaleElementReferenceException)
{
return null;
}
};
}
/// <summary>
/// Generate a random int no larger than the Collection of products
/// in the page. But due to lazy loading, the number generated was causing an out of range exception.
/// </summary>
private int GetPartialResultsCount()
{
string partialResultsCount = driver.FindElement(By.CssSelector(".resultsCount > p")).Text;
return Int32.Parse(Regex.Match(partialResultsCount, #"\d+").Value);
}
}
}
The test itself is actually very straight forward:
using NUnit.Framework;
using Model.Pages;
namespace Tests
{
public class ShoppingCartTests : BaseTests
{
[Test]
public void ValidateCartItemsCount()
{
int cartCount = open<HomePage>()
.GoToCarAccessoriesPage()
.AddRandomItemToCart()
.AddRandomItemToCart()
.GetCartCount();
// Validate cart has 2 items
Assert.AreEqual(2, cartCount);
}
}
}
Finally I just probably need to mention that the test will run and work if I only consider 12 items, but it has become like a personal obsession to get all the items that a user could see in the page. Also, no sure if I should share the actual website I am testing or this goes against the community rules or best practice. Probably is enough to say that is the most popular hardware shop in Australia and the website is fairly complex.
Thanks.

Maybe find the wrapped elements using an xpath selector like
"//*[contains(#class, 'lazyload-wrapper')]//button[contains(#data-locator, 'atcButton')]"
And then add the result to the one you usually get?

So, after many tries I found a way to make the test work and capture the elements wrapped in the lazyloader div. It's not very elegant and I am not exactly sure why do I need to scroll twice to the browser to actually perform the scrolling, and also I am not sure why the elements go missing if I remove one the pauses. By the way, the pauses I am using are simple Thread.Sleep(1000) wrapped in a helper class. Here is the code:
using System;
using OpenQA.Selenium;
using OpenQA.Selenium.Support.UI;
using System.Collections.ObjectModel;
using System.Linq;
using System.Diagnostics;
using Model.Helpers;
namespace Model.Pages
{
public class CarAccessoriesPage : BasePage
{
public CarAccessoriesPage(IWebDriver driver) : base(driver)
{
}
/// <summary>
/// Add a random product to the shopping cart.
/// </summary>
public CarAccessoriesPage AddRandomItemToCart()
{
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(5));
IJavaScriptExecutor jsex = (IJavaScriptExecutor)driver;
Random r = new Random();
// Wait for the page to load.
wait.Until(d => ((IJavaScriptExecutor)d).ExecuteScript("return document.readyState").Equals("complete"));
// Scroll
jsex.ExecuteScript("window.scrollTo(0, document.body.scrollHeight);");
// Slow down
BrowserHelper.Pause();
// Scroll again
jsex.ExecuteScript("window.scrollTo(0, document.body.scrollHeight);");
// Brake
BrowserHelper.Pause();
// Get all the 'add to cart' button elements
var elements = wait
.Until(PresenceOfAllElementsLocatedBy(By.CssSelector("[data-locator='atcButton']")));
// Test output
Trace.WriteLine($"Total products found: {elements.Count}");
// Select a random item
IWebElement ranItem = elements[r.Next(0, elements.Count)];
// Brake
BrowserHelper.Pause();
// Center the element so the top and bottom banners don't intercept the click
jsex.ExecuteScript("arguments[0].scrollIntoView({block: 'center'});", ranItem);
// Brake a little
BrowserHelper.Pause();
// Click 'add to cart'
ranItem.Click();
// Brake again. Important or last function in test does not work (assertion fails: expected != actual)
BrowserHelper.Pause();
return this;
}
/// <summary>
/// Get the number of items in the shopping cart.
/// </summary>
public int GetCartCount()
{
return Int32.Parse(driver.FindElement(By.ClassName("cartItemCount")).Text);
}
private Func<IWebDriver, ReadOnlyCollection<IWebElement>> PresenceOfAllElementsLocatedBy(By locator)
{
return (driver) =>
{
try
{
var elements = driver.FindElements(locator);
return elements.Any() ? elements : null;
}
catch (StaleElementReferenceException)
{
return null;
}
};
}
}
}

How to check for non existence of element using selenium web driver in C#

I goggled this issue but could not find a better answer, so... posting it here.
I click on a button in the browser, which opens up a form/div (which is generated dynamically). The form/div element does not exist until I press button.
Now, I am trying to check whether form/div element is existing or not. I tried with the below code. But it works when an element exists and throws exception (first method - timeout and for second, driver gets stopped) when the element does not exists.
Method 1:
ReadOnlyCollection<IWebElement> elements = Utility.Browser.FindElements(By.TagName("div")); // Utility.Browser is the browser instance.
var expElement = from e in elements
where e.GetAttribute("id").Contains("element id")
select e;
return expElement.Count() > 0;
and
Method 2:
string script = string.Format("return document.getElementById('{0}')", attValue);
IJavaScriptExecutor js = (IJavaScriptExecutor)Utility.Browser; // Utility.Browser is the browser instance.
var ele = js.ExecuteScript(script);
return ele != null;
Any help would be highly appreciated.
Thanks.

Look into WebDriverWait. You can define a wait function that will wait a specific amount of time to satisfy a specific condition. You can essentially say "wait for ten seconds for the element to appear". I'm on my phone and the exact syntax may be incorrect but it would look something like the following.
pulic bool ElementExist(IWebDriver driver)
{
var value = false;
var objWait = new WebDriverWait(driver, Timespan.FromMilliseconds(10000));
objWait.IgnoreExceptionTypes(typeof(WebDriverTimeoutException));
value = objWait.Until(b=>b.FindElements(By.TagName("div")).Count > 0);
return value;
}
You can specify which types of exceptions to ignore, such as ElementNotFound and StaleElement, and the function will continue to wait if those occur. You can also define a function and pass that as a parameter to the .Until function. My skills in lamda expressions and inline function definitions are lacking, otherwise I would give a better example but that is definitely the most customizable approach.

similarly to the other two answers already here, I fashion my test using an extension method along the lines of:
public static bool ElementExists(this IWebDriver driver, By condition, TimeSpan? timeSpan)
{
bool isElementPresent = false;
if (timeSpan == null)
{
// default to 15 seconds if timespan parameter is not passed in
timeSpan = TimeSpan.FromMilliseconds(15000);
}
var driverWait = new WebDriverWait(driver, (TimeSpan)timeSpan);
driverWait.IgnoreExceptionTypes(typeof(WebDriverTimeoutException));
isElementPresent = driverWait.Until(x => x.FindElements(condition).Any());
return isElementPresent;
}
I then use this in code as such:
var isElementPresent = _driver.ElementExists(By.ClassName("register"), TimeSpan.FromSeconds(90.00));
if (isElementPresent)
{
// do required processing...
}
Hope this helps
[edit] - you could of course refactor the extension method to return the required element, with a default of null if you wanted to do everything in a single action.
public static IWebElement FindElementAfterWait(this IWebDriver driver, By condition)
{
bool isElementPresent = false;
IWebElement singleElement = null;
var driverWait = new WebDriverWait(driver, TimeSpan.FromSeconds(90));
driverWait.IgnoreExceptionTypes(typeof(WebDriverTimeoutException));
isElementPresent = driverWait.Until(x => x.FindElement(condition) != null);
if (isElementPresent)
{
singleElement = driver.FindElement(condition);
}
return singleElement;
}
usage:
_driver.FindElementAfterWait(By.ClassName("register"));
also:
public static IWebElement FindElementAfterWait(this IWebDriver driver, Func<IWebDriver, IWebElement> condition)
{
IWebElement singleElement = null;
var driverWait = new WebDriverWait(driver, TimeSpan.FromSeconds(90));
driverWait.IgnoreExceptionTypes(typeof(WebDriverTimeoutException));
singleElement = driverWait.Until(condition);
return singleElement;
}
usage:
_driver.FindElementAfterWait(ExpectedConditions.ElementIsVisible(By.Id("firstName")))
enjoy...

The following function helps me to test the existence of an element on a page in C# Selenium code:
public static bool IsElementPresent(this IWebDriver driver, By by)
{
return driver.FindElements(by).Count > 0;
}
Please let me know if it helps you!

Following method is the one that I always use, and trust me really does what it says.
It will return true if the specified element is displayed else it will return false.
You can use it like : IsElementDisplayedByXpathVariableWait("Xpath_Of_The_Element",5);
5 is the number of times it will check if the element is displayed with a pause of 1 sec after every check.
public static bool IsElementDisplayedByXpathVariableWait(string xpath, int iterations)
{
bool returnVal = false;
int tracker = 0;
while (tracker < iterations)
{
try
{
tracker++;
IWebElement pageObject = _driver.FindElement(By.XPath(xpath));
if (pageObject.Displayed)
{
returnVal = true;
break;
}
}
catch (Exception e)
{
Wait(1000);
continue;
}
}
return returnVal;
}

webdriver - Utility / helper method - wait until class name contains specific style attribute

I have a webapp that will contain a class of "Loading" which, when fully loaded on the page, will contain a width property of 100% else it will contain nothing. I'm trying to perform a check on this style attribute but I keep getting a timeout. here is what I'm doing:
I'm calling code from within a helper / utility class as follows as this is something I will be using frequently in multiple classes:
Utility.WaitForStyle("Loading", Utility.driver);
In my helper / utility class I have the following code:
public static void WaitForStyle(string Class, IWebDriver driver)
{
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(20));
wait.Until<bool>((d) =>
{
try
{
IWebElement element = d.FindElement(By.ClassName(Class));
String elementresults = element.GetAttribute("style");
System.Diagnostics.Debug.WriteLine(elementresults);
return false;
}
catch (NoSuchElementException)
{
return true;
}
});
}
Note, The code above is currently just looking to check that it can get a handle on the class's style attribute but it's not getting to that point. I know the problem lies within the utility method as I can use the following code in individual classes:
IWebElement element = Utility.driver.FindElement(By.ClassName("Loading"));
String elementresults = element.GetAttribute("style");
System.Diagnostics.Debug.WriteLine(elementresults);
This will printout "width: 100%" as expected so I know that this block of code is actually working ok.
Does anyone know if I'm doing something silly in my utility method?

Here's my code to wait for an element attribute to have a specific value. It assumes the element passed to it has been verified to exist:
public bool WaitForAttribute(IWebDriver driver, IWebElement element, string attributeName, string attributeValue, int timeOut = 5)
{
// Build a function with a signature compatible with the WebDriverWait.Until method
Func<IWebDriver, bool> testCondition = (x) => element.GetAttribute(attributeName).Equals(attributeValue);
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(timeOut));
// Wait until either the test condition is true or timeout
try { wait.Until(testCondition); }
catch (WebDriverTimeoutException e) { }
// Return a value to indicate if our wait was successful
return testCondition.Invoke(null);
}

This is working for me, since more than 4 months.
public static WebDriverWait wait = new WebDriverWait(SeleniumInfo.Driver, TimeSpan.FromSeconds(20));
public static void WaitUntilAttributeValueEquals(this IWebElement webElement, String attributeName, String attributeValue)
{
wait.Until<IWebElement>((d) =>
{
if (webElement.GetAttribute(attributeName) == attributeValue)
{
return webElement;
}
return null;
});
}

Specified cast is not valid - HTMLDocument

I'm trying to use this snippet to test of an element has a specific text.
HtmlDocument element = webBrowser2.Document;
if (element.GetElementById("gbqfsa").InnerText == "Google Search")
{
HasSucceeded = 1;
}
return HasSucceeded;
However the first line throws the exception "Specified cast is not valid." What am I doing wrong?

Is it possible that you are using the wrong HtmlDocument class? WebBrowser.Document is of the type:
System.Windows.Forms.HtmlDocument
But I noticed that there is also another possible namespace:
System.Windows.Browser.HtmlDocument
I would check to make sure the namespace you included was System.Windows.Forms.HtmlDocument

I encounter this problem when return HtmlDocument as property from my custom user control. (Which embedded WebBrowser control)
Cause of error because access document from other thread.
/// <summary>
/// Error version '
/// </summary>
public HtmlDocument Document
{
get
{
// Throw error 'Specified cast is not valid'
return this.webBrowserMain.Document;
}
}
But I don't known why error is not 'CrossThread Operation access ...' but next code solved my problem
/// <summary>
/// Fixed version
/// </summary>
delegate HtmlDocument DlgGetDocumentFunc();
public HtmlDocument GetDocument()
{
if(InvokeRequired)
{
return (HtmlDocument)this.webBrowserMain.Invoke(new DlgGetDocumentFunc(GetDocument), new object[] { });
}
else
{
return this.webBrowserMain.Document;
}
}

Retry the request if the first one failed

I am using a xml web service on my web app and sometimes remote server fails to respond in time. I came up with the idea of re-request if first attempt fails. To prevent loop I want to limit concurrent request at 2. I want to get an opinion if what I have done below is ok and would work as I expect it.
public class ScEngine
{
private int _attemptcount = 0;
public int attemptcount
{
get
{
return _attemptcount;
}
set
{
_attemptcount = value;
}
}
public DataSet GetStat(string q, string job)
{
try
{
//snip....
attemptcount += attemptcount;
return ds;
}
catch
{
if (attemptcount>=2)
{
return null;
}
else
{
return GetStat(q, job);
}
}
}
}

public class ScEngine
{
public DataSet GetStat(string q, string job)
{
int attemptCount;
while(attemptCount < 2)
{
try
{
attemptCount++;
var ds = ...//web service call
return ds;
}
catch {}
}
//log the error
return null;
}
}

You forgot to increment the attemptcount. Plus, if there's any error on the second run, it will not be caught (thus, becomes an unhandled exception).

I wouldn't recurse in order to retry. Also, I wouldn't catch and ignore all exceptions. I'd learn which exceptions indicate an error that should be retried, and would catch those. You will be ignoring serious errors, as your code stands.

You don't want to solve it this way. You will just put more load on the servers and cause more timeouts.
You can increase the web service timeout via httpRuntime. Web services typically return a lot of data in one call, so I find myself doing this pretty frequently. Don't forget to increase how long the client is willing to wait on the client side.

Here's a version that doesn't use recursion but achieves the same result. It also includes a delay so you can give the server time to recover if it hiccups.
/// <summary>
/// The maximum amount of attempts to use before giving up on an update, delete or create
/// </summary>
private const int MAX_ATTEMPTS = 2;
/// <summary>
/// Attempts to execute the specified delegate with the specified arguments.
/// </summary>
/// <param name="operation">The operation to attempt.</param>
/// <param name="arguments">The arguments to provide to the operation.</param>
/// <returns>The result of the operation if there are any.</returns>
public static object attemptOperation(Delegate operation, params object[] arguments)
{
//attempt the operation using the default max attempts
return attemptOperation(MAX_ATTEMPTS, operation, arguments);
}
/// <summary>
/// Use for creating a random delay between retry attempts.
/// </summary>
private static Random random = new Random();
/// <summary>
/// Attempts to execute the specified delegate with the specified arguments.
/// </summary>
/// <param name="operation">The operation to attempt.</param>
/// <param name="arguments">The arguments to provide to the operation.</param>
/// <param name="maxAttempts">The number of times to attempt the operation before giving up.</param>
/// <returns>The result of the operation if there are any.</returns>
public static object attemptOperation(int maxAttempts, Delegate operation, params object [] arguments)
{
//set our initial attempt count
int attemptCount = 1;
//set the default result
object result = null;
//we've not succeeded yet
bool success = false;
//keep trying until we get a result
while (success == false)
{
try
{
//attempt the operation and get the result
result = operation.DynamicInvoke(arguments);
//we succeeded if there wasn't an exception
success = true;
}
catch
{
//if we've got to the max attempts and still have an error, give up an rethrow it
if (attemptCount++ == maxAttempts)
{
//propogate the exception
throw;
}
else
{
//create a random delay in milliseconds
int randomDelayMilliseconds = random.Next(1000, 5000);
//sleep for the specified amount of milliseconds
System.Threading.Thread.Sleep(randomDelayMilliseconds);
}
}
}
//return the result
return result;
}