We Keep Coding

Unable To Get Lazy Images To Load, Even After Scroll To Bottom Of The Page Using C# Selenium & Browserless.io

I am using the latest selenium in a .Net test using Browserless.io. I'm trying to scroll down the page slowly and allow the images to load.
When I view this page manually in my browser, you can see the images are lazy loaded in.
https://www.reuters.com/business/
I have created the following script to scroll down the page
[Fact]
public void Get_Lazy_Load_Images_Via_Chrome_Selenium()
{
IWebDriver driver;
var options = new ChromeOptions();
// Set launch args similar to puppeteer's for best performance
options.AddArgument("--disable-background-timer-throttling");
options.AddArgument("--disable-backgrounding-occluded-windows");
options.AddArgument("--disable-breakpad");
options.AddArgument("--disable-component-extensions-with-background-pages");
options.AddArgument("--disable-dev-shm-usage");
options.AddArgument("--disable-extensions");
options.AddArgument("--disable-features=TranslateUI,BlinkGenPropertyTrees");
options.AddArgument("--disable-ipc-flooding-protection");
options.AddArgument("--disable-renderer-backgrounding");
options.AddArgument("--enable-features=NetworkService,NetworkServiceInProcess");
options.AddArgument("--force-color-profile=srgb");
options.AddArgument("--hide-scrollbars");
options.AddArgument("--metrics-recording-only");
options.AddArgument("--mute-audio");
options.AddArgument("--headless");
options.AddArgument("--no-sandbox");
// Note we set our token here, with `true` as a third arg
options.AddAdditionalOption("browserless:token", "MYAPIKEY");
driver = new RemoteWebDriver(
new Uri("https://chrome.browserless.io/webdriver"), options.ToCapabilities()
);
driver.Navigate().GoToUrl("https://www.reuters.com/business/");
ScrollToBottom(driver);
Assert.NotNull(driver.Title);
driver.Quit();
}
private static void ScrollToBottom(IWebDriver driver)
{
long scrollHeight = 0;
do
{
var js = (IJavaScriptExecutor) driver;
var newScrollHeight = (long) js.ExecuteScript("window.scrollTo(0, document.body.scrollHeight); return document.body.scrollHeight;");
if(newScrollHeight == scrollHeight)
{
break;
}
scrollHeight = newScrollHeight;
Thread.Sleep(400);
} while (true);
}
However, no matter how much I Thread.Sleep the images just do not load? Any ideas where I am going wrong?

I tried to reproduce your scenario and came up with this simple solution
var collection = wait.Until(d => d.FindElements(By.XPath("//li[#class = 'story-collection__story__LeZ29 story-collection__default__G33_I']")));
foreach (var item in collection)
{
js.ExecuteScript("arguments[0].scrollIntoView({block: \"center\", inline: \"center\"});", item);
Console.WriteLine(item.FindElement(By.XPath(".//img")).GetAttribute("alt"));
}

Where to put implicit wait in Page Object Model?

I have a POM setup with
SearchPage
LogInPage
and then a Test file.
The SearchPage and LogInPage have all the locators, action methods (find, click, send keys) etc.
Currently in my Test file I have:
[TestInitialize]
public void Setup()
{
driver = new ChromeDriver();
driver.Manage().Window.Maximize();
driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(30);
}
One of the tests looks like this:
[TestMethod]
public void LogIn()
{
var searchPage = new SearchPage(driver);
var url = searchPage.GetUrl();
try
{
url.Should().Be(SearchPage.searchURL);
log.Debug("The LogIn test passed!");
}
catch(AssertFailedException ex)
{
log.Debug("The LogIn test failed", ex);
}
}
My question is, will that ImplicitWait in the [TestInitialize] carry across all Page Objects and methods/actions that are being called through the Tests themselves? Or is it only applicable to whatever actions happen on that Tests page itself (i.e. do I need to put the implicit wait in every Page class)?
BTW I realize explicit waits are probably better to use, but I want to get the hang of this first.

The implicit waits apply to all page models using that specific instance of the web driver.
See Implicit Wait Commands in Selenium WebDriver C# for more information.

When we doing page object model mostly we occour nullpointexception because every page should need create object.
if you create utils class for wait you can use it anywhere in project
**public WebElement waitForElement(By locator, int timeout)
{
WebElement element = new WebDriverWait(driver, timeout).until
(ExpectedConditions.presenceOfElementLocated(locator));
return element;
}**

Selenium. Bring-up window on the front

If you run the following code, then at each iteration of the cycle, the browser will bring up on the front and get focus.
public class Program
{
private static void Main()
{
var driver = new ChromeDriver();
driver.Navigate().GoToUrl("https://i.imgur.com/cdA7SBB.jpg");
for (int i = 0; i < 100; i++)
{
var ss = ((ITakesScreenshot)driver).GetScreenshot();
ss.SaveAsFile("D:/imgs/i.jpg");
}
}
}
The question is: why does this happen and can it be turned off? headless mod does not fit.
It seems that this always happens when Selenium needs to save / read the file or start the process.

To take a screenshot, chromedriver activates the window. It's by design and there's no option to avoid it even though it's technically possible.
For the relevant sources have a look at window_commands.cc.
You could however avoid the effect by moving the window off-screen:
driver.Manage().Window.Position = new Point(-32000, -32000);
or by launching the browser off-screen:
var options = new ChromeOptions();
options.AddArgument("--window-position=-32000,-32000");
UPDATE
You can avoid the activation by taking the screenshot directly via the devtool API. Here's a class to override GetScreenshot:
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Remote;
using JObject = System.Collections.Generic.Dictionary<string, object>;
class ChromeDriverEx : ChromeDriver
{
public ChromeDriverEx(ChromeOptions options = null)
: base(options ?? new ChromeOptions()) {
var repo = base.CommandExecutor.CommandInfoRepository;
repo.TryAddCommand("send", new CommandInfo("POST", "/session/{sessionId}/chromium/send_command_and_get_result"));
}
public new Screenshot GetScreenshot() {
object response = Send("Page.captureScreenshot", new JObject {{"format", "png"}, {"fromSurface", true}});
string base64 = (string)((JObject)response)["data"];
return new Screenshot(base64);
}
protected object Send(string cmd, JObject args) {
return this.Execute("send", new JObject {{"cmd", cmd}, {"params", args}}).Value;
}
}
usage:
var driver = new ChromeDriverEx();
driver.Url = "https://stackoverflow.com";
driver.GetScreenshot().SaveAsFile("/tmp/screenshot.png");
driver.Quit();

When you invoke Navigate().GoToUrl("url") method through your Automation script, it is expected that your script will be interacting with some of the elements on the webpage. So for Selenium to interact with those elements, Selenium needs focus. Hence opening up the browser, bring up on the front and getting the focus is the default phenomenon implemented through Navigate().GoToUrl("url").
Now Default Mode or Headless Mode is controlled by the ChromeOption/FirefoxOptions class which is passed as an argument while initializing the WebDriver instance and will call Navigate().GoToUrl("url"). So, Navigate().GoToUrl("url") would have no impact how the WebDriver instance is controlling the Mode of Operation i.e. Default Mode or Headless Mode.
Now when you try to invoke the method from ITakesScreenshot Interface i.e. ITakesScreenshot.GetScreenshot Method which is defined as :
Gets a Screenshot object representing the image of the page on the screen.
In case of WebDriver instance which extends ITakesScreenshot, makes the best effort depending on the browser to return the following in order of preference:
Entire page
Current window
Visible portion of the current frame
The screenshot of the entire display containing the browser
There may be some instances when the browser looses the focus. In that case you can use IJavascriptExecutor to regain the focus as follows :
((IJavascriptExecutor) driver).executeScript("window.focus();");

I was struggling with an issue when generic GetScreenshot() in parallel testing was causing browser to lose focus. Some elements were being removed from DOM and my tests were failing. I've come up with a working solution for Edge and Chrome 100+ with Selenium 4.1:
public Screenshot GetScreenshot()
{
IHasCommandExecutor executor = webDriverInstance as IHasCommandExecutor;
var sessionId = ((WebDriver)webDriverInstance).SessionId;
var command = new HttpCommandInfo(HttpCommandInfo.PostCommand, $"/session/{sessionId}/chromium/send_command_and_get_result");
executor.CommandExecutor.TryAddCommand("Send", command);
var response = Send(executor, "Page.captureScreenshot", new JObject { { "format", "png" }, { "fromSurface", true } });
var base64 = ((Dictionary<string, object>)response.Value)["data"];
return new Screenshot(base64.ToString());
}
private Response Send(IHasCommandExecutor executor, string cmd, JObject args)
{
var json = new JObject { { "cmd", cmd }, { "params", args } };
var command = new Command("Send", json.ToString());
return executor.CommandExecutor.Execute(command);
}

Selenium doesn't wait after submit until website is loaded

I am trying to submit a login form with Selenium from C#. But I can't make it wait after submit to wait the new page to load. Only thing that has worked is Thread.Sleep. What should I do to make it wait?
[TestFixture]
public class SeleniumTests
{
private IWebDriver _driver;
[SetUp]
public void SetUpWebDriver()
{
_driver = new FirefoxDriver();
// These doesn't work
//_driver.Manage().Timeouts().SetPageLoadTimeout(TimeSpan.FromSeconds(10));
//_driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(10));
}
[Test]
public void SubmitTest()
{
_driver.Url = "http://mypage.com";
_driver.FindElement(By.Name("username")).SendKeys("myname");
_driver.FindElement(By.Name("password")).SendKeys("myeasypassword");
_driver.FindElement(By.TagName("form")).Submit();
// It should wait here until new page is loaded but it doesn't
// So far this is only way it has waited and then test passes
//Thread.Sleep(5000);
var body = _driver.FindElement(By.TagName("body"));
StringAssert.StartsWith("Text in new page", body.Text);
}
}

I've found the best way to do this is to wait for an element on the first page to go stale, then wait for the element on the new page. The problem you are likely having is that you are waiting for the body element... which is going to exist on every page there is. If you want to just wait for an element, you should find an element that is unique to the page you are navigating to. If you still want to use the body tag, you can do this...
public void SubmitTest()
{
_driver.Url = "http://mypage.com";
_driver.FindElement(By.Name("username")).SendKeys("myname");
_driver.FindElement(By.Name("password")).SendKeys("myeasypassword");
IWebElement body = _driver.FindElement(By.TagName("body"));
_driver.FindElement(By.TagName("form")).Submit();
body = new WebDriverWait(_driver, TimeSpan.FromSeconds(10)).Until(ExpectedConditions.ElementIsVisible(By.TagName("body")))
StringAssert.StartsWith("Text in new page", body.Text);
}

Answer was practically in JeffC's answer:
I've found the best way to do this is to wait for an element on the first page to go stale, then wait for the element on the new page.
I solved this with this answer: https://stackoverflow.com/a/15142611/5819671
I put following code before reading body element from new page and now it works:
new WebDriverWait(_driver, TimeSpan.FromSeconds(10)).Until(ExpectedConditions.ElementExists((By.Id("idThatExistsInNewPage"))));

selenium testing in c# UnexpectedJavaScriptError

I want to find a textbox id= UserName and give it a value =sa,
There is something wrong with my testing.
The error show UnexpectedJavaScriptError.
What's going on? How can I solve this?
Here is my code.
public void SetupTest()
{
driver.Manage().Timeouts().ImplicitlyWait(new TimeSpan(0, 0, 30));
driver.Navigate().GoToUrl(**the WEBSITE url**);
}
public void Test1()
{
IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
js.ExecuteScript("document.getElementById('UserName').value='sa'");
}
thanks

I use Selenium at my day job, and this sort of code should work fine.
I think the problem may be that the element isn't there when you are trying to use it.
My suggestion is to try something like this:
try {
var element = document.getElementById('UserName');
if(element) {
element.value = 'sa';
}
} catch(e) {}
I actually have a "jquery" helper method in our selenium code that uses jQuery to find elements and then returns the [0] element so that we can use it with Selenium API.
private IWebElement GetFirstElement()
{
return (IWebElement)((IJavaScriptExecutor)_driver).ExecuteScript("return $(\"" + _selector + "\")[0];", null);
}
Additionally, you might just need to wait until the element is on the screen.
An easy way to do this is to use Selenium's FindElement(By...) because Selenium will wait a configurable amount of time for the element to appear.
If you do this, it might make more sense to avoid JavaScript altogether for this case, but what you are trying to do should work in a perfect scenario.

You cannot access DOM using selenium webdriver. Instead of using javascript you can achieve the same using the below code snippet
string text = "sa";
IWebElement element = driver.FindElement(By.Id("UserName"));
element.SendKeys(text);