I am trying to open a website and grab some data using Selenium with PhantomJS, however it takes a lot of time to open a website (about 30 second). And every time I open other link I have to wait 30+Seconds. What is wrong with my code ?
static void Main(string[] args)
{
IWebDriver browser = new PhantomJSDriver();
var URL = "http://www.cbssports.com/nba/playerrankings ";
browser.Navigate().GoToUrl(URL);
//Position
var title = browser.FindElements(By.CssSelector(".tableTitle"));
Console.WriteLine(title.First().Text);
Console.Read();
}
Things I have tried to do:
1.Set PhantomJS proxy type to none
2.Disable internet option: automatically detect settings
3.Disable IPv6 protocol
PhantomJS release notes claim, that there are some known issues with network performance on Microsoft Windows. According to release notes, solution is to set proxy type to none, however that doesn't work.
You have to wait 30 seconds because you haven't defined timeouts which are 30 seconds as default. You should use this predefined driver service.
var phantomJSDriverService = PhantomJSDriverService.CreateDefaultService();
IWebDriver browser = new PhantomJSDriver(phantomJSDriverService);
browser.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(0));
The default timeout for Selenium is 30 seconds. You are using browser.FindElements() (the plural version), which will wait the full 30 seconds before continuing!
You can reduce the timeout with browser.manage().timeouts().implicitlyWait(), or you can use explicit timeouts.
Related
Let's say that I want to run so called web acceptance tests on a modern (as of the day of the question) machine, which, let's say, routinely has somewhere between 16 and 128 logical cores. The number could be different in each particular case, but let's stick with this range for now.
By web acceptance test I mean a test, which opens a web page in one of the browsers (Chrome / FireFox / Edge / ... ) using a driver (e.g. chromedriver / geckodriver, etc...), manipulates a web page in whatever way the test wants, and then "collects" some output (e.g. - does the web page has this or that element OR did it navigate to this or that expected page). The actual detail are irrelevant.
Given that such tests naturally spend most of the time waiting (so that to be sure that once they want to manipulate some web page [element] then it has been loaded for sure) it is then seems reasonable to assume that if I have N logical cores on the machine, then I should be able to spawn at least N of such web acceptance tests.
A typical C# code to do that can be summarized as follows:
namespace WebAcceptanceTests
{
public static class Chrome
{
public static async Task Run(
Uri uri,
Func<ChromeDriver, Task> manipulate,
Action<ChromeDriver> validate)
{
var chromeDriverService = ChromeDriverService.CreateDefaultService();
chromeDriverService.HideCommandPromptWindow = true;
var options = new ChromeOptions();
// To make Chrome window invisible.
options.AddArgument("--headless");
using var driver = new ChromeDriver(chromeDriverService, options);
try
{
driver.Manage().Window.Maximize();
driver.Navigate().GoToUrl(uri);
await manipulate(driver);
validate(driver);
}
finally
{
driver.Quit();
}
}
}
}
where manipulate performs some "manipulation" of the page (e.g. attempts to click some buttons / enters some text / etc...) and validate performs some validation (e.g. if manipulate entered user name and password and then clicked login button, then did the site actually transitioned to logged in page). The actual details of what these manipulate and validate do are irrelevant. However, manipulate is a lengthy process because the site needs to load the page and do some "work" here or there. Therefore, we can model it by a method, which just waits and does nothing, e.g.:
public static async Task Manipulate(ChromeDriver driver)
{
// Do some useful stuff here instead of just waiting.
await Task.Delay(60_000);
}
However, if I start spawning such drivers then very quickly (with under 10 drivers created) some of the created drivers start producing weird errors like:
OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://localhost:52382/session timed out after 60 seconds.
The test server machine that I am getting these errors has 16 cores and enough RAM to open hundreds of Chrome tabs without any problems, yet a small number of chrome drivers (less than 10) seems not working in parallel.
Does anyone has any ideas how to make many chrome drivers work in parallel? Ideally I'd want to open (3-4X the number of cores) drivers because they will mostly wait and do nothing.
Thanks.
We achieve this using NUnit parallel run, parakkekuzabke by fixture.
Allocate driver during OneTimeSetup. Do whatever test need in single fixture.
On OneTimeTearDown, dispose driver.
We do this in base class that all web acceptance test fixtures are inherit
[Parallelizable(ParallelScope.Fixtures)]
public abstract class WebDriverTest
{
protected IDriver driver;
[OneTimeSetup]
public void PrepareDriver()
{
// ...
this.driver = new ChromeDriver(chromeDriverService, options);
// ...
}
[OneTimeTearDown]
public void CleanupDriver()
{
this.driver.Dispose();
}
[TearDown]
public void ScreenshotForFailedTest()
{
var testStatus = GetTestStatus();
if (!string.IsNullOrEmpty(testStatus) && testStatus.Equals("Failed"))
{
this.driver.TakeScreenshot(); // extension method with a take screenshot functionality
// log more details if needed
}
}
}
[OneTimeTearDown] is executed even if there is failed tests
As bonus we take screen
Using this snippet we run around 500 smoke tests against chrome in 5-6 minutes on each commit.
I have an application which is used to trigger long running data queues. By long running, I mean around 12-16 hours per queue and either of them cannot be executed in parallel. Each queue has individual steps which need to succeed before the next one runs.
I have already increased the timeouts while initializing ChromeDriver upto 1000 minutes
webDriver == new ChromeDriver(path,options,TimeSpan.FromMinutes(1000));
I am using WebDriverWait for checking after 1000 mins that all steps have been succeeded. In case of a failure, I still have to wait for 1000 minutes before I can tell the dev team about the failure.
Is there a better approach to solve this problem? It is also keeping my browser open for 1000 mins
Regarding your question -- is there a better way to solve this problem? With Selenium, not really. You'd have better luck taking a different approach, such as API, than through UI testing. However, it's still possible, just not ideal.
My best idea for this problem would be to set up some sort of controller that can manage your WebDriver instances and also keep track of the 12-16 hour queue time. Since I don't have any specific information about your project architecture or the queues you are testing, this will be a very generic implementation.
Here's a simple DriverManager class, that controls creating & terminating WebDriver sessions:
public class DriverManager
{
public IWebDriver CreateDriver
{
// code to initialize your WebDriver instance here
}
public void CloseWebDriverSession
{
Driver.Close();
Driver.Quit();
}
}
Next, here's a test case implementation that utilizes DriverManager to close & reopen WebDriver as needed.
public class TestBothQueues
{
// this driver instance will keep track of your session throughout the test case
public IWebDriver driver;
[Test]
public void ShouldRunBothQueues
{
// declare instance of DriverManager class
DriverManager manager = new DriverManager();
// start a webdriver instance
driver = manager.CreateDriver();
// run the first queue
RunFirstQueue();
// terminate the WebDriver so we don't have browser open for 12 hours
manager.CloseWebDriverSession();
// wait 12 hours
Thread.Sleep(TimeSpan.FromHours(12));
// start another WebDriver session to start the second queue
driver = manager.CreateDriver();
// run the second queue
RunSecondQueue();
// terminate when we are finished
manager.CloseWebDriverSession();
}
}
A few notes on this:
You can also convert this code into a while loop if you would like to start a WebDriver instance to check the queue on a time interval. For example, if the queue takes 12-16 hours to finish, you may want to wait 12 hours, then check the queue once per hour until you can verify it is completed. That would look something like this:
// first, wait initial 12 hours
Thread.Sleep(TimeSpan.FromHours(12));
// keep track of whether or not queue is finished
bool isQueueFinished = false;
while (!isQueueFinished);
{
// start webdriver instance to check the queue
IWebDriver driver = manager.CreateDriver();
// check if queue is finished
isQueueFinished = CheckIfQueueFinished(driver);
// if queue is finished, while loop will break
// if queue is not finished, close the WebDriver instance, and start again
if (!isQueueFinished)
{
// close the WebDriver since we won't need it
manager.CloseWebDriverSession();
// wait another hour
Thread.Sleep(TimeSpan.FromHours(1));
}
}
Hope this helps.
I have an automated task that runs for a long time. Sometimes, it takes over a minute for the webpage to fully load so I have tried to set the default timeout in two different ways. Both do not work, I still get that default timeout after 60 seconds error.
public IWebDriver driver;
public DataAuditsUtility()
{
driver = new ChromeDriver();
driver.Manage().Timeouts().PageLoad = TimeSpan.FromSeconds(180);
}
I have also tried like this...
public IWebDriver driver;
public ChromeOptions options;
public DataAuditsUtility()
{
options = new ChromeOptions();
options.AddArgument("--no-sandbox");
driver = new ChromeDriver(options);
}
From what I have seen, both of these should work.. I am starting to think it may be an issue with how I am instantiating the chrome driver. What I do, is I have a class DataAuditsUtility, that has its own instance of WebDriver and I instantiate it and set it up in the constructor. This class has a bunch of methods that make actions on the page.
In the test class, I instantiate this class and just call its methods. The automation runs great however the only issue is I can not figure out how to increase the URL timeout. Any ideas as to why the above does not work? Thanks!
Also worth nothing in this solution that I do not have a config file anywhere. We actually do not use selenium in the solution other than this one test class so I would like to contain all configurations in here rather than a config file elsewhere.
There was an issue reported where webdriver timed-out at a maximum of 60 seconds.
here is a quote from that:
Each driver class has a constructor overload that allows you to set
the timeout for each command. Here is an example for the FirefoxDriver
to set the command timeout to 5 minutes:
IWebDriver driver = new FirefoxDriver(new FirefoxBinary(), null,
TimeSpan.FromMinutes(5));
You can find similar constructor overloads for InternetExplorerDriver,
ChromeDriver, and RemoteWebDriver.
and
When using these constructors it appears that the WebDriverWait is
overridden.
IWebDriver driver = new FirefoxDriver(new FirefoxBinary(), null,
TimeSpan.FromHours(2));
WebDriverWait wait = new WebDriverWait(driver,
TimeSpan.FromMinutes(2)); ...
This caused it to wait for 2 hours rather than two minutes.
So it appears you can set a value on instantiation.
So I'm running automated UI tests with Selenium Web Driver (using PhantomJS as a headless browser) on a web application. When I get to a certain point in my test -- a page where the web app takes over a minute to load the next page (up to 3 minutes some times) -- It will fail.
I get below error message:
Result Message:
Test method UI_Tests.UT_GI_Template.Delivery_UI threw exception:
OpenQA.Selenium.WebDriverException: The HTTP request to the remote
WebDriver server for URL
http://localhost:45539/session/94ef38f0-a528-11e7-a7fd-69a0e29e333f/element/:wdc:1506697956275/value
timed out after 60 seconds. ---> System.Net.WebException: The request
was aborted: The operation has timed out.
I have set the wait time interval to 300 seconds yet it always times out after 60 seconds. Has anyone experienced this error? Here is my code:
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OpenQA.Selenium;
using OpenQA.Selenium.Remote;
using OpenQA.Selenium.Support.UI;
using OpenQA.Selenium.PhantomJS;
namespace UI_Tests
{
[TestClass]
public class UT_GI_Template {
private RemoteWebDriver wd;
[TestMethod]
public void Delivery_UI()
{
IWebDriver wd = new PhantomJSDriver();
try
{
WebDriverWait wait = new WebDriverWait(wd, new TimeSpan(0, 0, 300));
wd.Navigate().GoToUrl("example.com/QA");
wait.Until(d => (d.FindElements(By.CssSelector("#ctl00_ctl00_BodyContent_emailaddress")).Count != 0));
wd.FindElement(By.CssSelector("#ctl00_ctl00_BodyContent_emailaddress")).Click();
wd.FindElement(By.CssSelector("#ctl00_ctl00_BodyContent_emailaddress")).Clear();
wd.FindElement(By.CssSelector("#ctl00_ctl00_BodyContent_emailaddress")).SendKeys("coffutt#icom.com");
wait.Until(d => (d.FindElements(By.CssSelector("#ctl00_ctl00_BodyContent_emailpassword")).Count != 0));
wd.FindElement(By.CssSelector("#ctl00_ctl00_BodyContent_emailpassword")).Click();
wd.FindElement(By.CssSelector("#ctl00_ctl00_BodyContent_emailpassword")).Clear();
....
Test will always time out on step below -- Enter on next button triggers a new page to load (which takes up to 3 minutes) -- I have checked and the rest of my UI tests in the suite run fine with below step commented out.
wait.Until(d => (d.FindElements(By.CssSelector("#ctl00_ctl00_BodyContent_BodyContent_NextButton")).Count != 0));
wd.FindElement(By.CssSelector("#ctl00_ctl00_BodyContent_BodyContent_NextButton")).SendKeys(Keys.Enter);
wait.Until(d => (d.FindElements(By.XPath("//*[#class='grid8 alpha omega']/h1")).Count != 0)); //
string madeIt5 = wd.FindElement(By.XPath("//*[#class='grid8 alpha omega']/h1")).Text;
Is there something I don't know about time intervals or that I am doing wrong? I have included no ImplicitWaits anywhere else in the code and have not included any System.Sleep(s). Why do my tests always timeout after 60 seconds when i have the interval time set to 300 seconds with WebDriverWait wait = new WebDriverWait(wd, new TimeSpan(0, 0, 300));? Any help would be greatly appreciated, thanks!!
After a lot of investigation (and about a million google searches) I have figured out and fixed the problem affecting my automated Selenium tests. The 60 second HTTP timeout error was coming from the Selenium RemoteWebDriver's default settings, this line in the source code >
https://github.com/SeleniumHQ/selenium/blob/9ca04253b7ed337852d50d02c72cb44a13169b71/dotnet/src/webdriver/Remote/RemoteWebDriver.cs#L69
protected static readonly TimeSpan DefaultCommandTimeout = TimeSpan.FromSeconds(60);
The only way to change this value of 60 seconds is to do so when you instantiate your webdriver, in my case I am using PhantomJS and the code looked like this >
var timeOutTime = TimeSpan.FromMinutes(4);
var options = new PhantomJSOptions();
options.AddAdditionalCapability("phantomjs.page.settings.resourceTimeout", timeOutTime);
RemoteWebDriver wd = new PhantomJSDriver("C:\\Users\\MyName\\Source\\Workspaces\\ICompany\\Application\\packages\\PhantomJS.2.0.0\\tools\\phantomjs\\", options, timeOutTime);
My old tests were timing out because of an ASP.net __doPostBack button that would take longer than 60 seconds to fetch data from my database when I selected a lot of arguments/items and therefore would trigger the default command timeout for HTTP requests seen in the WebDriver source code. I know this was the problem because the test would never time out on the postback with under 3 arguments/items selected.
Hope this solution can help someone else, cheers!
update 24-10-2022:
webBrowser = new ChromeDriver(chromeDriverService, chromeOptions, TimeSpan.FromMinutes(5));
I'm developing an application (winforms C# .NET 4.0) where I access a lookup functionality from a 3rd party through a simple HTTP request. I call an url with a parameter, and in return I get a small string with the result of the lookup. Simple enough.
The challenge is however, that I have to do lots of these lookups (a couple of thousands), and I would like to limit the time needed. Therefore I would like to run requests in parallel (say 10-20). I use a ThreadPool to do this, and the short version of my code looks like this:
public void startAsyncLookup(Action<LookupResult> returnLookupResult)
{
this.returnLookupResult = returnLookupResult;
foreach (string number in numbersToLookup)
{
ThreadPool.QueueUserWorkItem(lookupNumber, number);
}
}
public void lookupNumber(Object threadContext)
{
string numberToLookup = (string)threadContext;
string url = #"http://some.url.com/?number=" + numberToLookup;
WebClient webClient = new WebClient();
Stream responseData = webClient.OpenRead(url);
LookupResult lookupResult = parseLookupResult(responseData);
returnLookupResult(lookupResult);
}
I fill up numbersToLookup (a List<String>) from another place, call startAsyncLookup and provide it with a call-back function returnLookupResult to return each result. This works, but I found that I'm not getting the throughput I want.
Initially I thought it might be the 3rd party having a poor system on their end, but I excluded this by trying to run the same code from two different machines at the same time. Each of the two took as long as one did alone, so I could rule out that one.
A colleague then tipped me that this might be a limitation in Windows. I googled a bit, and found amongst others this post saying that by default Windows limits the number of simultaneous request to the same web server to 4 for HTTP 1.0 and to 2 for HTTP 1.1 (for HTTP 1.1 this is actually according to the specification (RFC2068)).
The same post referred to above also provided a way to increase these limits. By adding two registry values to [HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings] (MaxConnectionsPerServer and MaxConnectionsPer1_0Server), I could control this myself.
So, I tried this (sat both to 20), restarted my computer, and tried to run my program again. Sadly though, it didn't seem to help any. I also kept an eye on the Resource Monitor while running my batch lookup, and I noticed that my application (the one with the title blacked out) still only was using two TCP connections.
So, the question is, why isn't this working? Is the post I linked to using the wrong registry values? Is this perhaps not possible to "hack" in Windows any longer (I'm on Windows 7)?
And just in case anyone should wonder, I have also tried with different settings for MaxThreads on ThreadPool (everything from 10 to 100), and this didn't seem to affect my throughput at all, so the problem shouldn't be there either.
It is matter of ServicePoint. Which provides connection management for HTTP connections.
The default maximum number of concurrent connections allowed by a ServicePoint object is 2.
So if you need to increase it you can use ServicePointManager.DefaultConnectionLimit property. Just check the link in MSDN there you can see a sample. And set the value you need.
For quicker reference for someone. To increase the connection limit per host you can do this in your Main() or anytime before you begin making the HTTP requests.
System.Net.ServicePointManager.DefaultConnectionLimit = 1000; //or some other number > 4
Fire and forget this method from your main method. Icognito user is correct, only 2 threads are allowed to play at the same time.
private static void openServicePoint()
{
ServicePointManager.UseNagleAlgorithm = true;
ServicePointManager.Expect100Continue = true;
ServicePointManager.CheckCertificateRevocationList = true;
ServicePointManager.DefaultConnectionLimit = 10000;
Uri MS = new Uri("http://My awesome web site");
ServicePoint servicePoint = ServicePointManager.FindServicePoint(MS);
}
For Internet Explorer 8:
Run Registry Editor and navigate to following key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_MAXCONNECTION SPERSERVER
and
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_MAXCONNECTION SPER1_0SERVER
If FEATURE_MAXCONNECTIONSPERSERVER and FEATURE_MAXCONNECTIONSPER1_0SERVER are missing then create them. Now create DWORD Value called iexplore.exe for both sub keys (listed above) and set their value to 10 or whatever number desired.