When I write a test in Visual Studio, I check that it works by saving, building and then running the test it in Nunit (right click on the test then run).
The test works yay...
so I Move on...
Now I have written another test and it works as I have saved and tested it like above. But, they dont work when they are run together.
Here are my two tests that work when run as individuals but fail when run together:
using System;
using NUnit.Framework;
using OpenQA.Selenium.Support.UI;
using OpenQA.Selenium;
namespace Fixtures.Users.Page1
{
[TestFixture]
public class AdminNavigateToPage1 : SeleniumTestBase
{
[Test]
public void AdminNavigateToPage1()
{
NavigateTo<LogonPage>().LogonAsCustomerAdministrator();
NavigateTo<Page1>();
var headerelement = Driver.FindElement(By.ClassName("header"));
Assert.That(headerelement.Text, Is.EqualTo("Page Title"));
Assert.That(Driver.Url, Is.EqualTo("http://localhost/Page Title"));
}
[Test]
public void AdminNavigateToPage1ViaMenu()
{
NavigateTo<LogonPage>().LogonAsCustomerAdministrator();
Driver.FindElement(By.Id("menuitem1")).Click();
Driver.FindElement(By.Id("submenuitem4")).Click();
var headerelement = Driver.FindElement(By.ClassName("header"));
Assert.That(headerelement.Text, Is.EqualTo("Page Title"));
Assert.That(Driver.Url, Is.EqualTo("http://localhost/Page Title"));
}
}
}
When the second test fails because they have been run together
Nunit presents this:
Sse.Bec.Web.Tests.Fixtures.ManageSitesAndUsers.ChangeOfPremises.AdminNavigateToChangeOfPremises.AdminNavigateToPageChangeOfPremisesViaMenu:
OpenQA.Selenium.NoSuchElementException : The element could not be found
And this line is highlighted:
var headerelement = Driver.FindElement(By.ClassName("header"));
Does anyone know why my code fails when run together, but passes when run alone?
Any answer would be greatly appreciated!
Such a situation normally occurs when the unit tests are using shared resources/data in some way.
It can also happen if your system under test has static fields/properties which are being leveraged to compute the output on which you are asserting.
It can happen if the system under test is being shared (static) dependencies.
Two things you can try
put the break point between the following two lines. And see which page are you in when the second line is hit
Introduce a slight delay between these two lines via Thread.Sleep
Driver.FindElement(By.Id("submenuitem4")).Click();
var headerelement = Driver.FindElement(By.ClassName("header"));
If none of the answers above worked for you, i solved this issue by adding Thread.Sleep(1) before the assertion in the failing test...
Looks like tests synchronization is missed somewhere... Please note that my tests were not order dependant, that i haven't any static member nor external dependency.
look into the TestFixtureSetup, Setup, TestFixtureTearDown and TearDown.
These attributes allow you to setup the testenvironment once, instead of once per test.
Without knowing how Selenium works, my bet is on Driver which seems to be a static class so the 2 tests are sharing state. One example of shared state is Driver.Url. Because the tests are run in parallel, there is a race condition to set the state of this object.
That said, I do not have a solution for you :)
Are you sure that after running one of the tests the method
NavigateTo<LogonPage>().LogonAsCustomerAdministrator();
is taking you back to where you should be? It'd seem that the failure is due to improper navigation handler (supposing that the header element is present and found in both tests).
I think you need to ensure, that you can log on for the second test, this might fail, because you are logged on already?
-> putting the logon in a set up method or (because it seems you are using the same user for both tests) even up to the fixture setup
-> the logoff (if needed) might be put in the tear down method
[SetUp]
public void LaunchTest()
{
NavigateTo<LogonPage>().LogonAsCustomerAdministrator();
}
[TearDown]
public void StopTest()
{
// logoff
}
[Test]
public void Test1()
{...}
[Test]
public void Test2()
{...}
If there are delays in the DOM instead of a thread.sleep I recommend to use webdriver.wait in combination with conditions. The sleep might work in 80% and in others not. The wait polls until a timeout is reached which is more reliable and also readable. Here an example how I usually approach this:
var webDriverWait = new WebDriverWait(webDriver, ..);
webDriverWait.Until(d => d.FindElement(By.CssSelector(".."))
.Displayed))
I realize this is an extremely old question but I just ran into it today and none of the answers addressed my particular case.
Using Selenium with NUnit for front end automation tests.
For my case I was using in my startup [OneTimeSetUp] and [OneTimeTearDown] trying to be more efficient.
This however has the problem of using shared resources, in my case the driver itself and the helper I use to validate/get elements.
Maybe a strange edge case - but took me a few hours to figure it out.
Related
I have a simple test method so far just to call the static method Main in Program.cs on a netcore mvc app.
My assert is never hit and the test just keeps running in the test explorer, even after 5 or 10min. Is it because the Program needs termination or is my implementation wrong for the test?
Advice appreciated
Thanks
Rob
[TestClass]
public class ProgramTests
{
[TestMethod]
public void ExecMain()
{
var args = new string[] { };
Program.Main(args);
// never hits test condition and test keeps running
Assert.IsTrue(true);
}
}
Your implementation is not wrong since you are following the AAA practice (Arrange, Act, Assert).
It seems, however, that you have not figured out what is to be tested. Are you expecting Program.Main to run forever? or should it terminate in some point? (something or someone should probably cause termination.)
If you can find out how to cause termination, you should do it in your test and then make the assertion (which obviously reflects your expectations from termination.)
I've been using Selenium and NUnit to do some automated testing, and up until now everything has been fine. The change I made recently was adding more than one test to a test class.
I'm pretty certain the issue lies with the code in my "Teardown" function in the test class. When I comment out
BrowserFactory.CloseAllDrivers();
Things run just fine.
This is the code for my "FrontEndAddItemToCartTest":
class FrontEndAddItemToCartTest : PageTest
{
[SetUp]
public void Initialize()
{
SetBrowser(BrowserFactory.BrowserType.Chrome); // Not headless
SetServer("testUrlNotGivenForSecurityPurposes");
StartTest(TestType.FrontEnd);
SetSize(MobileSize.XXLarge);
}
[Test]
public void StandardQuantityTest()
{
OrderItem standardQuantity = new OrderItem(new Product("500", ".25"), 500);
FrontEndActions.AddItemToCart(standardQuantity);
}
[Test]
public void CustomQuantityTest()
{
OrderItem customQuantity = new OrderItem(new Product("482", ".25"), 225);
FrontEndActions.AddItemToCart(customQuantity);
}
[TearDown]
public void EndTest()
{
BrowserFactory.CloseAllDrivers();
}
}
This is the error I get:
Message: OpenQA.Selenium.WebDriverException : Unexpected error. System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it
with a bunch of other stuff that I don't believe is relevant.
That being said, I have code in "SetBrowser(...)" that initializes a
new ChromeDriver();
but that doesn't seem to be enough.
The methods at the top are there to avoid having to do too much Selenium-y looking stuff in each of the tests - to make things more maintainable by people other than just myself - but it's what you would expect from typical Driver setup. I'm not sure why the second test is what causes the issue, but since things work fine w/o the CloseAllDrivers() being run, I'm imagining it's that.
By the looks of the attributes, you're using MSTest? Is it executing tests in parallel?
I guess regardless, for good test isolation you would want to change the CloseAllDrivers method to only dispose the driver used in the test it's cleaning up. I'd recommend starting there and see if that has the same problem.
Also, is CloseAllDrivers calling driver.Quit() or driver.Dispose()? Either of those are the ones you want to use.
OK so the issue was a combination of things:
1. I had Drivers[driver].Close(); and Drivers[driver].Quit(); -- only having .Quit() resolved one issue. Not really sure why Close() was causing problems, to be quite honest. The other (window not properly closing in the end) was due to the following code in my BrowserFactory:
if (driver == null)
{
Driver = new ChromeDriver((ChromeOptions)options); // options created elsewhere
Drivers.Add("Chrome", Driver); // This adds the driver to the list of Drivers currently up.
}
else
{
Driver = new ChromeDriver((ChromeOptions)options); // same as before
Drivers["Chrome"] = Driver; // **this** wasn't here before. This was the issue. Essentially, I was calling ```Quit()``` on the first instance of the driver, not on the fresh one created by the second test.
}
Thanks for all the help, guys. A combination of me determined to figure this out and your responses got me to the solution :)
I have about 12 unit tests for different scenarios, and I need to call one async method in these tests (sometimes multiple times in one test). When I do "Run all", 3 of them will always fail. If I run them one by one using "Run selected test", they will pass. The exception in output I'm getting is this:
System.AppDomainUnloadedException: Attempted to access an unloaded
AppDomain. This can happen if the test(s) started a thread but did not
stop it. Make sure that all the threads started by the test(s) are
stopped before completion.
I can't really share the code, as it's quite big and I don't know where to start, so here is example:
[TestMethod]
public async Task SampleTest()
{
var someProvider = new SomeProvider();
var result = await someProvider.IsSomethingValid();
Assert.IsTrue(result == SomeProvider.Status.Valid);
NetworkController.Disable();
result = await someProvider.IsSomethingValid();
Assert.IsTrue(result == SomeProvider.Status.Valid);
NetworkController.Enable();
}
EDIT:
The other 2 failing methods set time to the future and to the past respectively.
[TestMethod]
public async Task SetTimeToFutureTest()
{
var someProvider = new SomeProvider();
var today = TimeProvider.UtcNow().Date;
var result = await someProvider.IsSomethingValid();
Assert.IsTrue(result == SomeProvider.Status.Valid);
TimeProvider.SetDateTime(today.AddYears(1));
var result2 = await someProvider.IsSomethingValid();
Assert.IsTrue(result2 == SomeProvider.Status.Expired);
}
Where TimeProvider looks like this:
public static class TimeProvider
{
/// <summary> Normally this is a pass-through to DateTime.Now, but it can be overridden with SetDateTime( .. ) for testing or debugging.
/// </summary>
public static Func<DateTime> UtcNow = () => DateTime.UtcNow;
/// <summary> Set time to return when SystemTime.UtcNow() is called.
/// </summary>
public static void SetDateTime(DateTime newDateTime)
{
UtcNow = () => newDateTime;
}
public static void ResetDateTime()
{
UtcNow = () => DateTime.UtcNow;
}
}
EDIT 2:
[TestCleanup]
public void TestCleanup()
{
TimeProvider.ResetDateTime();
}
Other methods are similar, I will simulate time/date change, etc.
I tried calling the method synchronously by getting .Result() out of it, etc, but it didn't help. I read ton material on the web about this but still struggling.
Did anyone run into the same problem? Any tips will be highly appreciated.
I can't see what you're doing with your test initialization or cleanup but it could be that since all of your test methods are attempting to run asynchronously, the test runner is not allowing all tasks to finish before performing cleanup.
Are the same few methods failing when you run all of the tests or is it random? Are you sure you are doing unit testing and not integration testing? The class "NetworkController" gives me the impression that you may be doing more of an integration test. If that were the case and you are using a common class, provider, service, or storage medium (database, file system) then interactions or state changes caused by one method could affect another test method's efficacy.
When running tests in async/await mode, you will incur some lag. It looks like all your processing is happening in memory. They're probably passing one an one-by-one basis because the lag time is minimal. When running multiple in async mode, the lag time is sufficient to cause differentiation in the time results.
I've run into this before doing NUnit tests run by NCrunch where a DateTime component is being tested. You can mitigate this by reducing the scope of your validation / expiration logic to match to second instead of millisecond, as long as this is permissible within your acceptance criteria. I can't tell from your code what the logic is driving validation status or expiration date, but I'm willing to bet the async lag is the root cause of the test failure when run concurrently.
Both tests shown use the same static TimeProvider, thus interference by methods like ResetDateTime in the cleanup and TimeProvider.SetDateTime(today.AddYears(1)); in a test are to be expected. Also the NetworkController seems to be a static resource, and connecting/disconnecting it could interfere with your tests.
You can solve the issues in several ways:
get rid of static resources, use instances instead
lock the tests such that only one test can be run at a time
Aside from that, almost every test framework offers more than just Assert.IsTrue. Doesn't your framework offer an Assert.AreEqual? That improves readabilty. Also, with more than one Assert in a test, custom messages indicating which of the test failed (or that an Assert is for pre-condition, not the actual test) are recommended.
I was wondering if it was possible to have a start-up script before running any load tests? For example, perhaps to seed some data or clear anything down prior to the tests executing.
In my instance I have a mixed bag of designer and coded tests. Put it simply, I have:
Two coded tests
A designer created web test which points to these coded tests
A load test which runs the designer
I have tried adding a class and decorating with the attributes [TestInitialize()], [ClassInitialize()] but this code doesn't seem to get run.
Some basic code to show this in practice (see below). Is there a way of doing this whereby I can have something run only the once before test run?
[TestClass]
public class Setup : WebTest
{
[TestInitialize()]
public static void Hello()
{
// Run some code
}
public override IEnumerator<WebTestRequest> GetRequestEnumerator()
{
return null;
}
}
Probably should also mention that on my coded tests I have added these attributes and they get ignored. I have come across a workaround which is to create a Plugin.
EDIT
Having done a little more browsing around I found this article on SO which shows how to implement a LoadTestPlugin.
Visual Studio provides a way of running a script before and also after a test run. They are intended for use in deploying data for a test and cleaning up after a test. The scripts are specified on the "Setup and cleanup" page in the ".testsettings" file.
A load test plugin can contain code to run before and after any test cases are executed, also at various stages during test execution. The interface is that events are raised at various points during the execution of a load test. User code can be called when these events occur. The LoadTestStarting event is raised before any test cases run. See here for more info.
If you are willing to use NUnit you have SetUp/TearDown for a per test scope and TestFixtureSetUp/TestFixtureTearDown to do something similar for a class (TestFixture)
Maybe a bit of a hack, but you can place your code inside the static constructor of your test class as it will automatically run exactly once before the first instance is created or any static members are referenced:
[TestClass]
public class Setup : WebTest
{
static Setup()
{
// prepare data for test
}
public override IEnumerator<WebTestRequest> GetRequestEnumerator()
{
return null;
}
}
Is there anyway to add an attribute to a [Test] method in a [TestFixture] so that only that method runs? This would be similar to the way the [CurrentFixture] attribute can be used to only run a single fixture. I ask as sometimes when I test the model I want to profile the sql being executed and I only want to focus on a single test. Currently I have to comment out all the other tests in the fixture.
Updated:
The code I'm using to initiate the test follows, I'm really looking for a solution I can weave into this code.
public static void Run(bool currentFixturesOnly) {
using(AutoRunner auto = new AutoRunner()) {
if(currentFixturesOnly) { // for processing [CurrentFixture]s only
auto.Domain.Filter = FixtureFilters.Current;
}
auto.Verbose = true;
auto.Run();
auto.ReportToHtml();
}
}
If you use a test runner like TestDriven.Net, ReSharper or Icarus then you can select the specific test to run and just run that. If you're using the command-line tools, consider using a filter.
eg.
Gallio.Echo MyTestAssembly.dll /f:Name:TheNameOfTheParticularIWantToRun