I am writing some unit tests for the persistence layer of my C#.NET application. Before and after the tests of a test class execute, I want to do some cleaning up to erase possibly inserted dummy values, therefore, this cleaning up happens in methods marked with the attributes [ClassInitialize()] and [ClassCleanup()].
(I know that a better way would be to use an in-memory database, but it is not really doable so far as we depend on lots of stored procs....)
I would like to output some information about the results of the cleaning up, but I can not find a way to get the output in the test results with VISUAL Studio 2010.
This is what I am doing so far :
///... lots of stuff before ...
//global for the test run
private static TestContext context;
//for each test
private IRepository repo;
#region Initialisation and cleanup
/// <summary>
/// Execute once before the test-suite
/// </summary>
[ClassInitialize()]
public static void InitTestSuite(TestContext testContext)
{
context = testContext;
removeTestDataFromDb();
}
[ClassCleanup()]
public static void CleanupTestSuite()
{
removeTestDataFromDb();
}
private static void removeTestDataFromDb()
{
context.WriteLine("removeTestDataFromDb starting");
using (ISession session = NHibernateHelper.OpenSession())
{
IDbConnection cn = session.Connection;
IDbCommand cmd = cn.CreateCommand();
//remove anyt test data
cmd.CommandText = #"DELETE FROM SomeTable
WHERE somefield LIKE 'easyToFindTestData%Test'";
int res = cmd.ExecuteNonQuery();
context.WriteLine("removeTestDataFromDb done - affected {0} rows", res);
}
}
[TestInitialize()]
public void InitTest()
{
repo = new MyRepositoryImplementation();
}
[TestCleanup()]
public void CleanupTest()
{
//cleanup
repo = null;
}
#endregion
I'm trying to use context.WriteLine() ...
I also tried just using Console.WriteLine() with the same results.
How do you write to standard output in the ClassInitialize part and where can you access that output ?
The [ClassInitialize] and [ClassCleanup] run just once for all the tests in that class. You'd be better of using [TestInitialize] and [TestCleanUp] which run before and after each test. Also try wrapping the complete test in a database transaction. This way you can simply rollback the operation (by not committing the transaction) and your database stays in a consistent state (which is essential for trustworthy automated tests).
A trick I do for integration tests is to define a base class that all my integration test classes can inherit from. The base class ensures that each test is ran in a transaction and that this transaction is rolled back. Here is the code:
public abstract class IntegrationTestBase
{
private TransactionScope scope;
[TestInitialize]
public void TestInitialize()
{
scope = new TransactionScope();
}
[TestCleanup]
public void TestCleanup()
{
scope.Dispose();
}
}
Good luck.
The trace output from a ClassInitialize and ClassCleanup appears in the result summary.
You can access it by doing the following
Open the Test Results windw [ Test -> Windows -> Test Results ]
There should be a link named "Test run completed" on the top left corner of the [Test Results] window.
Click the clink
It should open a window with the header "Result Summary" and it will show the debug trace created during ClassInitialize and ClassCleanup
You can see the Console output on each test if you double-click the test method in the Test Results pane. It is also present in the .trx xml results file.
In addition, if you specify the "Define DEBUG constant", you can use the
System.Diagnostics.Debug.WriteLine("ClassInitialize Method invoked, yeah.");
.. which will end up in the "Output" pane.
Related
I've spent several hours debugging the issue when log messages are not shown in console output from the class method, which is called via NUnit TestCaseSource.
So I have a class, where I perform logging for debugging purposes.
public class TestHelper
{
private readonly Logger logger;
public TestHelper()
{
logger = LogManager.GetCurrentClassLogger();
}
public IEnumerable<int> GetTestData()
{
List<int> testData = new();
for (var i = 0; i < 10; i++)
{
logger.Info("This message is ignored when is called from NUnit test data provider {i}", i);
testData.Add(i);
}
return testData;
}
}
I have a test fixture:
public class DemoTestFixture
{
private static readonly ClassWithLoggingInside ClassWithLoggingInside = new();
private static readonly TestHelper TestHelper = new();
private readonly Logger logger = LogManager.GetCurrentClassLogger();
[Test]
[TestCaseSource(nameof(GetTestData))]
public void LoggingFromTestDataSourceIsIgnored(int i)
{
Assert.DoesNotThrow(() => ClassWithLoggingInside.Log());
logger.Debug("Message from test fixture {i}", i);
}
[Test]
public void LoggingIsShown()
{
Assert.DoesNotThrow(() => TestHelper.GetTestData());
}
private static IEnumerable<int> GetTestData()
{
IEnumerable<int> testData = TestHelper.GetTestData();
foreach (int i in testData) yield return i;
}
}
And when the TestHelper.GetTestData() is called from the parameterized test - console output from the method is NOT shown, however, the messages are logged into the file.
When the method is called not as a part of the test data provider - the messages are perfectly logged into the console target.
I'm using latest NLog logger, .NET5, latest NUnit. Tried the following with log4net logger - same result.
Tests were run using Resharper/dotnet test command.
I suspect that the issue lies somewhere in the static initialization, but can't understand why file target logging works fine and there are problems with console output. Do you have any explanation?
The demo project can be found here.
The answer is relatively simple but probably not helpful to you. :-(
The reason any logging you do do the console appears in the NUnit output is that NUnit captures console output while it is running tests.
However, execution of your test case source doesn't take place while NUnit is running the tests, but while it is discovering tests. Discovery happens ages (in computer time) before it ever begins test execution.
I suspect that this is made more confusing because we allow you to place the data source in the same class as the tests themselves, although it may also be in a separate class. Nevertheless, the execution of that static method is not part of running the test. In fact, it's only after that method is run that NUnit even knows how many tests there are.
If you need to your logged output to be captured and displayed by NUnit, it must be done in the test method itself or in one of the setup or teardown methods.
I have a question on running UnitTests sequentially. Unfortunately in scenario it is not an option to run them parallel or mock the database. The project is written in .NET core 3.1 and the UnitTests need to execute database operations before and after a Unittest has run.
After reading https://www.meziantou.net/mstest-v2-execute-tests-in-parallel.htm and a lot of other articles about sequential UnitTesting I came up with this (simplified):
BaseClass:
namespace XY.Test
{
[TestClass]
public class BaseTest: TimerModel
{
private static readonly DbCreator Creator = new DbCreator();
public static readonly DbConnectionManager ConnectionManager = new DbConnectionManager();
[TestInitialize]
public void BaseTestInitialize()
{
CreateTestData();
}
[TestCleanup]
public void BaseTestCleanup()
{
RemoveTestData();
}
public void CreateTestData()
{
RemoveTestData();
Creator.ExecuteSqlFromScript(ConnectionManager, #"Resources\CreateTestData.sql");
}
public void RemoveTestData()
{
Creator.ExecuteSqlFromScript(ConnectionManager, #"Resources\EmptyTestDataTables.sql");
}
}
}
TestClass:
[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)] //<-- Also tried out Workers = 1 and Scope = ExecutionScope.MethodLevel
namespace XY.Test.Models
{
[TestClass]
public class TerminalConfigModelTest: BaseTest
{
[TestMethod]
[DoNotParallelize]
public void TestMethod1()
{
...
}
[TestMethod]
[DoNotParallelize]
public void TestMethod2()
{
...
}
}
}
For some reason, no matter what I do, the UnitTests are being executed parallel. What do I have to change in order to have them executed sequentially?
When I execute all tests in the test class, the TestInitialize of the base class is called twice before the TestCleanup is run. This causes the CreateTestData method to fail as indexes prevent a double insert of the test data.
What I would expect:
TestInitialize1 is called
TestMethod1 is executed
TestCleanup1 is called
TestInitialize2 is called
TestMethod2 is executed
TestCleanup2 is called
...
What happens:
TestInitialize1 is called
TestMethod1 is executed
TestInitialize2 is called before TestCleanup1 is called
TestMethod2 execution fails
Am I missunderstanding the [DoNotParallelize] option?
Paralelism isn't the problem here, my tests are definitely sequential and [ClassCleanup] also screwed me over. It's just unintuitive and weird, more info here.
I wanted to use ordered tests but it seems it's a legacy functionality only in MSTest-v1 and new versions of Visual Studio don't even support it.
Best thing i can tell you now is just don't use [ClassCleanup].
Use [TestCleanup] or [AssemblyCleanup].
On the xUnit website it says the following about constructor:
xUnit.net creates a new instance of the test class for every test that
is run, so any code which is placed into the constructor of the test
class will be run for every single test. This makes the constructor a
convenient place to put reusable context setup code where you want to
share the code without sharing object instances (meaning, you get a
clean copy of the context object(s) for every test that is run).
I have the following code:
public class ProfilePageTest
{
public ProfilePageTest(Role role)
{
AuthRepository.Login(role)
}
[Theory]
[Roles(Role.Editor, Role.Viewer)]
public void OpenProfilePageTest(Role role)
{
var profile = GetPage<ProfilePage>();
profile.GoTo();
profile.IsAt();
}
}
Is it possible to pass the role from the theory attribute to the constructor, so I don't have to do AuthRepository.Login(role) in every test method that I have.
No, that's not possible. The constructor will be run before anything else, as with any constructor you're used to. I don't see the harm in calling AuthRepository.Login(role) in every test though, because it's a single line of code.
This is quite an excellent blog post about the different ways you can pass data into xUnit tests, but all of them are passing in data to individual methods (tests) rather than in the constructor.
If you are looking to set something up for multiple tests you should have a look int IClassFixture
Quick run down, you setup a class with the shared data:
public class DatabaseFixture : IDisposable
{
public DatabaseFixture()
{
Db = new SqlConnection("MyConnectionString");
// ... initialize data in the test database ...
}
public void Dispose()
{
// ... clean up test data from the database ...
}
public SqlConnection Db { get; private set; }
}
And then in your tests you can "inject" the class (along with the data) into the test class:
public class MyDatabaseTests : IClassFixture<DatabaseFixture>
{
DatabaseFixture fixture;
public MyDatabaseTests(DatabaseFixture fixture)
{
this.fixture = fixture;
}
// ... write tests, using fixture.Db to get access to the SQL Server ...
}
first of all I'm new to SpecFlow.
I have a feature file which I have / want to automate using MSTest to run as a functional test involving a fully set up server, data access ...
For this purpose I have to configure the server with the data in the SpecFlow's 'Given' blocks and start it afterwards. I also have to copy some files to the test's output directory.
In the non-SpecFlow functional tests I was using the ClassInitialize attribute to get the TestDeploymentDir from the TestContext; something like this:
[ClassInitialize]
public static void ClassSetup(TestContext context)
{
TargetDataDeploymentRoot = context.TestDeploymentDir;
}
Now with SpecFlow I can't use this attribute anymore as it is used by SpecFlow itself.
Some new attributes do exist, like BeforeFeature which acts similarly BUT it doesn't pass on the TestContext as a parameter.
I just need to get access to the TestContext's TestDeploymentDir in order to copy some files there before really lauching my functional test server - easily doable without SpecFlow but almost impossible with SpecFlow.
How to deal with this issue?
Is it possible at all?
Thanks a lot for advice!
robert
Environment:
Visual Studio 2012
SpecFlow 1.9.0.77
Since SpecFlow 2.2.1 the TestContext is available via Context Injection. (https://github.com/techtalk/SpecFlow/pull/882)
You can get it from the container directly:
ScenarioContext.Current.ScenarioContainer.Resolve<Microsoft.VisualStudio.TestTools.UnitTesting.TestContext>()
or via context injection:
public class MyStepDefs
{
private readonly TestContext _testContext;
public MyStepDefs(TestContext testContext) // use it as ctor parameter
{
_testContext = testContext;
}
[BeforeScenario()]
public void BeforeScenario()
{
//now you can access the TestContext
}
}
In order to have access to values in the TestContext you have to create partial class for each scenario file you have in which you add the .
using Microsoft.VisualStudio.TestTools.UnitTesting;
using TechTalk.SpecFlow;
/// <summary>
/// Partial class for TestContext support.
/// </summary>
public partial class DistributionFeature
{
/// <summary>
/// Test execution context.
/// </summary>
private TestContext testContext;
/// <summary>
/// Gets or sets test execution context.
/// </summary>
public TestContext TestContext
{
get
{
return this.testContext;
}
set
{
this.testContext = value;
//see https://github.com/techtalk/SpecFlow/issues/96
this.TestInitialize();
FeatureContext.Current["TestContext"] = value;
}
}
}
Then you could access the deployment directory from your steps using
var testContext = (TestContext)FeatureContext.Current["TestContext"];
var deploymentDir = testContext.TestDeploymentDir;
If you have too many scenarios, then you probably has to automate creation of such files with T4.
You can create a Plugin and customize the IUnitTestGeneratorProvider implementation. The following should add the line to MSTest's class initialize.
// It's very important this is named Generator.SpecflowPlugin.
namespace MyGenerator.Generator.SpecflowPlugin
{
public class MyGeneratorProvider : MsTest2010GeneratorProvider
{
public MyGeneratorProvider(CodeDomHelper codeDomHelper)
: base(codeDomHelper)
{
}
public override void SetTestClassInitializeMethod(TestClassGenerationContext generationContext)
{
base.SetTestClassInitializeMethod(generationContext);
generationContext.TestClassInitializeMethod.Statements.Add(new CodeSnippetStatement(
#"TargetDataDeploymentRoot = context.TestDeploymentDir;"));
}
}
[assembly: GeneratorPlugin(typeof(MyGeneratorPlugin))]
public class MyGeneratorPlugin : IGeneratorPlugin
{
public void RegisterDependencies(ObjectContainer container)
{
}
public void RegisterCustomizations(ObjectContainer container, SpecFlowProjectConfiguration generatorConfiguration)
{
container.RegisterTypeAs<MyGeneratorProvider, IUnitTestGeneratorProvider>();
}
public void RegisterConfigurationDefaults(SpecFlowProjectConfiguration specFlowConfiguration)
{
}
}
}
And reference it in the App.config file:
<specFlow>
<plugins>
<add name="MyGenerator" type="Generator"/>
</plugins>
</specFlow>
Next time you re-save the .feature files the generated code in ClassInitialize should set the TargetDataDeploymentDirectory.
I had to do something similar. Here's my working code https://github.com/marksl/Specflow-MsTest and blog post http://codealoc.wordpress.com/2013/09/30/bdding-with-specflow/
There is a FeatureContext as well as the more commonly used
ScenarioContext. The difference of course is that the FeatureContext
exists during the execution of the complete feature while the
ScenarioContext only exists during a scenario.
For example:
Add to context:
ScenarioContext.Current.Add("ObjectName", myObject);
Get:
var myObject = ScenarioContext.Current.Get<object>("ObjectName");
You can read more about it here.
I have the following base object in my Tests
[TestClass]
public abstract class TestClassBase
{
private TransactionScope _transactionScope;
[TestInitialize]
public virtual void TestInitialize()
{
_transactionScope = new TransactionScope(TransactionScopeOption.RequiresNew, new TransactionOptions { Timeout = new TimeSpan(0, 10, 0) });
}
[TestCleanup]
public virtual void TestCleanup()
{
_transactionScope.Dispose();
}
}
I have a Test that does the following
[TestMethod]
public void SaveToDatebaseTest(
{
Program program = new Program();
program.Name = "Unit Test Program, I should not exists, when Test is done";
ProgramManager programManager = new ProgramManager()
programManager.Save(program);
}
When I run the test, the records still exists in the database.
I want to avoid using TransactionScope within every test method
You need to change your TestCleanup method, right now you Dispose of the TransactionScope, I believe that is actually doing an implicit commit? (you would think you would need to call Complete() though?), since there are no errors it is not Rolling back the transaction.
try this
[TestCleanup]
public virtual void TestCleanup()
{
// using System.Transactions;
Transaction.Current.Rollback();
_transactionScope.Dispose();
}
i know this question is kind of old, but it may help out someone who might/will be struggling on this in the future like i just did earlier. so if you're using
Entity Framework
Nunit (v3)
the code below won't work.
class A
{
private TransactionScope _trans;
[SetUp]
public void setup()
{
_trans = new TransactionScope();
}
[TearDown]
public void done()
{
if(_trans != null)
_trans.Dispose();
}
[Test]
public void doSomeDbWrite()
{
//your code to insert/update/delete data in db
}
}
I have tried (and doesn't work) creating the TransactionScope before creating the DB Context or the other way around. I think it has something to do with EF itself which i suppose is encapsulated in their own transactions or whatever. i did not dig deeper on that part. anyway, here's how i did it with EF and transactions to ensure that your unit test DB is clean after the unit test is done.
class A
{
private DbContext_DB;
private DbContextTransaction _trans;
[SetUp]
public void setup()
{
DB = new DbContext();//create your db context
_trans = DB.Database.BeginTransaction();
}
[TearDown]
public void done()
{
_trans.Rollback();
DB = null;
}
}
Hopefully it will help others searching for this at this time :-)
In .NET Core async Task [TestInitialize] methods are unsupported and will result in the behaviour you describe.
Async [TestInitialize] methods appear to run in a different thread context to the [TestMethod] itself, even if TransactionScopeAsyncFlowOption.Enabled is set in the TransactionScope's options. Transaction.Current will be null in the test method and any changes will not be rolled back when [TestCleanup] disposes the TransactionScope.
Unfortunately there are no plans to support async flow in TestInitialize methods despite this working in older versions of .NET (.NET Framework 4.8 at least).
The only work around at present is to make the [TestInitialize] method non-async and .Wait() / .Result your async calls but please upvote the GitHub issue above to show support for this feature.
If you are using VS2005 or VS2008 (not sure about 2010), you can try MSTestExtensions for automatic database transaction rollback after tests methods are completed. I've used it with MS SQLServer and it's Distributed Transaction Coordinator service running.
I know this may not be exactly what you are looking for, but it may be of help to your situation.
I think what you are trying to achieve here is to create a generic Test class that implements the initialize and cleanup so you don't have to repeat that code for every test, is that correct? Are you even sure that those methods are even called? Be aware that you can't subclass [TestInitialize] and [TestCleanup], so you would have to call those methods from the sublcass. (See https://stackoverflow.com/a/15946140/1638933).
I have the same issue but I used the following code to fix the issue Maybe it can help you:
private readonly TransactionScope _scope;
public MyTests()
{
var options = new TransactionOptions()
{
IsolationLevel = IsolationLevel.Snapshot
};
_scope = new TransactionScope(TransactionScopeOption.Required, options, TransactionScopeAsyncFlowOption.Enabled);
}
[TestCleanup]
public void Cleanup()
{
_scope.Dispose();
}