We have an application that has many SPROCs being developed and maintained by multiple developers, and we are trying to automate the process to keep track of modifying and testing the SPROCs. We currently have a table in our database that is populated and modified based on a trigger that fires when a SPROC is created, modified, or deleted. In this table there is a column that specifies whether the SPROC was tested and deemed a success by a unit test. We are using Visual Studio's Test Explorer and Unit Test designer to handle the SQL Unit tests. We have them functioning fine, but are trying to add automate to update the database after a test succeeds. Is there some kind of event or something similar that is touched by every successful unit test? If not, then at least something that can catch the results and allow some kind of additional logic after a(n) (un)successful execution?
Within the TestMethod itself, one of the objects returned is the SqlExecutionResult[] testResults object. Within this object is the hasError attribute, that when successful is set to true. It seems testResults isn't populated on some errors and is only ever null. Is there some method or something similar called by ALL unit tests upon completion that might be able to look back/use testResults to get confirmation of success? Something that can be used and catch the output from all unit tests?
We found the results using a couple of slightly adjacent posts.
It comes down to creating a base test class that only has the TestCleanup() method and referencing the TestContext.CurrentTestOutcome.
You will then have the other test classes reference this as a base class and remove the reference to TestCleanup() in all of the other tests. This will allow for any kind of extra work to be done based on a success or failure of a unit test. To avoid this extra work, you could probably create a template using the base class. However, at this time we are not putting in the effort to figure this out since it's not a necessity.
These are the posts we referenced:
How to get test result status from MSTest?
In MSTest how to check if last test passed (in TestCleanup)
This is something for all tests executed, not specifically passing tests only. You can create a stored procedure to reset and update that testing table/db as you need, and then use the [SetUp] method to ensure it triggers before every test is executed
As a basic example, using nUnit I have done
private IMyRepo _repo;
[SetUp]
public void Init()
{
_repo = new MyRepoImpl();
//reset db to a known state for each test
using (SqlConnection c = new SqlConnection(Settings.GetConnString()))
{
SqlCommand cmd = new SqlCommand
{
Connection = c,
CommandText = "DbReset",
CommandType = CommandType.StoredProcedure
};
c.Open();
cmd.ExecuteNonQuery();
}
}
You can setup the DbReset stored procedure used here to update that specific column
But a major caveat here is that this will fire before every test, not just successful or failed ones, and I'm not sure if there is such a specialization for a something to trigger for only passed tests.
In nUnit, [TearDown] methods are guaranteed to fire assuming the [SetUp] method did not throw an exception so you will have the same issue here. If there was a scenario where [TearDown]'s don't fire if the test fails, that could've been a hacky approach to solve your problem, but these types of methods are typically aimed at object creation and cleanup, so I highly doubt there is another testing suite that will attempt to do this (as, even in a failed test, the developer would still want cleanup to take place)
Sorry I can't provide the fix to the exact scenario you have, but I hope this gets you closer to your answer
Related
I haven't been coding for too long, so apologies if this isn't articulated very well.
So, I've been trying to get my head around using constructors and IDisposable in order to write unit tests in xunit that test some functions relating to creating and deleting files. Initially I started writing tests without any sort of 'setup'/'teardown' equivalent, but I ran into the problem of tests passing on first run, but failing second time round as the file I was testing could be deleted had already been deleted in the previous instance of running the test! So, now I'm trying to include 'File.Create("file-path")', 'File.Delete("file-path")' type methods in my constructor, but find that my tests all break as soon as I incorporate one of these methods. This doesn't happen with 'Directory.CreateDirectory()', or 'Directory.Delete()'.
To exemplify, here is an example of the sort of thing I have been trying with the constructor...
public class SetUp : IDisposable
{
protected SetUp()
{
Directory.CreateDirectory(#"c:\Projects\Tests\Test");
File.Create(#"c:\Projects\Tests\test.txt");
}
public void Dispose()
{
Directory.Delete(#"c:\Projects\Tests\Test", true);
File.Delete(#"c:\Projects\Tests\test.txt");
}
}
This is necessary for the following test:
[Theory]
[InlineDataAttribute(#"c:\Projects\Tests\test3.txt")]
public void CanRemoveFile(string filePath)
{
//Assign
var myInstanceOfApphelper = new AppHelper();
bool fileExists = true;
//Act
myInstanceOfApphelper.RemoveFile(filePath);
fileExists = myInstanceOfApphelper.CheckFileExists(filePath);
//Assert
Assert.Equal(false, fileExists);
}
...as without this constructor part to create the file before running the tests, there will be no file at that location to delete next time I run the test, and it will fail. However, as soon as I include the line File.Create(#"c:\Projects\Tests\test.txt");, all of my tests break. The errors I am returned are all
"The process cannot access the file 'c:\Projects\Tests\test.txt'
because it is being used by another process."
but I can't understand this as most of the tests don't use 'c:\Projects\Tests\test.txt anyway!
Can anyone tell me why this won't work? How could I go about testing the creation and deletion of files in the xunit framework in such away that the tests consistently pass? Can you point out any obvious errors with the way I've written this?
The behaviour you describe means that you keep a handle on the file between two tests. I suggest you double check the way your setup and teardown works because it seems like you're doing it wrong (not calling them before and after each tests).
Bonus: I recommend not using the real file system while doing unit tests. There are in memory file systems which allows to completely clean your workspaces and be sure you are not bothered by the physical file system.
For instance check: https://github.com/bobvanderlinden/sharpfilesystem
Is there any way I can filter test results by specifying a keyword that should NOT appear?
Context:
I've written some C# classes and methods, but did not implement those methods for now (I made them throw a NotImplementedException so that they clearly indicate this). I also written some test cases for those functions, but they currently fail because the methods throw the NotImplementedException. This is ok and I expect this for now.
I want to ignore these tests for now and look at other test results that are more meaningful, so I was trying to figure out how I can list results that do not have the "NotImplementedException". However, I can only list the results that do have that keyword, not those that don't. Is there any way I can list the results that don't? Using some wildcards or something?
I see a lot of information about the new Test Explorer in VS2012, but that's not a feature in 2010, which is what I'm using.
You can sort of cheat to pass this tests, if you want to, by marking that this test expects an exception to be thrown and thereby passes the test.
[TestMethod]
[ExpectedException(typeof(NotImplementedException))]
public void NotYetImplementedMethod(Object args)
{
....
}
Alternatively you can create categories for your tests. This way you can choose which tests to run in the Test explorer, if you assign a category to most of your tests.
[TestMethod]
[Testcategory("NotImplementedNotTested")]
public void NotYetImplementedMethod(Object args)
{
....
}
Last but not least the simplest solution [Ignore]. This will skip the tests alltogether.
[TestMethod]
[Ignore]
public void NotYetImplementedMethod(Object args)
{
....
}
Reference:
http://channel9.msdn.com/Shows/Visual-Studio-Toolbox/Getting-Started-with-Unit-Testing-Part-1
http://channel9.msdn.com/Shows/Visual-Studio-Toolbox/Getting-Started-with-Unit-Testing-Part-2
How to create unit tests which runs only when manually specified?
I also written some test cases for those functions
If your tests are linked to Test Cases work items on TFS you could simply set the Test Case's State to Design. Then, on your Test Plans exclude all test cases that are on Designed state.
If they are not linked to actual Test Cases work items (let's say a batch of unit tests), I believe the best solution is the Ignore attrbute (as #Serv already mentioned). Because I don't think you want to run tests that are not implemented yet and also waste time to find out how to exclude them from test results.
I have several suites of integration tests implemented in C#/NUNit. Each test suite is a separate class, each fixture setup creates and populates a SQL Server database from scripts. This all used to work just fine prior to Resharper 5.1.
Unfortunately, Resharper 5.1 begins to run several fixtures at the same time. This is a breaking change - they are all attempting to create and populate the same database, which obviously ends up in an mess. Is there any way I could have Resharper run my test fixtures serially?
If not, what would you suggest to run my NUnit test fixtures serially, one fixture at a time?
The order in which individual tests run does not matter.
I don't know whether it is possible to prevent ReSharper from running tests in parallel; if not, the following hack might work: Create a static class with a static readonly Monitor member. Then, in [TestFixtureSetUp], call Enter() on the monitor, and call Exit() on the monitor in [TestFixtureTearDown]. That way, only one test fixture will be allowed to run at a time. Not pretty, though...
Are you sure about this ? I just tried this out.. by putting a trace of the following form in tests in 3 diff NUnit fixtures followed by a "Run all". Doesn't seem to be running in parallel.
Trace.WriteLine(DateTime.Now.ToString("hh:mm:ss.ffff") + "VC:Start");
Trace.WriteLine(DateTime.Now.ToString("hh:mm:ss.ffff") + "VC:Done");
Output I see is : (R# Build 5.1.1753.1)
01:06:41.6639IOC
01:06:41.6649Done - IOC
01:06:41.6679VC:Start
01:06:41.6729VC:Done
01:06:41.2439Mef
01:06:41.6589Mef-Done
Unless the code being tested does its own transaction management, you can run each test inside a transaction. In that way, the different tests won't disturb each other. Also, you don't have to worry about cleaning up after each test, since the each test's transaction can simply abort after the test is completed.
In our projects, we usually let our integration tests derive from a class that looks more or less like this:
public class TransactionalTestFixture
{
private TransactionScope TxScope;
[SetUp]
public void TransactionalTestFixtureSetUp()
{
TxScope = new TransactionScope(TransactionScopeOption.RequiresNew,
new TransactionOptions {IsolationLevel = IsolationLevel.Serializable});
}
[TearDown]
public void TransactionalTestFixtureTearDown()
{
TxScope.Dispose();
}
}
Give them alphabetical names, i.e. prefix them with a letter that signifies their running order. If you need this to happen you should be able to accept this nasty naming convention too.
I am working in an application that is mostly single-thread, single user.
There are a few worker threads here and there, and they are only using thread safe objects and classes. The unit tests are actually testing those with multiple threads (explicitly created for the tests), and they test fine.
The VSTS unit tests fail when testing business objects and sub-systems that are not thread safe. It is okay for them not to be thread-safe, that's the way the application uses them.
But the 'one thread per TestMethod' approach of MS tests kills us. I've had to implement object locks in many unit test classes just to ensure that the tests are run one after the other (I don't really care about the order, but I can't have two test methods hitting the same object at the same time).
The code looks like this:
[TestClass]
public class TestSomeObject
{
static object turnStile = new object();
...
[TestMethod]
public void T01_TestThis()
{
lock(turnStile)
{
.. actual test code
}
}
[TestMethod]
public void T02_TestThat()
{
lock(turnStile)
{
-- actual test code
}
}
}
Is there a better/more elegant way to make the test run sequentially?
Use an Ordered Test.
Test > New Test > Ordered Test
you can Use Playlist
right click on the test method -> Add to playlist -> New playlist
you can then specify the execution order
There is the notion of an "Ordered Test" in which you can list tests in sequence. It is more geared towards ensuring a certain sequential order, but I can't see how that would be possible if B doesn't wait for A to complete.
Apart from that, it is unfortunate that your tests interfere with each other. There are Setup / TearDown methods that can be used per test such that it may after all be possible to isolate the tests from each other.
You can specifically require a mutex for each test execution, either in the specific tests you want to serialize, or for all the tests in a class (whatever shares the same mutex string).
For an entire test class, you can use the TestInitialize and TestCleanup attributes like so:
private readonly Mutex testMutex = new Mutex(true, "MySpecificTestScenarioUniqueMutexString");
[TestInitialize]
public void Initialize()
{
testMutex.WaitOne(TimeSpan.FromSeconds(1));
}
[TestCleanup]
public void Cleanup() {
testMutex.ReleaseMutex();
}
To be clear this isn't a feature of tests, ANY locking structure should work. I'm using the system provided Mutexes in this case:
https://msdn.microsoft.com/en-us/library/system.threading.mutex(v=vs.110).aspx
I finally used the ordered test method. It works well.
However, I had a hell of a time making it work with the NAnt build.
Running only the ordered test list in the build requires using the /testmetadata and /testlist switches in the MSTest invocation block.
The documentation on these is sketchy, to use a kind description. I google all over for examples of "MSTest /testmetadata /testlist" to no effect.
The trick is simple, however, and I feel compelled to give it back to the community, in case someone else bumps into the same issue.
Edit the test metadata file (with a .vsmdi extension), and add a new list
to the list of tests (the first node in the tree on the left
pane. Give it the name you want, for example 'SequentialTests'.
If you used a /testcontainer switch for the MSTest invocation, remove it.
Add a switch for MSTest
-> /testmetadata:
Add a switch for MSTEst
/testlist:SequentialTests (or whatever name you used)
Then MSTest runs only the tests listed in the test list you created.
If someone has a better method, I'd like to hear about it!
I used ordered tests, also configured them easily on jenkins just use command
MSTest /testcontainer:"orderedtestfilename.orderedtest" /resultsfile:"testresults.trx"
When writing a test for persistently stored data, I come up with a test along the lines of:
[TestMethod]
public void DoCreateDeleteTest() {
PersistentDataStore pds = new PersistentDataStore();
bool createSuccess = pds.Save("id", "payload");
Assert.AreEqual(true, createSuccess);
bool deleteSuccess = pds.Delete("id");
Assert.AreEqual(true, deleteSuccess);
}
As long as everything works, this seems fine. The function has no prior dependencies and it cleans up after itself. The problem is: when the .Save() method performs the save but returns false/failure. The assertion fires and the delete is not called so it doesn't clean up after itself.
After this, there is persisted data in the database with name "id" and all future saves fail.
The only way I can think to get around it is to do a precautionary delete before the save, but that seems like way to large a hack.
Put the delete in a method marked with the TestCleanup attribute (I assume you are using MSTest).
By the way your test is also testing two different things: whether the save works and it also tests the delete. Tests should only test one thing at a time.
Wrap both within the one transaction? Do a delete in a catch?