I have the NUnit GUI runner to run tests on my unit test assembly every time that it is built. The problem is that when I try to build, the .dll in the Debug folder is in use by NUnit, which prevents it from being built, which prevents the automatic test run. Is there a way around this?
Can I suggest a slightly different approach but along the same lines as #UvarajGopu....
As long as you are using Visual Studio higher than "Express" version, and if you write your UnitTests in a separate project (typically alongside the project being tested, with a ".UnitTests" suffix), then do this:
Set your UnitTests project as being the StartUp project (right-click in Solution Explorer, "Set as Startup Project")
In the Project Properties for that project, in the "Debug" tab, choose "Start External Program", and choose the NUnit GUI executable. Put the name of UnitTests assembly in "Command line arguments".
Now you can simply press F5 (to start debugging) which will build your projects, and launch the NUnit GUI for you. This has the added advantage that if your tests fail, you can add breakpoints, and step through using the debugger (without needing to attach the debugger manually).
Whenever you are building your solution. I suggest you to close the Nunit, which is loaded with your .dll. After unloading try to build, then your build will succeed.
As Nunit is using you .dll. It won't allow your build to be success.
I had a similar problem a while ago and my final solution was to resign from using NUnit GUI runner and just run them directly from code.
In my project I referenced to NUnit libraries needed by those freshly built dlls plus nunit.core and nunit.utils. The code itself is very simple:
TestResult ExecuteTests(string testAssemblyPath) {
CoreExtensions.Host.InitializeService();
TestPackage testPackage = new TestPackage(testAssemblyPath);
testPackage.BasePath = Path.GetDirectoryName(testAssemblyPath);
RemoteTestRunner testRunner = new RemoteTestRunner();
testRunner.Load(testPackage);
TestResult testResult = testRunner.Run(new NullListener(), TestFilter.Empty, true, LoggingThreshold.Warn);
testRunner.Unload();
CoreExtensions.Host.UnloadService();
return testResult;
}
The TestResult object is very powerful. Among other things, it contains all the results, subresults, tests themselves etc. In order to analyze them, you can either create a simple parser or use one of the possibilities given by NUnit libraries. My favorite is XmlResultWriter but there are others available as well. All of them can be found in nunit.util.dll.
Unfortunately, that would still block the loaded dll from being rebuilt. I avoided this problem by running this in a separate AppDomain and unloading this domain after the tests are finished running. Then dll is nicely freed and you can do with it whatever you want.
Related
I am running Visual Studio 2017 & the Test Runner cannot discover the Unit Tests unless Visual Studio is running as an Administrator. Running as Administrator is next to impossible at my new workplace.
As such...
Q: How can get the Test Explorer to discover Unit Tests without being an administrator?
UPDATE:
I have confirmed the following...
POSITIVE: The Unit Test Project is set to build
POSITIVE: The Test Project is a genuine Unit Test Project (w/ magic guids)
NEGATIVE: The Test View is not an available option in VS2017 Community Edition
A summary of various fixes for this no tests found issue.
You have 2 versions of the Microsoft.TestAdapter installed (reference Microsoft.VisualStudio.TestPlatform.TestFramework and Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions). Check NuGet Package Manager, version 1.1.11 doesn't work, you need to use 1.1.18 (at time of writing for it to work).
Also check your Bin folder &/or your Build Agents references folder for duplicate versions.
Make sure that your test project is set to build. If the test assembly isn't being built, VS won't be able to find any tests:
Magic GUIDs are needed in the Test Project:
C#:
{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}
VB:
{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{F184B08F-C81C-45F6-A57F-5ABD9991F28F}
Trying to get ignored tests running again. Same error message occurs if you remove the Ignore label. Does not automatically re-enable the test. This article takes you through the last step. http://richallen.blogspot.com/2008/05/ms-test-re-enabling-ignored-tests.html
Ie
Open the Test List Editor, selected the ignored test, right click and select "Enable" from the context menu.
Make sure your method belongs to a class with the [TestClass] Attribute and the method it's marked with the [TestMethod] Attribute.
Use Test View to find your test.
Open your Properties window (F4), and make sure your test is enabled
Some people have accidentially added a Class Library project instead of Test Project. Running Tests in a Class Library doesn't work, it has to be a Test Project, probably because of the Magic GUIDs.
Check whether the solution is running on 64bit. If so change it to x86.
Marking tests as static makes them fail to appear in the test list.
Close down Visual Studio and delete the .vsmdi (visual studio test meta data) file. This will be regenerated.
I'm working with Unit Test using VS2013 Professional. In particular I'm using the NUnit framework (NUnit TestAdapter for VS2013). My problem is that when I run my tests then VS starts building all the projects inside the solution. Currently the Unit Test project does not reference any solution project.
If I simply code a single test method like:
[Test]
public void SimpleTestMethod(){
Assert.That("a", Is.EqualTo("a"));
}
and the Unit Test project is in a Solution with N project, when I run my test then VS will build all N-1 project... In my case this behavior is boring because it takes too much time (the solution contains many projects) and some projects contain errors.
Is there a way to run my SimpleTestMethod() without complete solution building?
Break your test project to multiple projects that only reference a subset of the solution's projects.
This is also good test housekeeping - have a separate unit test project for each solution project instead of one huge project with dependencies to anything else. There are several advantages to this:
Tests run faster
It's a lot easier to isolate test cases, especially configuration settings
You can version projects and their test cases together
A good naming practice is to name your test projects the same as their target projects with a .Tests suffix. You can also create a solution folder (not a real folder) called "Tests" and move the test projects in it.
As for the why: Test runners use the Unit Test assembly and its dependencies to run their tests. If any of the assembly's dependencies change, the assembly and the dependencies have to be rebuilt. Visual Studio doesn't know what the external tool will call so it has to build all changed assemblies and their dependents.
If the build fails, there are no valid assemblies for the test runner to use so VS has to rebuild the entire solution before the runner can work. In this case, the obvious solution is to fix the error.
There are some stopgap options you can use until you can fix the error:
Temporarily remove the broken project from the build configuration
Split the solution so that you have a solution that can be built and tested
I struggled with this for a very long time as well. I actually hated the automatic build process, even when everything was successful.
I started to run tests through the command line. No build process is necessary. You can write your own .bat files and keep logs of test results. There are plenty of command line parameters that can be added to customize for what you are looking for.
https://msdn.microsoft.com/en-us/library/jj155796.aspx
Is it possible to let Resharper (or NUnit?) know that I want each test to look for an App.config under it's own project, even when running all tests in a solution together?
Background:
I'm using NUnit and the test-runner that ships with Resharper, and I've got several test-projects in the same solution. Some of my tests depend on config-files located under their respective projects.
When I run a test-project by itself, it will use it's internal App.config, and everything works fine. When I try to run all the tests in the solution, or use the shortcut to run all tests in current test session however, no config-file will be selected, and any test depending on a config will fail by default.
For this reason, I typically end up running all tests in the solution once first, then right clicking the nodes in the test-runner for each of the config-dependent projects, and running them separately afterwards.
Solved:
Apparently assemblies containing NUnit-tests can be run in separate processes or domains using command line options.
For the testrunner under Resharper, this setting can be found under Resharper > Options > Unit Testing.
There is an option "Use separate AppDomain for each assempbly with tests". Checking that solved my problem.
I am using Visual Studio 2012 for a solution with a C# and a C++/CLI .dll, with the C++/CLI dll referencing native .dlls such as boost. The C++ code is compiled as x64.
When I open VS, I can clean and build my project.
Using test explorer, I can run my tests.
As soon as I've used test explorer to run tests once, I cannot rebuild the project. It seems that VS2012 Test Explorer keeps a lock on my C++/CLI-dll, and there I get the following error:
LNK1104: cannot open file 'C:\Dev\LockExample\bin\Debug\cli.dll'
As a result of this, whenever I have run the tests using Test Explorer, I need to restart VS2012 before I can continue developing. Obviously this is not a sustainable development process.
Testing and rebuilding works without problem with C#-only dlls - as far as I can tell the problem only occurs with DLLs that use native x64 code.
After some more testing, I found that the villain here is vstest.executionengine.exe. Using handle (from SysInternals), I see that vstest.executionengine.exe holds locks for the .dll and the .pdb of the cli-dll. It does not hold any locks for managed-only dlls.
How can I get Visual Studio Test Explorer to release the locks on the C++/Cli dlls after the test runs completed?
In Visual Studio 2013 this problem can easily be fixed by unchecking the option "Keep Test Execution Engine Running" under "Test -> Test Settings" in the menu.
I found the answer in another post:
vstest.executionengine.x86.exe not closing
After some more searching, I found this post on connect.microsoft.com. The last hint in workarounds does solve the problem, although it's an ugly hack.
I can rebuild if I add the following as pre-build events to my C++/CLI dll:
taskkill /F /IM vstest.executionengine.exe /FI "MEMUSAGE gt 1"
taskkill /F /IM vstest.executionengine.x86.exe /FI "MEMUSAGE gt 1"
This will kill the vstest.executionengine.exe process, thereby freeing the lock on my .dll file.
I've also encountered this issue while testing involving native dlls. The workaround (solution?) I found was to add a DeploymentItemAttribute to the tests - not sure if this is generally true, but it certainly worked for me. It's a bit of a pain if there are lots of them (I had 6 in my case), but once done it was easy to copy and paste to the other tests.
So my unit test class looks something like this:
[TestClass]
public class TestMyClass
{
[TestMethod]
[DeploymentItem("firstnative.dll")]
[DeploymentItem("secondnative.dll")]
public void TestMyMethod()
{
//Code which (indirectly) uses the above native dlls.
}
}
I've also been fighting this problem and initially used the "taskkill" workaround, but I just stumbled over an option in the VS2013 settings which seems to more elegantly solve this problem:
Remove the check-mark on the
Keep test execution engine running between test runs
option found at
Tools / Options / Web Performance Test Tools
I had written a C# utility to be able to load and unload native libraries in here
I used it in the test project as below. Since it unloads dll in Dispose, I don't get build errors.
public interface INativeCrypto : INativeImport
{
[ImportFunction("mynative.dll"]
int NativeMethod();
}
[TestClass]
public class UnitTest1
{
public void TestMethod1()
{
INativeCrypto impl = NativeImport.Create<INativeCrypto>("");
// Use methods in impl
int i = impl.NativeMethod();
//...
}
}
Adding some stuff to the answer of #frodesto, (in case of VS2013), "Test>Test Setting>Keep Test Executin Engine running" configuration is stored in the user configuration (SUO file). This can be a bit nasty in case of this error happens in the TFS Build agent, beacause it uses a service default user.
To fix this case, first kill the existing vstest.executionengine.exe, modify the user used by TFS Build agent to execute with the user you are logged on. Open the solution stored in TFS Build agent workspace and unselect the option.
After that, TFS Build agent will read the "keep test execution engine" option beacause the SUO file is for the same user.
In my program, I use SevenZipSharp to generate zip files. SevenZipSharp is a managed DLL which loads another DLL, 7z.dll. I am manually setting SevenZipSharp's path to 7z.dll using SevenZipCompressor.SetLibraryPath.
When I execute my program in Debug mode, this all works fine, and it generates the zip file as nice as you please. However, when I execute my unit tests with mstest, SevenZipSharp always gives me the following error:
Test method threw exception:
SevenZip.SevenZipLibraryException: Can not load 7-zip library or
internal COM error! Message: failed to load library..
I suspect that MSTest might be doing something that is preventing SevenZipSharp from being able to load 7z.dll, like running in a security-tight sandbox (or something. I'm new to C# and MSTest...)
Does anyone have an idea about what might be happening?
Thank you!
Though the question posits a questionable scenario, the general problem of MSTest not loading required DLLs seems to be a common one, deserving of a less dismissive answer.
By default MSTest will copy the assemblies it believes are required by the test container to the Out folder of the default results folder, which changes for each run.
MSTest does not always automatically infer the necessary assemblies correctly; if there is no explicit direct reference to an assembly it won't be copied. Also, native DLLs are typically not detected.
I am not aware of a direct option to set the MSTest search path. You can determine the search path using procmon.exe as suggested above (it is basically the standard Windows DLL search).
Unintuitively, the default search path does not include the launch directory and I think this is a cause of confusion. When the tests are running the current directory is the test results "Out" directory, not the MSTest launch directory.
However, it is possible to control MSTest search behaviour (and the copying behaviour) with a test settings file. You can easily create and edit these through Visual Studio (see the Test menu) and then specify the created settings file on the MSTest command line. You can use different settings files for Visual Studio and MSTest.
By this means you can control exactly what DLLs are copied to your test directory.
See Create Test Settings to Run Automated Tests from Visual Studio to get more information on this.
Of course, DLL load failures may be due to missing dependencies, and the DLL mentioned in the error message may itself be present. You can use the dependency viewer or procmon to pick up unexpected dependencies in DLLs.
Consider using Process Monitor (aka procmon.exe) from the excellent SysInternals tools to monitor your test harness (MSTest). It will show you where the executable is looking for 7z.dll.
Visual Studio 2017 (and possibly 2015) provides two new ways to indicate that a native dll or other file is required by your tests without the need for a test settings file:
1: Add a link to the dll to your test project and tell VS to copy it to the output directory. Right-click the project in Solution Explorer and choose Add > Existing Item. Browse to 7z.dll, click the down arrow next to the Add button, and choose Add as Link. Then select the new 7z.dll item in Solution Explorer, alt-Enter to bring up Properties, and set "Copy to Output Directory" to "Copy if newer" (or "Copy always").
2: Attach a DeploymentItemAttribute to your test class. This attribute's constructor takes a single string argument, which is the path to the file you want to make available to tests in the class. It is relative to the test project's output directory.
Are you sure you want your unit tests to involve external libraries? Ideally you should have mechanisms to replace external stuff with e.g. mock objects, because testing external libraries like that actually turns your test into an integration test.
For popular libraries such as SevenZipSharp you can assume that it is properly tested, and you can have manual integration tests to verify that it performs correctly in your program.
I would consider getting rid of that dependency through dependency injection, mocking frameworks, etc, and let your unit tests solely test your own code.
Investigate the factory method or abstract factory design patterns for tips on how to easily replace such dependencies.
A good start would be to create your own ICustomZipInterface, and use the wrapper pattern to encapsulate your zip logic for production code. In your unit tests, replace that wrapper class with a dummy implementation. The dummy implementation might for example record how you access your zip component and you could use that information to validate your code, rather than checking if a zip file is actually created.