Currently I am struggeling with some unit tests that run fine from within visual studio but fail in Teamcity
I tracked down the problem to mstests.exe
Let's say I do the following steps:
Create a new Test project
Add a new Test class with the following test
[TestMethod]
public void TestCanCreateSqLiteConnection()
{
// Create the DbProviderFactory
var factory = DbProviderFactories.GetFactory("System.Data.SQLite");
// Create the DbConnection.
var connection = factory.CreateConnection();
// Assign connection string
connection.ConnectionString = "Data Source=database.sqlite";
// check the result
Assert.IsTrue(connection.GetType().Name.Equals("SQLiteConnection"));
}
Add an app.config file and add this:
<system.data>
<DbProviderFactories>
<remove invariant="System.Data.SQLite" />
<add name="SQLite Data Provider"
invariant="System.Data.SQLite"
description=".Net Framework Data Provider for SQLite"
type="System.Data.SQLite.SQLiteFactory, System.Data.SQLite" />
</DbProviderFactories>
</system.data>
Install "System.Data.SQLite (x86/x64)" via nuget
Run test from Visual Studio (2010). It should run fine:
Now I want to run the same test via mstest.exe so I:
Open a Visual Studio 2010 command prompt
Navigate to the bin\debug folder
Execute
mstest.exe /testcontainer:TestProject1.dll /detail:errormessage
The test eventually failes with
System.DllNotFoundException: Unable to load DLL 'SQLite.Interop.DLL':
The specified module could not be found. (Exception from HRESULT:0x8007007E)
Now if I extend the call to mstest.exe with testsettings the test runs fine.
mstest.exe /testcontainer:TestProject1.dll /detail:errormessage
testsettings:..\..\..\Local.testsettings
The Local.testsettings contains nothing special, even if I create a new testsettings file and use this, the test passes.
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings id="fc837936-41d1-4987-8526-34f9336569f5" name="TestSettings1" enableDefaultDataCollectors="false" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Description>default test run</Description>
<Deployment enabled="false"/>
</TestSettings>
So the main question is, why this has an impact on my test run, and how I can run my tests form commandline without specifying a *.testsettings file.
You need to use a DeploymentItem to ensure the file is copied to the deployment directory when testing via the command line. I created a base class for all of my test classes that depend on the SQLite database.
[TestClass]
[DeploymentItem("Resources\\empty-db.sqlite", "Resources")]
[DeploymentItem("x64\\SQLite.Interop.dll", "x64")]
[DeploymentItem("x86\\SQLite.Interop.dll", "x86")]
public class SQLiteTest
{
[TestInitialize()]
public void ClearDatabase()
{
File.Copy("Resources\\empty-db.sqlite", "test-db.sqlite", true);
}
}
I encountered a similar error message a while back. If I remember correctly, the following was the crux of the issue: (it may not be 100% relevant for the OP, but it might be useful to somebody who hits this down the line).
The problem was that my unit tests all failed in Release mode with an exception complaining about the availability (or lack thereof) of the SQLite.Interop.dll. I realised that when built in Debug mode, the bin\Debug folder had 2 sub folders (x64 and x86) each with a copy of SQLite.Interop.dll but, in Release mode these files/folders did not exist.
To resolve it, I created the x64 and x86 folders in my project and added the appropriate version of SQLite.Interop.dll too them, setting the Copy to ouput setting to Copy if newer. (I had originally used 'Copy always' but it seems the MS Test engine doesn't shut down when the test run is complete - which can lock the file. Since the dll shouldn't change regularly the Copy if newer option was a suitable approach).
This enabled my unit tests to pass in release mode - but unfortunately (as in the OP's case) they did not work when run from the command line. Still trying to figure that one out - I think it's because MSTest is 32-bit and SQLite uses native code which is (maybe) 64-bit but the finer detail necessary to resolve this eludes me at present.
Two year later and it is still a pain to get SQLite working in unit tests.
Yesterday I included the current SQLite nuget package into a unit test project with <Deployment enabled="true"/> enabled and could not access sqlite by the dbproviderfactories method.
I included the SQLite interop directories with
<Deployment>
<DeploymentItem filename="packages\System.Data.SQLite.Core.1.0.98.1\build\net40\" />
</Deployment>
But that was not enough. Accessing the provider with
DbProviderFactories.GetFactory("System.Data.SQLite");
still threw an error Failed to find or load the registered .Net Data Provider error unless I made this call var factory = new System.Data.SQLite.SQLiteFactory(); After that I could acces SQLite witht the DbProviderFactories.
I included this in the ClassInitialize method so is is only executed once.
I've encountered a similar issue where the tests run through fine in Visual Studio 2013 but if run directly by MSTest many would fail. No SQL Lite used!
In the end I just added a default .testsettings file to the MSTest call and now the results are consistant.
Related
We use SQLite in-memory databases in our dotnet codebase for integration testing our repository layer with ORMLite. When I try to run a test on Mac I get this error:
System.DllNotFoundException
Unable to load shared library 'SQLite.Interop.dll' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libSQLite.Interop.dll, 0x0001): tried: 'libSQLite.Interop.dll' (no such file), '/usr/local/lib/libSQLite.Interop.dll' (no such file), '/usr/lib/libSQLite.Interop.dll' (no such file), '/Users/aaron/Projects/joinder/DASH/Test/ClientSiteData.Tests/bin/Debug/net6.0/libSQLite.Interop.dll' (no such file)
at System.Data.SQLite.UnsafeNativeMethods.sqlite3_config_none(SQLiteConfigOpsEnum op)
Is there a way to get this working on Mac so I don't have to revert to my Windows machine every time I need to run tests?
Can you try installing one of the other OrmLite.SQLite packages, e.g. to use the Microsoft.Data.Sqlite ADO.NET Provider use:
<PackageReference Include="ServiceStack.OrmLite.Sqlite.Data" Version="6.*" />
I easily configured to get the coverage result for .NET Core Projects in Azure DevOps, but no luck with .NET Framework Projects. So, I would be so grateful to get suggestion on this because coverlet documentation is clearly saying that we can also use it for .NET Framework Projects.
This question is kinda similar to mine but I didn't see any answer there,
Can you use Coverlet to get code coverage data in a .NET Framework project?
Finally, I found a simpler solution. Here it is,
Add <IsTestProject>true</IsTestProject> in test project file.
Run the commend dotnet test/dotnet test /p:CollectCoverage=true being at test project location(where
TestProject.csproj exist)
You might get following error after running the command,
The imported project "C:\Program Files\dotnet\sdk\3.1.100\Microsoft\VisualStudio\v16.0\WebApplications\Microsoft.WebApplication.targets" was not found.
In the project file(which you want to unit test and get code coverage), change following
Import statement
This, <Import Project="$(VSToolsPath)\WebApplications\Microsoft.WebApplication.targets" Condition="'$(VSToolsPath)' != ''" />
To, <Import Project="$(VSToolsPath)\TeamTest\Microsoft.TestTools.targets" Condition="Exists('$(VSToolsPath)\TeamTest\Microsoft.TestTools.targets')" />
This worked for me both locally and with Azure DevOps.
Note: Don't forget to install Coverlet.msbuild in your test project.
Update:
Above approach works only if you don't get ".Microsoft.WebApplication.targets was not found" error. Commenting suggested import statement will make publish fail at the end, which is obvious. So, I ended up using Coverlet.Console and it's working smoothly without any error. But, to use coverlet.console I needed TestProject.dll file instead of project file(.csproj); so I had to add extra build task for test project. Here is the documentation how to install and use Coverlet.console
Hope this will be helpful for the ones who end up landing here.
Yes, you can can code coverage data from a ASP.NET project. And it's simple as #riQQ suggested in this thread.
The following content is for supplements and reproduce #riQQ's answer in above thread.
Prepared a webform app and .net framework unit test, added the coverlet.runsettings file in repo, the content refering Coverlet options supported by VSTest integration:
<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="XPlat code coverage">
<Configuration>
<Format>json,cobertura</Format>
<Exclude>[coverlet.*.tests?]*,[*]Coverlet.Core*</Exclude> <!-- [Assembly-Filter]Type-Filter -->
<Include>[coverlet.*]*,[*]Coverlet.Core*</Include> <!-- [Assembly-Filter]Type-Filter -->
<ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
<ExcludeByFile>../dir1/class1.cs,../dir2/*.cs,../dir3/**/*.cs,</ExcludeByFile> <!-- Absolute or relative file paths -->
<IncludeDirectory>../dir1/,../dir2/,</IncludeDirectory>
<SingleHit>false</SingleHit>
<UseSourceLink>true</UseSourceLink>
<IncludeTestAssembly>true</IncludeTestAssembly>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
Configure the VSTest task:
It can generate the coverage file successfully:
Passed TestMethod1
Results File: D:\a\_temp\TestResults\VssAdministrator_fv-az38_2020-03-17_07_53_28.trx
Attachments:
D:\a\_temp\TestResults\*******-****-****-****-**********\VssAdministrator_fv-az38 2020-03-17 07_53_18.coverage
Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Note: If you want to run this build on self hosted agent, you might need to make sure the VS Enterprise is installed, refer to "Cannot find CodeCoverage.exe" on self-hosted agent.
So I have this really simple test:
[TestMethod]
public void CheckAllParametersExist()
{
try
{
new ParametersService().Get();
Assert.IsTrue(true);
}
catch(Exception ex)
{
Assert.Fail(ex.Message);
}
}
ParametersService().Get() runs through a parameters class finding all the properties and tries to populate them with values from a database. Occasionally when we release the website we might forget to publish some values, so I wanted a unit test in Bamboo to identify any parameters we may have missed.
To the crux of my problem: In Visual Studio 2017 the unit test passes fine. With MSTest is fails with:
Assert.Fail failed. The type initializer for 'System.Data.Entity.Internal.AppConfig' threw an exception.
Looking this particular error up it looks like I've got the wrong or mixed versions of EntityFramework. Having tried to fix this myself I have removed Entity framework from the Tests project nuget and app.config, then under the solution I consolidated the EntityFramework nuget and included Tests project again, still the same error. App.config shows version 6.0.0.0, nuget has installed 6.2.0
I am stuck, if anyone can suggest any solutions or identify any reasons why I might be seeing this problem I would be greatful.
Fyi: I am running MSTest with /testcontainer:tests.dll in the Tests project bin output debug folder.
So it turns out the problem was with the App.Config. The problem is that we have some sections of the configuration loaded from external files. In this case connection strings. In VS it loads the config file for connections strings just fine. In MSTest it copies the DLLs to another folder but doesn't include the folders/config files.
Also MSTest has now been retired, as of VS2013 we should be using VsTest.console, however Bamboo hasn't caught up with that yet.
I am facing a problem with my UnitTest. I want to Test my Data-Access which is done using a repository based on NPoco. I have therefore written a couple of tests and the test project retrieves NUnit, NPoco, System.Data.SQLite, and some other Stuff via NuGet.
This is the app.config of the TestProject:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<connectionStrings>
<add name="RepositoryTests.Properties.Settings.ConnectionString" connectionString="Data Source=db.sqlite;Version=3" />
</connectionStrings>
<system.data>
<DbProviderFactories>
<remove invariant="System.Data.SQLite"/>
<add name="SQLite Data Provider" invariant="System.Data.SQLite" description=".NET Framework Data Provider for SQLite" type="System.Data.SQLite.SQLiteFactory, System.Data.SQLite" />
</DbProviderFactories>
</system.data>
</configuration>
In VS, the Project builds fine. Triggering the tests in Visual Studio works as well.
Building the Projects with MSBUILD works as well. Running the tests via nunit after building them with msbuild.exe raises a Exception though:
Unable to find the requested .Net Framework Data Provider. It may not be installed.
This is only the case, when executing the tests using nunit directly (sth. like nunit-console.exe myproject.csproj /config:Release). Triggering them in VS is no problem.
Does anybody know how to solve this problem?
The problem is caused by the nunit test runner having its own app.config in which your settings are not present.
The simplest way to solve the problem is to move the configuration out of your app.config and into the code itself. Whatever you have in app.config can be done inside code.
Another solution is to move the configuration into a seperate file and then explicitly load that file's configuration using code. Just make sure that the file is copied to the output folder on build.
You could use something like:
public static class TestFactory
{
public static DatabaseFactory DbFactory { get; set; }
public static void Setup()
{
var fluentConfig = FluentMappingConfiguration.Configure(new OurMappings());
//or individual mappings
//var fluentConfig = FluentMappingConfiguration.Configure(new UserMapping(), ....);
DbFactory = DatabaseFactory.Config(x =>
{
// Load the connection string here or just use a constant in code...
x.UsingDatabase(() => new Database("connString");
x.WithFluentConfig(fluentConfig);
x.WithMapper(new Mapper());
});
}
}
See here for more details.
Then in the test fixture:
[TestFixture]
public class DbTestFixture
{
[TestFixtureSetUp]
public void Init()
{
TestFactory.Setup();
}
[Test]
public void YourTestHere()
{
var database = TestFactory.DbFactory.GetDatabase();
...
}
}
I got it working by creating an copy of the app.config-file and give that copy the name of the Test-Project, followed by .config. So if we assume, that the project is named Test.Project the copy must be named Test.Project.config. Nunit doesn't seem to load the automatically generated Test.Project.dll.config. This info can be found somewhat disguised in the NUnit-docs (Configuration Files -> Test Configuration File -> 3rd paragraph, last sentence).
In VS, in the Properties-Section of that copied file, i set the copy-to-output-directory-property to always.
Executing the tests with nunit-console.exe then lead to another exception (Bad-Image), which was caused by NUnit not finding the SQLite.Interop.dll-file. This could be solved kind of hacky by adding that file, which already resides in the x64 or x86 folder, as an existing element to the solution in VS and also setting the copy-to-output-dir-property to always.
We're using .NET 4.5 and have had no problem with open generic exports in MEF up until a month ago when it suddenly stopped working. 70 tests on our CI server turned red and we traced it down to missing parts in the container.
I found it strange, so I wrote this test code:
var catalog = new TypeCatalog(typeof(Impersonator<>), typeof(Cache<>));
var parts = catalog.Parts;
But it looks like none of the open generic types will register in the catalog. Parts is a TypeCatalog with Types = <Empty>.
Usually we go about this by declarative exports using attributes as in this SO question, but none of the alternatives seems to be working anymore.
Any idea will be most appreciated.
Any idea what triggered it to start failing? Did you get a .NET update in that timeframe?
Out of curiosity if you write a simple console app targeting .NET 4.5 does this issue repro?
The reason I ask is I believe there was an update released for .NET 4.5 to make sure open generics didn't break existing .NET 4.0 apps and so if for some reason your tests are running in .NET 4.0 mode then then the open generic support will be disabled which could cause what you are seeing.
I resolved this in my web app by putting the entry < httpRuntime targetFramework="4.5" /> in my web.config. For non-web apps, make sure the app.config has the entry
< supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />.
see http://blogs.msdn.com/b/webdev/archive/2012/11/19/all-about-httpruntime-targetframework.aspx for more info.
I know this is an old issue, but I have some additional information that may help others.
I ran into the same issue. My unit tests were working then suddenly stopped. We narrowed it down to a pair of Windows updates: KB2840642v2 and KB2840642. After uninstalling these, MEF started working properly again in unit tests. I didn't see anything in the release notes for those updates that should affect things, but you know how that goes.
However, I am now running Windows 8 unfortunately, and the issue has popped up again. I do not have those updates installed. I think they were incorporated natively into the operating system.
When you say “on our CI server”, that does not tell us much about how you’re running the tests. However, I seem to get an experience like yours using vstest.console.exe.
I found a situation where MEF’s automatic open type closing feature, new in .net-4.5, will appear to break when running tests with vstest.console.exe. Basically, vstest.console.exe might decide to run in .net-4.0 compatibility mode as if /Framework:Framework40 had been passed to it. .net-4.5 assemblies will load and run perfectly fine even when loaded by a .net-4.0 runtime, but MEF detects that you’re running in .net-4.0 mode and disables its support for automatically closing open generic types. You can get MEF to behave properly again by forcing vstest.console.exe to run in .net-4.5 mode by passing the /Framework:Framework45 switch.
(This tested on a machine with .net-4.6.1 installed, though, not sure if that changes things yet more).
Minimal Repro
When I make a simple test assembly targeting the .net-4.5 framework, vstest.console.exe correctly guesses that it should run the tests with .net-4.5. However, when I make an assembly that integrates into a more complex build environment, vstest.console.exe suddenly starts running in .net-4.0 mode rather than .net-4.5 mode. So I started with my complex build environment and started paring things down until the issue disappeared.
To get vstest.console.exe to guess the wrong framework, you need two assemblies. In one, define a custom assembly Attribute. In another, apply that custom assembly attribute and define your unit tests.
First assembly:
using System;
[AttributeUsage(AttributeTargets.Assembly)]
public class BreakVSTestFrameworkDetectionAttribute : Attribute
{
}
Second assembly referencing the prior one:
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.ComponentModel.Composition;
using System.ComponentModel.Composition.Hosting;
[assembly: BreakVSTestFrameworkDetection]
[InheritedExport]
public interface IGenericService<T>
{
void Print(T thing);
}
public class SomeGenericService<T> : IGenericService<T>
{
public void Print(T thing) => Console.WriteLine($"{typeof(T)}:{thing}");
}
[TestClass]
public class UnitTest1
{
[TestMethod]
public void Run()
{
using (var catalogue = new ApplicationCatalog())
using (var container = new CompositionContainer(catalogue))
{
container.GetExportedValue<IGenericService<string>>().Print("asdf"); // System.String:asdf
container.GetExportedValue<IGenericService<int>>().Print(123); // System.Int32:123
}
}
static void Main(string[] args) => new UnitTest1().Run();
}
With these two assemblies and the defined test, I get the following outcomes in different scenarios:
Running the tests using Visual Studio 2015 Community Editions’s GUI works perfectly fine.
Running the code directly as an executable works fine.
C:\Users\ohnob\OneDrive\Documents\Visual Studio 2015\Projects\MefGenericsUnitTests>bin\Debug\MefGenericsUnitTests.exe
System.String:asdf
System.Int32:123
Running the tests with something like mstest (except I can’t tell why one would or wouldn’t use this instead of vstest) magically works (at least in this case) (output trimmed).
C:\Users\ohnob\OneDrive\Documents\Visual Studio 2015\Projects\MefGenericsUnitTests>"\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\MSTest.exe" /testcontainer:bin\Debug\MefGenericsUnitTests.exe
Microsoft (R) Test Execution Command Line Tool Version 14.0.23107.0
Copyright (c) Microsoft Corporation. All rights reserved.
Loading bin\Debug\MefGenericsUnitTests.exe...
Starting execution...
Results Top Level Tests
------- ---------------
Passed UnitTest1.Run
1/1 test(s) Passed
Running with straight up vstest.console.exe fails (output trimmed):
C:\Users\ohnob\OneDrive\Documents\Visual Studio 2015\Projects\MefGenericsUnitTests>"\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" bin\Debug\MefGenericsUnitTests.exe
Microsoft (R) Test Execution Command Line Tool Version 14.0.24720.0
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
Failed Run
Error Message:
Test method UnitTest1.Run threw exception:
System.ComponentModel.Composition.ImportCardinalityMismatchException: No exports were found that match the constraint:
ContractName IGenericService(System.String)
RequiredTypeIdentity IGenericService(System.String)
But vstest.console.exe /Framework:Framework45 succeeds:
C:\Users\ohnob\OneDrive\Documents\Visual Studio 2015\Projects\MefGenericsUnitTests>"\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" /Framework:Framework45 bin\Debug\MefGenericsUnitTests.exe
Microsoft (R) Test Execution Command Line Tool Version 14.0.24720.0
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
Passed Run
Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 3.7855 Seconds
I think you can not use open generic and should provide a concrete running class like:
var catalog = new TypeCatalog(typeof(Impersonator<type1>), typeof(Impersonator<type2>), typeof(Cache<type3>), typeof(Cache<type4>));
var parts = catalog.Parts;
I tried using interfaces like IImpersonator and ICache and does not work also:
interface IImpersonator { }
class Impersonator<T> : IImpersonator
...
var catalog = new TypeCatalog(typeof(IImpersonator), typeof(ICache));
var parts = catalog.Parts;