I am using temporary database for my unit tests that are created and deleted with each test but have been very careful to ensure that it shares the same setup as my 'real' database (enlist=false;multipleactiveresultsets=true;etc) but have been constantly running into MSDTC escalation issues wherever TransactionScope is being used.
After switching TransactionScope for Transaction I have solved the MSDTC escalation issue and run straight into a new problem - the transaction works perfectly as I use the application in the web browser, but it fails in unit testing with the following error:
SqlConnection does not support parallel transactions.
Without posting tons of code, has anyone else run into similar problems where the test environment and dev environment appear to work differently? Is there any general reason why a parallel transaction would work in dev and not in unit test mode?
If I cannot simulate exactly what is happening with my tests then what is the point in testing at all?
One big difference that I failed to account for was where I may be using RenderAction in a view, and therefore calling more controller actions than the test would, which may explain MSDTC/Transaction issue.
Related
We updated our solution from SpecFlow1.9 to 2.0 and NUnit2.6.4 to 3.2.1. After adapting some attributes and project settings, all tests run fine in NUnit. However, when the SpecFlow tests are executed with NCrunch, we get a SpecFlowException:
TechTalk.SpecFlow.SpecFlowException : The ScenarioContext.Current static accessor cannot
be used in multi-threaded execution. Try injecting the scenario context to the binding
class. See http://go.specflow.org/doc-multithreaded for details.
at TechTalk.SpecFlow.ScenarioContext.get_Current()
We intentionally designed our SpecFlow tests for a single-threaded environment (to keep the effort low) and we just want to continue executing these tests in one thread. So instead of injecting the scenario context as the proposed solution (we use NInject instead of the SpecFlow mini-IoC) we're looking for some setting to convince SpecFlow that it is running in a single-threaded environment.
Here are the NCrunch 2.23.0.2 settings:
I entered in the Assembly.cs files of all SpecFlow tests the following attribute:
[assembly: Parallelizable(ParallelScope.None)]
Without success; the exception keeps showing up.
Does anybody have a clue how to force SpecFlow2.0 in NCrunch2.23.0.2 with NUnit3.2.1 so that it thinks it's executing in a single-threaded environment?
Thank you for your effort!
2016-5-31: update
I installed the new version 2.1 of SpecFlow (available since 2016-5-25) but it didn't solve the problem.
I created an example project with a minimum amount of code to generate the problem. The calculator implementation is statefull and cannot be tested in a multithreaded environment.
SpecFlow throws the exception due to the (dummy) static reference ‘ScenarioContext.Current’ in CustomContext. Yes I know you should inject it if you intend to run in a multithreaded test environment. The problem is that SpecFlow THINKS it is in a multithreaded environment, but it isn’t and it shouldn't.
On investigation, this appears to be a 3-way compatibility problem between NCrunch, SpecFlow, and NUnit3.
As part of its behaviour, NCrunch will re-use test processes by calling into them multiple times (i.e. once for each batch of tests in the Processing Queue). Because NUnit3 kicks off a new thread for each test session, it ends up using a different thread for each call into SpecFlow.
SpecFlow identifies multi-threaded execution by tracking thread IDs, and since each session has a new thread, it incorrectly thinks the code is being run in parallel when actually it's just different threads being used synchronously.
Setting the 'Test process memory limit' global NCrunch configuration setting to '1' will allow you to work around the problem, as this will cause NCrunch to throw away a test process after each batch, rather than re-using it. Unfortunately, this will have a significant impact on performance.
I've reported this problem to SpecFlow. Because of it's nature, the most sensible thing would be for it to be fixed in SpecFlow itself - https://github.com/techtalk/SpecFlow/issues/638
You need to regenerate the code-behind- files of the Feature files after upgrade.
See the upgrade steps here: http://gasparnagy.com/2016/01/specflow-tips-how-to-upgrade-your-project-to-specflow-v2/
The company I work for uses Visual Studio to develop its website and all of its features, and there is also a separate site that's been developed for testing the site. This 'testing' site can run individual test cases against the website, and must be run for each possible case.
Everything is written in VB.NET and each time the program is run a single thread is created to run the test. However, at the 'end' of the test the thread seems to still lingers. The stop button in Visual Studio must be manually clicked in order to terminate the application. Also, a process icon lingers in the task bar long after the application has closed.
It appears to me that the program is not correctly terminating all threads run during the tests, but I'm not sure if this is an issue worth brining up in the office, so I ask the following question...
What is the purpose of properly closing an application and all threads running on it, and what are the consequences, if any, of not doing so?
Well it's probably a small problem now, but it's not a good practice, IMHO. Imagine what would happen if the same code was now being executed by a continuous integration server, for instance, TeamCity (or Jenkins, or...), and the unit tests are being run continuously and automatically, by said build server.
What would happen to the build status when those threads fail to close down cleanly? We often face this problem due to bad design decisions in threading, or due to simple (and possibly, idiotic) mistakes in our unit testing code. The net effect though, is a hung build process.
I've seen CI servers hang for almost half a day before someone (mercifully) killed the build process. Essentially, this indicates a problem in our code that may or may not become a huge issue. If this was server-side code, there is potential for this code to lead to a pretty bad situation. My advice would be to dig out your introspection toolkits (memory profiling, perf profiling, etc) and see what exactly is going on, and resolve it.
We had a similar problem with an application that is being called to index SPA pages on our application server. It was throwing an exception in some cases and threads were not closing. The biggest downside is that it will consume the servers memory which is bad
Another downside as it runs as a web application that it will consume available ports and stop running when it run out of available ports.
The code should be modified to peacefully kill the thread after finishing or on exceptions and of course report any.
I am trying to connect to a Domino Session which works fine when testing in isolation, however as soon as I put it in a thread I run into problems.
This works fine when I put in a Unit test, so I know I can connect to the server fine.
NotesSession session = new NotesSession();
string DominoPassword = Helpers.GetConfigSetting("DominoPassword");
session.Initialize(DominoPassword);
However when I put this in a thread (want to be able to connect to Domino several times) I
get the following error
"{Unable to evaluate expression because the code is optimized or a native frame is on top of the call stack.}"
A couple of years ago I definitely got farther than this with testing of multi-threaded Domino Interop in C#. I don't recall seeing that error. It might be due to a difference between our respective dev/test environments. It's long enough ago, though, that I don't recall what my environments were when I ran my tests.
But whether or not the Domino objects are thread-safe remains an open question. See here. As far as I know, IBM has never said that they are. (Also, last time I checked, IBM said that the classes are not supported on Win64 platforms even in Win32 applications, so that might be entering into your problem.
We have a unit tests project which is still using the "old-style" private accessors in many tests.
Since they're a maintenance nightmare, we're trying to get rid of them, and move to new new Microsoft Fakes framework, using Shims where needed.
Recently we wrote some new unit tests which use Shims, and noticed that for some reason this caused a few OTHER, old, tests, which were not modified, to run considerably slower. By slower I mean run times of about ~10sec instead of ~900millisec for the affected tests.
Running the affected tests on their own didn't seems to have this effect though - it only occurs when running them after tests with Shims.
Initially we thought this might be simply due to initialization problems, causing tests to influence one another.
However, after some experimentation, we found that the slowdown occurs even without actually adding any new test code. Simply adding the following snippet before one of the slowed-down tests caused the same effect of the test running slower:
using (ShimsContext.Create()) {}
Debugging seemed to show that the the code being tested was indeed running much slower (not the unit test code itself) , but we couldn't identify which part of it. We're not able to identify why these tests are affected while others are not.
At this point we tried profiling these tests (using the new "profile test" option in VisualStudio). However, it turns out that profiling tests with Shims is not possible for some reason. The following exception was thrown:
Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationException: UnitTestIsolation instrumentation failed to initialize. Please restart Visual Studio and rerun this test
As a last resort we also tried moving all tests using Shims to a separate test project in the same solution. This did seem to help, and all test run times returned to normal. We used test playlists to run each project's tests before the other's, and in both cases run times were OK. It's not really a solution though, and feels more like circumventing the actual issue.
So, we're not sure how to proceed. Any thoughts and ideas would be helpful.
Thanks.
The Microsoft documentation, Better Unit Testing with Microsoft Fakes (RTM).pdf, states that you will see a performance decrease when using Shims.
This article also goes over the performance impact of Shims:
http://technet.microsoft.com/en-us/windows/jj863250.aspx
"Other" tests should be explicitly executed in shimless context (ShimsContext.ExecuteWithoutShims), because it looks like even disposed ShimsContext in other tests may have detours to logic that doesn't use shims.
Recently I've been tasked with writing and running unit tests in C# for a desktop application. In the past, I've written mainly automated functional/GUI tests. These get run on dedicated QA machines because these tests can cause crashes, data loss, Blue Screen of Death, etc. Should a similar setup be done for running unit tests? Are there any risks with running unit tests on my local machine?
Unit tests should run on your local machine AND on the build server. They are an invaluable resource as feedback for the developer. The DEV should write the unit tests. Before checking in, he should run the unit tests to make sure he does not break anything. The build server will then run the unit tests again, in order to make sure nothing it actually broken, and that the integration with other code, if any, succeeded.
When the unit tests have run, preferably, the build server should then run integration tests and automated UI tests. When these processes have finished, a build has been produced with no known defects - at least none that is covered by the tests. In other words a green build means the software integration went well, and the developers can keep checking in - or at some point start doing manual testing before actually releasing a piece of software.
Software testing nirvana happens when enough tests has been automated, so that the developers and the business can feel safe releasing a product based only on the success of this stream of automated tests. In reality, only a few products will be this mature - and some amount of manual testing is required.
Unit tests does not touch external systems or interfaces. Therefore, there are no risks by running them locally.
Unit tests should be so isolated that they can be run on every machine without any further setup. Unit tests by definition test only functional units on a very granular level without touching external systems (i.e. database, file system, etc.)
Unit Testing is typically done by the Developers while developing. So Run it on your Development environment. Probably your Local machine.
well if your local machine is a dev machine I see no risk since the machine is not a production end user machine. You can also consider TFS Lab management (in case you have Microsoft TFS as ALM / source control system). in that case you can create virtual or physical test environment and the application gets deployed and tested in those defined environments.
not sure what the names of similar test products are for non TFS scenarios.