In my project we had BDD tests which I have written using specflow, nUnit and Watin. I run these tests from visual studio using resharper. Now I want to expose these features and scenarios to non technical people and I want them to run these tests.
Something like I want to list all the tests in a browser and user should be able to run those tests by clicking on them. Can this be achieved ? Is there any addin ?
Currently we use Team Foundation Server as our build server.
TeamCity, a Continuous integration server by JetBrains provides this as a webbased functionality. It even provides statistics and test output results.
It supports nUnit out of the box.
SpecFlow and Watin are supported with some configuration.
The BIGGEST problem you are going to have is that the plain text feature file, automatically gets converted to a xxx.feature.cs file by the SpecFlow Visual Studio plugin. So your process is this,
Modify xxxx.feature file
Find some way to get the SpecFlow plugin to generate xxx.feature.cs
Compile
Run tests by using NUnit/Xunit (as configured)
Gather and present test success report
To me this is a process has a name, I'd called it development.
BDD however is a different process, it's all about collaboration and communication with the business in order to devise a specification. In the beginning there were no tools, but the process still worked.
A number of my co-workers have been using BDD techniques on a variety of real-world projects and have found the techniques very successful. The JBehave story runner – the part that verifies acceptance criteria – is under active development.
Dan North - Introducing BDD 2006
Don't get caught up on the tools alone or you'll miss the vital part of the process. You'll get so much benefit by working with your BA to define the new specification together collaboratively.
P.S. Another way to consider this is that the specification and code should always be in step. Just by defining a new example, we don't magically move the code forwards to meet that example. Instead the most common practice is to develop the code to meet the new example, and then check in the new specification and code as a single change set.
You can use the Pickles project to produce stakeholder-friendly documentation (including HTML) from the Gherkin specifications in your source control.
https://github.com/picklesdoc/pickles
There's no facility for running the tests from the HTML. It's open-source so perhaps you can extend it this way... however, I personally don't see the value in having non-technical users actually execute the specifications. I would have your continuous integration server run the SpecFlow tests and generate a step definition report periodically. The non-technical users can then browse to these reports to see current project status.
To give access to your feature files to non technical people you can use http://www.speclog.net/
Spec log will allow non tech to edit and create new features and will automatically synchronise them with TFS.
Unfortunately it's not free and you can't run the specs from that tool.
Related
I am working on an existing test automation framework that uses the following for UI tests on a windows application:-
C# for creating UI cases
winAppDriver to interact with UI objects
nUnit for validation
specflow for BDD
I don't have much idea about microsoft projects. Here is a simple structure of the project:-
Application
Source
Modules
features
Tests
Each test folder has a "app.config" file in it that supplies the config for the test to work like DB username, pwd, services urls etc.
We execute our test case from "Test Explorer" pane in VS Enterprise which is not the best way if the test cases need to be run remotely in jenkins. As I said I don't have much frame of reference when it comes to Microsoft apps. So, here are some questions to which I have been looking for answers; there seems to be no definite consensus online. Just curious to know how others are maintaining their projects. Here goes:-
is there a jenkins friendly way of running these test cases? using a cmd line or a runner file perhaps?
if I find a way to accomplish #1 how do I inject app.config properties at runtime?
how does one execute these cases on remote machine? Mine is a desktop windows app. What would a high-level strategy look like? I assume I will have to get a remote machine and install the app on that machine?
any pointers, resources to read about would be helpful. Just looking for a nudge in the right direction.
Since you are using nUnit, it has a default approach of running tests from console see here
It looks like you need to pass parameters from the test runner into the runtime. You can use this approach
Execute on remote machine strategy depends on your current infrastructure. E. g. if you are using Gitlab, you should set up the GitLab CI runner into your machine and set up the GitLab pipeline.
looking for a nudge in the right direction
See CI\CD test run best practices
as posted here:
http://connect.microsoft.com/VisualStudio/feedback/details/748493/missing-funtionality-of-creating-a-new-unit-test-from-editor
there is no automatic way to add UT to existing code
Posted by doronneu on 6/14/2012 at 4:53 AM So How do I add new tests
to an existing testing project without coding them manually? Posted by
Microsoft on 6/14/2012 at 4:48 AM Hi,
.
The Test-> New Test menu was intentionally removed. In Visual Studio
2012, we have rearchitected the Unit Test framework to make it more
extensible. Thus we have adapters for nUnit/xUnit etc built already
along with the in-box mstest.
The 'New Test' menu does not have any extension points where 3rd party
adapters can plug-in. Therefore we decided to remove it. We moved all
the New Test Project types into the File -> New -> Project dialog.
Is there any workarround (resharper?) that enable adding UT to existing methods?
The best workaround I've found for Unit Testing in VS2012 is to go back to VS2010!
They are generally project-compatible, so jumping back and forth between the two versions of VS generally works very well (for C# projects).
The unit testing support in VS2012 has been butchered and is totally unfinished - how it was released in this terrible state is beyond me. (given how much we pay per licence, it's outrageous!) In 2010 you could easily get it to create all the boilerplate code you needed for a unit test, but that has all gone from 2012. In 2010 you could run your tests and see the progress. Gone, all gone - in 2012 you have no idea what is happening (if anything). In 2010 you would get results from the test run and have some idea of what happened. In 2012 it gives you an incomplete list of lies and then you check in your "passing" code and start swearing when your build server falls over with many failed tests that 2012 just swept under the carpet. In conclusion: my opinion is simply "Don't do unit testing in 2012, use 2010".
Hopefully VS 2013 is not long off, and they will have had a chance to implement unit testing support in it.
Now that I re-read your question and looked at the "MS Issue Post" it does look like they've moved the standard menu layout a bit. Also according to it there is no workaround and you need to contact MS support site regarding this.
However, there is enough information here to install a NUnit and start writing your tests. This still does not mean automatic though.
The project I'm working on has a bunch of service-tier unit tests using Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute. I want to look into some web frontend automated generation tool for these tests. I don't care if I need to use some other framework like NUnit. I need some decent way to have an easy web frontend for looking at test results, that also allows adding new tests in an easy manner.
After a bit of investigation I realised that we already have TeamCity for the builds. Do I need anything else to setup test browsing from teamcity?
We use Cruise Control, NAnt, SubVersion and NUnit together to provide continuous integration. Every commit triggers a build and runs all the unit tests. The cruise control dashboard show build results, unit test results and code coverage for each build. Is that the kind of thing you are looking to do or do you want some kind of web based ad hoc test runner?
Continuous Integration systems normally let you do this and usually have a web front end.
I know that you could set this up using CruiseControl.Net (which is free), the other system that has been recommended to me in the past is TeamCity so I'm sure that could do this too (and its also free as long as you don't configure too many projects).
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I've searched for this info and see similar questions, but not one that matches very closely to this. If I missed one, I apologize. I was hoping that you'd be able to point me in a direction. I'm working on a Silverlight based project and my team is finally moving toward implementing unit testing. I and another memeber of my team are responsible for recommending a unit testing framework for the project. Our basic criteria are:
The project contains standard ASMX .NET webservices and a Silverlight front end. We'd prefer, if at all possible, to maintain the same test attributes throughout, rather than use one set for Silverlight tests and another for other code.
Integration with VS 2008 is fairly important. We'd like to keep it all under one roof if that's feasible. We'd be happy with simply being able to kick the unit test off from VS.
Automated build/checkin testing. We are currently working in a completely non-automated VS 2008/VSS 2005 environment. We are in the process of converting to SVN for source control and our corporate office will be assisting us in using MSBuild to automate the build process. We'd certainly prefer to be able to integrate into this environment as much as possible. I'm not certain as to the details of this process as yet since I'm not directly involved. If there is more detail that you need on this, please let me know and I'll see what I can find out.
At this point, my collegue and I are looking at NUnit (along with possble Silverlight options) and MSTest in conjunction with the Silverlight Testing Framework that Jeff Wilcox wrote. While the corpoarate standard is NUnit, they are open to other options as there aren't any other teams doing Silverlight work.
I'm about halfway through Roy Osherove's Unit Testing Book, so I'm getting a feel for writing tests in general. I'm not married to a particular framework, but corporate seems to be most open to NUnit or MSTest. I'd also like to get my hands on a good tutorial of MSTest, if possible.
Thanks for taking the time. If you need any other info from me, feel free to ask.
Cheers,
Steve
Tread carefully, and make sure to experience these possibilities first. It's easiest to know what you're looking at by writing a few tests in each, and figure out what your workflow will be.
The Silverlight unit test systems out there are less mature than the officially shipped, full-fledged test frameworks for the desktop .NET environment, so your experience may vary. Understand that the Silverlight test solutions in place today (that actually run inside of a web browser) exist to specifically help folks get coverage of their code and components within the context of the Silverlight platform, and not necessarily for rapid, or easy, test-driven development and testing that you can get through the integrated Visual Studio tools.
The sandboxed security model for Silverlight makes many of the standard testing tools you may expect much more difficult.
It's also a different platform, really, so it may not make sense to perform every kind of test (example: a "load test" for a web app... you can "stress" a Silverlight app, or look into its "performance", but a "load test" is about stressing your machine that hosts the .XAP file, and is not very interesting for Silverlight apps specifically).
If you're more concerned about testing your business logic and having nice integration, strongly consider writing mockable/IoC code enough that you may be able to develop most of your app and test a version of it built as a regular desktop .NET class library ~ you could then have a subset of tests that are Silverlight in-browser only.
To your notes:
Are you actually going to be testing the web service calls with your tests? This sort of integration/client + server testing can be very, very difficult to do properly between Silverlight client code and a web service.
You won't get much integration with Visual Studio 2008's IDE for the Silverlight unit test framework. You can have a Silverlight test project / app that you can press F5 to open, and debug, in Visual Studio and run the tests. However, you won't have right-click "Run All Tests" or test case management tools working.
Automated test check-in for the Silverlight unit testing framework is something that is developing slowly, some folks have written a set of Silverlight test runners on CodePlex. You may not get this easily, and your build/test machine will need to be setup so that the build service runs instead as an actual user process, so that it can open and control web browser(s)
In the future, on the Silverlight Toolkit team, we are going to release some level of automated test support in the future, along with simple code coverage support, but these tools are not readily available as of today (Oct. '09)
I'm not going to recommend a unit testing framework, but I can warn that the Test attributes in MSTest are sealed (unable to be inherited in a new custom attribute which you create), unlike NUnit. In my mind, that fact shouldn't matter too much because I also suggest that, whichever testing framework you end up with, stick with its single Test attribute rather than creating your inherited one. To denote special tests (such as Silverlight like you mentioned) you should add an additional attribute. The tricky bit comes in when you want to only run the tests that have your custom attribute applied (as opposed to all of the unit tests), and that would require a custom driver for the tests.
I hope this helps.
I've got a desktop application written in C# created using VS2008 Pro and unit tested with Nunit framework and Testdriven.net plugin for VS2008. I need to conduct system testing on the application.
I've previously done web based system tests using Bad Boy and Selenium plugin for Firefox, but I'm new to Visual Studio and C#.
I would appreciate if someone could share their advice regarding this.
System testing will likely need to be done via the UI. This gives you two options:
1) You can manually conduct the test cases by clicking on elements.
2) You can automate the test cases by programming against the UI. There are plenty of commercial tools to do this or you can use a programming framework like the Microsoft UI Automation Framework. These tend to use the accessibility APIs built into Windows to access your UI.
Whether you go the manual or automated route depends on how many times you will be running the tests. If you are just going to run them once or twice, don't spend the time automating. You will never earn it back. If you are going to run them often, automating can be very handy.
A word of caution: Automating the UI isn't hard, but it is very brittle. If the application is changing a lot, the tests will require a lot of maintenance.
As Thomas Owens commented on your question, first you must decide what kind of system testing you want to do. But assuming you want start with Functional System Tests. Prepare use cases you want to automate. Than you must find proper tool.
Just for start:
AtoIT – is not test atomization tool but it lets automate some tasks. So you could record/script use cases. Not really recommended, but can be done.
HP QuickTestPro – easily can be done with this tool via recording/scripting but it is expensive, so maybe not worth it for personal use.
IBM Robot – as HP QTP.
Powershell – you could write scripts in powershell and execute them. If you would use dedicated ide-like tools for powershell you could record test also. I did some web automation via powershell and it worked. With a bit of work probably you could script around your desktop app.
And the best would be to try different tools, and use one that suits you best. Try this link and this link.
System tests usually have use cases, end to end scenarios and other scripted functions that real people execute. These are the tests that don't lend themselves well to automation as they are asking your unit-tested cogs to work with each other. You might have great unit tests for your "nuts" and your "wrenches" but only a comprehensive system test will let you know if you have the right sized wrench for the nut at hand, how to select/return it from/to the drawer, etc.
In short - manual tests.
If you're willing to put money down, you could look at something like TestComplete.
Although I haven't really used it yet (our company just bought it), it seems quite nice. You can record clicks and keypresses and stuff, define success criteria, and it will replay the test for you later. It appears to be quite smart about UI changes - it remembers which button you clicked, not just the (x,y) of each click.
It's scriptable, or drag-and-drop programmable.
I'm not affiliated in any way, and this is not an endorsement, because I haven't really formed an opinion of it yet.
Perhaps NUnitForms could be useful for you?