Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I've searched for this info and see similar questions, but not one that matches very closely to this. If I missed one, I apologize. I was hoping that you'd be able to point me in a direction. I'm working on a Silverlight based project and my team is finally moving toward implementing unit testing. I and another memeber of my team are responsible for recommending a unit testing framework for the project. Our basic criteria are:
The project contains standard ASMX .NET webservices and a Silverlight front end. We'd prefer, if at all possible, to maintain the same test attributes throughout, rather than use one set for Silverlight tests and another for other code.
Integration with VS 2008 is fairly important. We'd like to keep it all under one roof if that's feasible. We'd be happy with simply being able to kick the unit test off from VS.
Automated build/checkin testing. We are currently working in a completely non-automated VS 2008/VSS 2005 environment. We are in the process of converting to SVN for source control and our corporate office will be assisting us in using MSBuild to automate the build process. We'd certainly prefer to be able to integrate into this environment as much as possible. I'm not certain as to the details of this process as yet since I'm not directly involved. If there is more detail that you need on this, please let me know and I'll see what I can find out.
At this point, my collegue and I are looking at NUnit (along with possble Silverlight options) and MSTest in conjunction with the Silverlight Testing Framework that Jeff Wilcox wrote. While the corpoarate standard is NUnit, they are open to other options as there aren't any other teams doing Silverlight work.
I'm about halfway through Roy Osherove's Unit Testing Book, so I'm getting a feel for writing tests in general. I'm not married to a particular framework, but corporate seems to be most open to NUnit or MSTest. I'd also like to get my hands on a good tutorial of MSTest, if possible.
Thanks for taking the time. If you need any other info from me, feel free to ask.
Cheers,
Steve
Tread carefully, and make sure to experience these possibilities first. It's easiest to know what you're looking at by writing a few tests in each, and figure out what your workflow will be.
The Silverlight unit test systems out there are less mature than the officially shipped, full-fledged test frameworks for the desktop .NET environment, so your experience may vary. Understand that the Silverlight test solutions in place today (that actually run inside of a web browser) exist to specifically help folks get coverage of their code and components within the context of the Silverlight platform, and not necessarily for rapid, or easy, test-driven development and testing that you can get through the integrated Visual Studio tools.
The sandboxed security model for Silverlight makes many of the standard testing tools you may expect much more difficult.
It's also a different platform, really, so it may not make sense to perform every kind of test (example: a "load test" for a web app... you can "stress" a Silverlight app, or look into its "performance", but a "load test" is about stressing your machine that hosts the .XAP file, and is not very interesting for Silverlight apps specifically).
If you're more concerned about testing your business logic and having nice integration, strongly consider writing mockable/IoC code enough that you may be able to develop most of your app and test a version of it built as a regular desktop .NET class library ~ you could then have a subset of tests that are Silverlight in-browser only.
To your notes:
Are you actually going to be testing the web service calls with your tests? This sort of integration/client + server testing can be very, very difficult to do properly between Silverlight client code and a web service.
You won't get much integration with Visual Studio 2008's IDE for the Silverlight unit test framework. You can have a Silverlight test project / app that you can press F5 to open, and debug, in Visual Studio and run the tests. However, you won't have right-click "Run All Tests" or test case management tools working.
Automated test check-in for the Silverlight unit testing framework is something that is developing slowly, some folks have written a set of Silverlight test runners on CodePlex. You may not get this easily, and your build/test machine will need to be setup so that the build service runs instead as an actual user process, so that it can open and control web browser(s)
In the future, on the Silverlight Toolkit team, we are going to release some level of automated test support in the future, along with simple code coverage support, but these tools are not readily available as of today (Oct. '09)
I'm not going to recommend a unit testing framework, but I can warn that the Test attributes in MSTest are sealed (unable to be inherited in a new custom attribute which you create), unlike NUnit. In my mind, that fact shouldn't matter too much because I also suggest that, whichever testing framework you end up with, stick with its single Test attribute rather than creating your inherited one. To denote special tests (such as Silverlight like you mentioned) you should add an additional attribute. The tricky bit comes in when you want to only run the tests that have your custom attribute applied (as opposed to all of the unit tests), and that would require a custom driver for the tests.
I hope this helps.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I have searched and tried a few things already namely this thread:
How to test a WPF user interface?
I have tried getting started with Systems.Windows.Automation and TestAutomationFX(3rd party tool). My opinion is that while good for simple things, TestAutomation kind of bombs when it goes multiple UI levels down(a usercontrol within a usercontrol from a loaded assembly) and I may have to manually tweak their code behind to get what I want. System.Windows.Automation seems old and I would have to do everything manually which may take more time than I want to devote as I am not full time on automation creation. I have also downloaded the Inspect.exe from the Windows SDK for Windows 7, which works great for reflecting objects in my UI. Both testers run fine for simple code behind but then when it gets a few layers down it seems a bit involved. I was also going to try the 'TestSTack/White' on GitHub that moved from the original Project White.
I was curious if anyone had experience recently in UI automation that a non developer could use in a QA position. I was thinking of getting VS 2013 Test Pro but that seems like overkill potentially and is more expense than the VS 2013 Pro from what I could see. Basically this is not load testing or verbose complex dynamic entity results changing, just function testing with hopefully ten or so runs of different variables. It is just a more confusing layout as we are combining the Prism method, Microsoft.Practices.Prism, with MVVM as well.
I do not mind developing something in VS 2013 and .NET 4.5 but I was hoping to get something that I am not developing a whole other set of projects for, but to save time. I am an extreme noob at unit testing projects but the end goal is really:
Give a non developer an exe or some environment to automatically run a Click Once UI written in WPF following some Prism and MVVM architecture.
Hopefully have some type of CSV, config, or other method he can change variables to run certain tests on.
Have it be able to input the exe of the click once app in a config or changeable manner(Click Once is funny finding it in my experience of opening Task Manager and then 'open location' of the click once app, which differs from box to box).
This may be a lot to ask, or it may be simple for those that do unit testing every day, I dunno. I am up to try 3rd party products, non .NET products to run .NET, and coding in C# in VS to make a project(s) to run my UI(as long as it can be ran on boxes not have VS).
Ideally, you wouldn't need to have many UI tests at all -- the bulk of your application's logic should be tested via unit tests. With MVVM, you can easily test your viewmodel to ensure that buttons are disabled/enabled when they're supposed to be, and so on.
Testing your core business logic via your UI is a recipe for disaster. Just don't do it. UI tests are very brittle and tend to need to be re-recorded or updated if your UI changes to any significant degree. If your tests fail for reasons unrelated to the core logic you're trying to test changing, you're less prone to trust that the tests validate what they're supposed to validate. If you don't trust the tests, you'll start to ignore failures. "Oh, that test fails sometimes, it's no big deal." If you can't trust your test to be accurate 100% of the time, why bother having the test?
So what you want to test via a UI testing tool is the very top-most level of the UI, just to make sure that your viewmodel is correctly bound to your view. This boils down to, really, just a handful of tests. For that, you can easily use Coded UI. The tricky part is making sure that all of your controls are automation-friendly, which does involve giving the controls proper names and making sure you're attaching the correct automation properties to your controls.
Coded UI is available in VS Premium and above, and you don't need to be using Microsoft Test Manager to manage and run the tests, although it's certainly easier.
It sounds like what you're really after is MTM, though. You want your manual testers to be able to record tests by interacting with your application, then play them back later. This is exactly what MTM was designed for, and what it excels at.
Sadly this answer sounds like too late for the stage you are at but I am happy with the basics of MVVM Light Toolkit:
Basically you start by setting up with some IOC container and dependency injection scheme and PRISM. Your Services can then provide design time, run time and test time implementations with mocking etc. There are some videos and tutorials but like most of wpf it is hard as a newcomer to sort through ancient obsolete stuff vs relevant best practices.
MVVM Light at least has a focus on enabling Blend to work at design time and smells like some kind of best practice.
Now for the part where this sadly does not answer your question: the idea is to be able to layout in Blend so you can see what things will look like without endless tweak-compile-run cycles. Testing is purely on the underlying ViewModel and Model. You then arm wave that the app works because the UI bindings are unlikely to be wrong / are relatively simple to manually run through and verify. That last part works for my project but may be deeply unsatisfying for yours.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I am interested in Behavior Driven Development (BDD)
Are there any good frameworks and/or tooling available for the .Net platform (preferably C# and ASP.Net)?
And when has BDD proved most appropriate to use in real world scenarios? (auxiliary question)
+1 for people's recommendation of SpecFlow for scenarios; never used it but heard many good things about it. I've been using plain old NUnit with a little DSL like this. MSTest would work similarly.
You can also do BDD in the unit space, which is what MSpec is designed to do. I'm personally hating MSpec, but the rest of the team here love it. They like writing examples of how the code works. I like to show why the behavior is valuable. It's a subtle distinction and if you're not worried about doing this at a unit level it won't hit you.
Other frameworks to look at include Concordion, Fitnesse.NET (please put FitSharp behind it!) and TickSpec.
In the real world, the most valuable bit of BDD by a long way is the conversations, not the automated tests. Here's a couple of quick hints and tips for making it work:
Don't write automated tests over things which are in flux. It just commits you to stuff you got wrong. Wait until the UI has settled a bit then do it.
If you don't care much about your UI, but do care about data integrity, write the scenarios over the controller / presenter layer (eg: for admin screens).
Don't start with login. Start by describing a valuable part of the application for which you might log in. Do that first (assume you only have one user). You'll get quicker feedback on the risky bits.
Seek quick feedback on the risky bits, which will usually be the bits which you've never done before. Use scenarios to have conversations around them. Write anything interesting you discover down, but forget the scenarios which are obvious - they're obvious! Don't worry about automating them to start with. Having conversations is more important than writing down conversations is more important than automating conversations.
Good luck! If you want to know more about BDD, I put together a page of relevant links here.
On googling I found Behavior Driven Development (BDD) with SpecFlow and ASP.NET MVC. You might find it useful, take a look. Also go through Behavior-Driven Development with SpecFlow and WatiN
A nice presentation on Pros and Cons of BDD
A channel 9 video Behavior-Driven Development in the Real World
and last but not least an InfoQ article Behavior Driven Development In .NET
LightBDD is an open source framework allowing to write BDD tests that are easy to read but also easy to maintain and extend when project grows larger.
The main features that it offers are:
easy to read scenarios,
easy maintenance of tests,
integration with well known testing frameworks (NUnit / MbUnit / MsTest / xUnit),
scenario steps execution tracking and execution time measurement,
test execution summary report generation in HTML (an example report), XML and plain text format.
It bases on tests that are written purely in code, which means native support for refactoring, code analysis, test running and all other features that Visual Studio / Intellisense / Resharper offers.
An example test written in this framework looks as follows:
[TestFixture]
[FeatureDescription(
#"In order to access personal data
As an user
I want to login into system")] //feature description
[Label("Story-1")]
public partial class Login_feature //feature name
{
[Test]
[Label("Ticket-1")]
public void Successful_login() //scenario name
{
Runner.RunScenario(
Given_user_is_about_to_login, //steps
Given_user_entered_valid_login,
Given_user_entered_valid_password,
When_user_clicked_login_button,
Then_login_is_successful,
Then_welcome_message_is_returned_containing_user_name);
}
}
More information about framework could be found on project wiki page and project main page.
Also MSpec is a good framework.
I use it in the Microsoft stack you mention (C#, ASP.Net and MVC) and I like his syntax.
BDD helps you thinking in business/feature oriented way not just in a code way.
So you are most focused on business value.
It also helps in user acceptance test to create a trust between you and customer.
There is a great tool, called SpecFlow.
SpecFlow is inspired by Cucumber — the well known BDD framework for Ruby on Rails. And has a huge amount of advantages.
You should definitely check it out.
One interesting BDD frameworks is Concordion.NET. It is an open source BDD framework for the Microsoft stack that uses NUnit to execute the Concordion.NET tests: https://github.com/concordion/concordion-net
As Concordion specifications are written in simple HTML, it provides a good basis for a living documentation system. You can use a what-you-see-is-what-you-get (WYSIWYG) editor like BlueGriffon to describe your intended behavior of your software in simple HTML documents and instrument them to verify your system under test.
According to the excellent classification of BDD tools, Concordion.NET focuses on business readable input (and reaches business readable output as well). It moves even beyond BDD and supports ATDD: http://assertselenium.com/2012/11/05/difference-between-tdd-bdd-atdd/
Spec4Net (https://bitbucket.org/fthomsen/spec4net/) is a nice framework too. We use it extensively at work. The learning curve is almost non-existing and the natural flow seems intuitive.
I am looking for any ASP.NET (WebForms & C#) app which has some good unit tests in its solution. Good meaning testing different kinds of edge cases and does a good code coverage. Any app on CodePlex, GitHub or anywhere is fine.
This is for educational purposes so I prefer smaller apps than large ones.
Any recommendations?
Clarification:
While the app is WebForms, the unit tests I am interested is more on business logic, not the UI. Yes any .NET app can do but if it is WebForms with some UI testing, the better.
ScrewTurn wiki is open-source, version 3.x is coded in C# and ASP.NET 3.5, and the source comes bundled with tests for the components.
I personally believe that unit tests in the business logic layer are not sufficient for any web application, no matter what the framework used.
The best policy is to have a serious "smoke test", perhaps kept in an Excel workbook, where you exercise every known possible set of inputs and actions and are able to determine that "everything works" whenever you do a new deployment.
Unit tests are valuable, but they cannot reproduce the behavior of an application in production.
Take a look at this. Don't know exactly if it's what you wanted :) It is ASP.NET with MVP pattern implemented and it has some test coverage - .NET Framework 3.5 Bundle
I've just finish a web e-commerce site.
For the business logic I create a separate dll, and I use NUnit to test it. Nunit is simple to use, it's simple (and fast) to launch with a .bat file and it is possible to use a graphic interface.
In the web site, I use NLOG for logging user action. In this case the best way (after YOU test it for a while) is to give your site to some final customer, and watch the log file every time there is an exception.
I know you are looking for coding examples to learn unit testing but
Foundations of Programming - Building Better Software By Karl Seguin
is a good book to help guide you through the process of unit testing a mocking. It's only 79 pages long and should get you up and running quickly.
http://openmymind.net/FoundationsOfProgramming.pdf
I've got a desktop application written in C# created using VS2008 Pro and unit tested with Nunit framework and Testdriven.net plugin for VS2008. I need to conduct system testing on the application.
I've previously done web based system tests using Bad Boy and Selenium plugin for Firefox, but I'm new to Visual Studio and C#.
I would appreciate if someone could share their advice regarding this.
System testing will likely need to be done via the UI. This gives you two options:
1) You can manually conduct the test cases by clicking on elements.
2) You can automate the test cases by programming against the UI. There are plenty of commercial tools to do this or you can use a programming framework like the Microsoft UI Automation Framework. These tend to use the accessibility APIs built into Windows to access your UI.
Whether you go the manual or automated route depends on how many times you will be running the tests. If you are just going to run them once or twice, don't spend the time automating. You will never earn it back. If you are going to run them often, automating can be very handy.
A word of caution: Automating the UI isn't hard, but it is very brittle. If the application is changing a lot, the tests will require a lot of maintenance.
As Thomas Owens commented on your question, first you must decide what kind of system testing you want to do. But assuming you want start with Functional System Tests. Prepare use cases you want to automate. Than you must find proper tool.
Just for start:
AtoIT – is not test atomization tool but it lets automate some tasks. So you could record/script use cases. Not really recommended, but can be done.
HP QuickTestPro – easily can be done with this tool via recording/scripting but it is expensive, so maybe not worth it for personal use.
IBM Robot – as HP QTP.
Powershell – you could write scripts in powershell and execute them. If you would use dedicated ide-like tools for powershell you could record test also. I did some web automation via powershell and it worked. With a bit of work probably you could script around your desktop app.
And the best would be to try different tools, and use one that suits you best. Try this link and this link.
System tests usually have use cases, end to end scenarios and other scripted functions that real people execute. These are the tests that don't lend themselves well to automation as they are asking your unit-tested cogs to work with each other. You might have great unit tests for your "nuts" and your "wrenches" but only a comprehensive system test will let you know if you have the right sized wrench for the nut at hand, how to select/return it from/to the drawer, etc.
In short - manual tests.
If you're willing to put money down, you could look at something like TestComplete.
Although I haven't really used it yet (our company just bought it), it seems quite nice. You can record clicks and keypresses and stuff, define success criteria, and it will replay the test for you later. It appears to be quite smart about UI changes - it remembers which button you clicked, not just the (x,y) of each click.
It's scriptable, or drag-and-drop programmable.
I'm not affiliated in any way, and this is not an endorsement, because I haven't really formed an opinion of it yet.
Perhaps NUnitForms could be useful for you?
We have written our own integration test harness where we can write a number of "operations" or tests, such as "GenerateOrders". We have a number of parameters we can use to configure the tests (such as the number of orders). We then write a second operation to confirm the test has passed/Failed (i.e. there are(nt) orders).
The tool is used for
Integration Testing
Data generation
End-to-end testing (By mixing and matching a number of tests)
It seems to work well, however requires development experience to maintain and write new tests. Our test team would like to get involved, who have little C# development experience.
We are just about to start a new Greenfield project and I am doing some research into the optimum way to write and maintain integration tests.
The questions are as follows:
How do you perform integration testing?
What tool do you use for it (FitNess?, Custom?, NUnit)?
I am looking forward to peoples suggestions/comments.
Thanks in advance,
David
Integration testing may be done at a user interface level (via automated functional tests - AFT) or service/api interface level.
There are several tools in both cases:
I have worked on projects that successfully used Sahi or Selenium for AFT of web apps, white for AFT for .NET WPF or winforms apps, swtBot for AFT of Eclipse Rich client apps and frankenstein for AFT of Java swing apps.
Fitnesse is useful for service/api level tests or for tests that run just below the UI. When done right, it has the advantage of having business-readable tests i.e. non-developers can read and understand the tests. Tools like NUnit are less useful for this purpose. SOAPUI is particularly suited for testing SOAP web services.
Factors to consider:
Duration: Can you tolerate 8 hour test runs?
Brittleness: AFTs can be quite brittle against an evolving application (e.g. ids and positions of widgets may change). Adequate skill and effort is needed to not hard code the changing parts.
Fidelity: How close to real world do you want it to be? e.g. You may have to mock out interactions with a payment gateway unless the provider provides you a test environment that you can pummel with your tests.
Some nuances are captured here.
Full disclosure: The author is associated with the organization behind most (not all) of the above free and open source tools.
You could try the Concordion framework for writing user acceptance tests in HTML files. It takes a BDD-style approach. There is a .Net port as well
It's not out of Beta yet, but StoryTeller looks promising: