In my current project we are using a TFS Build server for continuous integration (build + run unit tests). We also have a set of automated acceptance tests written as SpecFlow features.
However, these are not integrated into the continuous integration workflow. Today, the application is deployed manually and the acceptance tests are invoked manually.
We would like to automate this in the form of a script/console application or some kind of existing CI tool.
This is what we would like to do periodically, e.g. once every hour:
Ask TFS if there are any new builds
If yes: get the latest successful build from TFS
Deploy the application to our test machine
Execute the SpecFlow tests against the deployed build
Collect the result and present it on some form of web page
Are there any existing tools or frameworks for this? I have read about existing CI servers but they doesn't seem to fit my description. If not, any advice on how to achieve step 1, 2 and 5 programatically or by using command line tools?
In my humble opinion TFS is capable of doing everything that you listed without involving any additional tools. What you might need to do is to setup a Lab Environment and use specific Workflow build definition to achieve it. You need also Test controller and test agents.
The easiest way might be to setup Standard Lab environment which might act in this way -
Build - Deploy - Test workflow
Build got triggered, then got deployed into the lab environment (might be a bunch of either physical or VM machines with installed test agents on them and connected to the Test controller), after that all test are executed and the result is consolidated as a part of the build results.
Hope this helps a bit!
-Rado
Related
I would like to perform automated unit tests on my web-api project after publish it on IIS but I don't know how ? I can launch tests in Visual Studio, no problem for that.
I have some unit tests in a different project, must I generate the unit tests project an deploy it in the api-web site, and use another soft to perform tests ?
Thanks for your response.
Firstly, what you are proposing here does not sound like Unit tests. They are integration tests that test the entire integration and end to end working across application layers.
Regarding the automated execution, you can easily set that up using CI/CD. You can use VSTS or TeamCity like solutions to run your test cases at a particular schedule. Or, you can run your tests using command line from a system, that has Visual Studio installed.
I have a XAML build definition running my MSTest unit tests and it takes a long time to execute all of them. I have 4 build agents(VMs with 1 core each) setup in the build environment. I am using Microsoft.TeamFoundation.Build.Activities.RunAgileTestRunner activity in the XAML which is inside a "Run on Agent"(Microsoft.TeamFoundation.Build.Workflow.Activities.AgentScope) activity.
Currently, the unit tests execute on one of the agent. I want to be able to distribute the unit tests to other available agents. Is there a way to do that through TFS configuration or by changing the build definition.
As a workaround, I can split up the unit test project into multiple projects and run them separately on the available agents however I won't be able to see the build summary in one place.
What would be best possible solution for this?
Just check this article : How to run automated tests on different machines in parallel?
The Recommended option:
Create multiple environments with each environment consisting of single machine (read single agent).
Divide all your test cases onto different sets and queue one set on one environment and other set on other environment and so on.
With this each set of test will run in parallel and on different machine.
Also reference below articles:
Hint 2: Configure to run multiple testrunners in parallel in their
own application domain
How To: Run Tests On Different Machines In Parallel
Besides, If upgrading to VNext build is an option, then you can easily achieve that by following the steps mentioned in below articles:
Running Unit Tests on different machine during TFS 2015 build
Speeding up Unit Test Execution in TFS
I am completely new to Azure and PowerShell but have been tasked with setting up a build and deploy solution for several app services.
We currently have a build server (Azure VM) that is running CruiseControl.NET to build and test some C# .NET solutions that should be deployed in Azure.
This build server currently handles the following tasks:
Pulling code from source control when commits happen
Building the projects
Running some unit test cases
Copying output/binaries to an output location
However, as it exists now, developers of each of our app services need to 'Publish' their services manually from their development machines by clicking the button in Visual Studio once they have verified that the build and test cases have passed in the test environment on the server.
As I am hoping for a completely automated solution, I expect I need to use something like PowerShell or the Azure Cross Platform CLI (npm) to do this?
I'm extremely confused with the Azure Service Management vs Azure Resource Management versions with the new Azure Powershell 1.0. All of our services appear to be the newer Resource Management versions, not 'classic'.
The eventual goal is to have the build server do the following
Pulling code from source control when commits happen
Building the projects
Running some unit test cases
Copying output/binaries to an output location
If the build and test cases are successful, update the service in azure to the latest build
I am hoping there is a way to set up these projects, or take the existing binaries that result from the builds, and have them be deployed into Web Apps using the new Azure Resource Management Powershell features.
Any advice or resources for more information about how this can be done?
Hopefully this makes some sense. Please let me know if I am going about this completely the wrong way or direct me to a more correct forum.
Thanks!
have you consider to use Azure App Service? where you can get those build infrastructure for free. e.g https://azure.microsoft.com/en-us/documentation/articles/web-sites-publish-source-control/
Once you setup continues deployment, you will get below three when there is push event (if you are using git)
Pulling code from source control when commits happen
Building the projects
Copying output/binaries to an output location
and to "Running some unit test cases", you can create your own batch or powershell script with post deployment hook https://github.com/projectkudu/kudu/wiki/Post-Deployment-Action-Hooks
I'm trying to make the Visual Studio Online build service run my nSpec tests. I've downloaded the nSpec test adapter (which works fine locally), unzipped the DLLs and uploaded those to a separate TFS repository. I've configured the hosted build controller and set the reference to this repo.
As far as I know this should be sufficient, but my build simply doesn't find any unit tests. If I try to edit the build definition -> Click Process, it downloads custom assemblies it looks like, but it just writes this message to the console:
No assemblies were found in the custom assembly path. The assemblies may not exist or you may not have permissions to read them. Contact your Team Foundation Administrator for more information.
I've tried the same procedure with mSpec as well, but exactly the same happens here. nUnit seems to work though, but if I'm not entirely wrong, that might actually be installed already on hosted build server.
Unfortunately this requires the nSpec plugin to be installed on the build server and the TFS Hosted Build servers and you can only use the plugins provided. I believe that the servers only currently support MSTest, nUnit, and xUnit.
To get this working you will need to create your own custom build server. You can install a build server on Azure and install the tools you need. Then you can run your own build configuration there.
In Visual Studio Team Edition there is a load test tool where you can select unit tests, integration tests etc... to form scenarios which are then used in a load test session with different configurations for example varying usage levels.
Looking at for example the NUnit framework, it would be possible to develop a form of such a tool, along with a base load testing code base, but I was interested if anyone knows of any open source project which has has already done a similar thing which could then be of use for this situation and possibly extended?
Cheers for now,
Andrew
Does this help?
http://www.webload.org/
It's only a load tester, not also built with Unit Tests, Integration Tests, etc - However how about a Continuous Integration setup with CruiseControl.NET as your CI Server which would run automated builds and carry out the tests and then use Webload for your load testing? You can certainly get CruiseControl.NET to run external tools - but IDK whether or not Webload has a console which could automatically run the load tests for you - but it probably could ;)