I have a XAML build definition running my MSTest unit tests and it takes a long time to execute all of them. I have 4 build agents(VMs with 1 core each) setup in the build environment. I am using Microsoft.TeamFoundation.Build.Activities.RunAgileTestRunner activity in the XAML which is inside a "Run on Agent"(Microsoft.TeamFoundation.Build.Workflow.Activities.AgentScope) activity.
Currently, the unit tests execute on one of the agent. I want to be able to distribute the unit tests to other available agents. Is there a way to do that through TFS configuration or by changing the build definition.
As a workaround, I can split up the unit test project into multiple projects and run them separately on the available agents however I won't be able to see the build summary in one place.
What would be best possible solution for this?
Just check this article : How to run automated tests on different machines in parallel?
The Recommended option:
Create multiple environments with each environment consisting of single machine (read single agent).
Divide all your test cases onto different sets and queue one set on one environment and other set on other environment and so on.
With this each set of test will run in parallel and on different machine.
Also reference below articles:
Hint 2: Configure to run multiple testrunners in parallel in their
own application domain
How To: Run Tests On Different Machines In Parallel
Besides, If upgrading to VNext build is an option, then you can easily achieve that by following the steps mentioned in below articles:
Running Unit Tests on different machine during TFS 2015 build
Speeding up Unit Test Execution in TFS
Related
I have a unique case where I need to run the integration tests on the same server that the API has been deployed to. In our current pipeline, we build the API and then deploy it to the alpha test server. What I want to achieve is adding another step after the deployment that will build and deploy the xunit test project and run it on the same server then get the results of the test back and if any of the tests fail I want to revert the deployment. I know that I can run the tests from the command line in windows, but first I basically need the entire bin folder to be created/build and put on the server. I know it's not the usual way to be doing integration tests in azure pipelines but that's a requirement for our project. We are not going to use Docker containers for a while. We use the new YAML pipelines.
That is perfectly straightforward. In your pipeline, just add a task (after the deployment task) to run the tests! The build artifact will already be present with all the integration test code, so you just call the test runner in much the same way as you would when running them locally.
I would like to perform automated unit tests on my web-api project after publish it on IIS but I don't know how ? I can launch tests in Visual Studio, no problem for that.
I have some unit tests in a different project, must I generate the unit tests project an deploy it in the api-web site, and use another soft to perform tests ?
Thanks for your response.
Firstly, what you are proposing here does not sound like Unit tests. They are integration tests that test the entire integration and end to end working across application layers.
Regarding the automated execution, you can easily set that up using CI/CD. You can use VSTS or TeamCity like solutions to run your test cases at a particular schedule. Or, you can run your tests using command line from a system, that has Visual Studio installed.
I have two different application installed on two different machines/servers. I have done the automation using C# of both the application, they are integrated at data level and i have to trigger a single build and my scripts would randomly execution on both machines. Both are desktop applications.
Now the problem is can we configure two agents with single TFS build installed on two different machines, like my some of the scripts would run on Machine A and there would be a listener A, and in the same build some of the scripts would run on machine B where there is other application and i would surely install the agent as well.
Refer to these steps below:
Configure multiple agents in an agent pool.
Choose Multi-agent option in Execution plan of Phase
Choose Based on number of tests and agents in Batch tests box of Visual Studio test
Related article: Test batching options with VSTest task
I think I know the answer to this question but I haven't found anything particularly definitive.
TFS 2015
I have many very simple tests that need to be run in a distributed fashion. However from my understanding, I can either go parallel at the assembly level (which is ridiculous) or I can set multiple test agents in the runsettings file and distribute tests that way. I'd prefer not to have to spin up 10 tiny VMs but rather a one or two respectable sized VMs instead with multiple instances of the VSTA.
Thanks for any responses.
Test Agents are mainly for running Functional Tests that have automated UI interaction. These Test, which require desktop interaction can only be run one at a rime per VM.
If you are talking about Automated Tests that do not have dektop interaction then you can use Team Build, add the Run Visual Studio Tests and tick the Parallel option.
Although the screenshots are from VSTS this option is the same in TFS 2015. There is no support for parallel tests in the old legacy Xaml Builds.
In my current project we are using a TFS Build server for continuous integration (build + run unit tests). We also have a set of automated acceptance tests written as SpecFlow features.
However, these are not integrated into the continuous integration workflow. Today, the application is deployed manually and the acceptance tests are invoked manually.
We would like to automate this in the form of a script/console application or some kind of existing CI tool.
This is what we would like to do periodically, e.g. once every hour:
Ask TFS if there are any new builds
If yes: get the latest successful build from TFS
Deploy the application to our test machine
Execute the SpecFlow tests against the deployed build
Collect the result and present it on some form of web page
Are there any existing tools or frameworks for this? I have read about existing CI servers but they doesn't seem to fit my description. If not, any advice on how to achieve step 1, 2 and 5 programatically or by using command line tools?
In my humble opinion TFS is capable of doing everything that you listed without involving any additional tools. What you might need to do is to setup a Lab Environment and use specific Workflow build definition to achieve it. You need also Test controller and test agents.
The easiest way might be to setup Standard Lab environment which might act in this way -
Build - Deploy - Test workflow
Build got triggered, then got deployed into the lab environment (might be a bunch of either physical or VM machines with installed test agents on them and connected to the Test controller), after that all test are executed and the result is consolidated as a part of the build results.
Hope this helps a bit!
-Rado