We Keep Coding

Unable to execute CodedUI tests from VSTS

I have created a CodedUI project in VS2017, i am able to execute the tests on my machine directly in VS2017.
I can easily bind test methods to VSTS online Test case. However, when i build and deploy the project and the assemblies on VSTS online and try to lauchn them from VSTS, i always got errors related to Deploying a Test agent on target machine(s) to be able to execute the test.
Just for information, the web pages we want to test are on a on-premise website, so not accessible from internet.
Do someone know how to make this work, ie how to set up the test machine, how to set up the Test agent on that machine (the microsoft documentation is not reallly clear about that), and what we need to make VSTS be able to communicate with our on-premise test machine.

The easy way is that you can setup a private build agent that can access test machine, then build the project through this private agent: Deploy an agent on Windows.
If you want to use Hosted agent, your test machine need to be accessible from internet.

Detect if my tests are running on Visual Studio Team Services

I've wrote a lot of integration tests for an API I'm working on.
If I run the integration tests locally I want to self host the API and run the integration tests against it.
However I have a build definition on VS Team services that runs the integration test after the API was deployed, I want to run the tests against the deployed API, but to do that I have to change my code so it no longer self host it and instead tested against the deployed API.
Is there a way to detect at runtime where my tests are being run?
Thanks

Just probe for one of the VSTS predefined build variables. For example:
if (Environment.GetEnvironmentVariable("SYSTEM_DEFINITIONID") != null)
{
Console.WriteLine("Running from VSTS...");
}

If you are using VSTS Hosted Build agent to run the build and test, the machine name of these agents has a format like "TASKAGENTX-XXXX". You can use it (Environment.MachineName) to determine if the test is ran on VSTS or your local machine.
By the way, what is the difference between the testing run on your local machine and VSTS? Is it just a different URL? If yes, you can add a config file to your project and update your tests to read the URL in the config file for testing. And then use SlowCheetah to perform a transformation to switch the URL when build and test from VSTS.

Building C# project using jenkins

First of all, im not sure if there is a specific forum thats more suitable for request about Jenkins, but I figured I just post it here.
So, I have this C# project, it right now requires Visual Studio 2015 to build and I would like to automate builds with Jenkins. This is the first time I'm using Jenkins and I was able to set it all up, it fetches the code from the gitlab repository, but right now there is no build action configured. For "fun" i tried to simply make it use a "windows batchfile" as build file and added echo test. Checking the console output it downloads the repository code and then tries to execute the batchfile, result is:
java.io.IOException: Cannot run program "cmd" (in directory "/var/lib/jenkins/workspace/Project"): java.io.IOException: error=2, No such file or directory
This of course makes sense, but I wonder now, is there a way to build a C# project with jenkins on a debian server without resorting to mono or something? I know that people build C++ projects for windows with Jenkins on a linux root, so there has to be some sort of cross compiling possibilities.
If the Visual Studio 2015 is a dealbreaker I can also strap it down to VS13 or even below. Any help is greatly appreciated!
/EDIT
Its probably important to say that Mono is NOT an option. Im using features like WPF that are not available for Mono.

Use Jenkins' MSBuild plugin.
Jenkins.NET is a good blog about using Jenkins in a .NET environment. First post is here.

How do I configure Visual Studio Online to run a custom unit test framework?

I'm trying to make the Visual Studio Online build service run my nSpec tests. I've downloaded the nSpec test adapter (which works fine locally), unzipped the DLLs and uploaded those to a separate TFS repository. I've configured the hosted build controller and set the reference to this repo.
As far as I know this should be sufficient, but my build simply doesn't find any unit tests. If I try to edit the build definition -> Click Process, it downloads custom assemblies it looks like, but it just writes this message to the console:
No assemblies were found in the custom assembly path. The assemblies may not exist or you may not have permissions to read them. Contact your Team Foundation Administrator for more information.
I've tried the same procedure with mSpec as well, but exactly the same happens here. nUnit seems to work though, but if I'm not entirely wrong, that might actually be installed already on hosted build server.

Unfortunately this requires the nSpec plugin to be installed on the build server and the TFS Hosted Build servers and you can only use the plugins provided. I believe that the servers only currently support MSTest, nUnit, and xUnit.
To get this working you will need to create your own custom build server. You can install a build server on Azure and install the tools you need. Then you can run your own build configuration there.

Framework for running automated acceptance tests after build?

In my current project we are using a TFS Build server for continuous integration (build + run unit tests). We also have a set of automated acceptance tests written as SpecFlow features.
However, these are not integrated into the continuous integration workflow. Today, the application is deployed manually and the acceptance tests are invoked manually.
We would like to automate this in the form of a script/console application or some kind of existing CI tool.
This is what we would like to do periodically, e.g. once every hour:
Ask TFS if there are any new builds
If yes: get the latest successful build from TFS
Deploy the application to our test machine
Execute the SpecFlow tests against the deployed build
Collect the result and present it on some form of web page
Are there any existing tools or frameworks for this? I have read about existing CI servers but they doesn't seem to fit my description. If not, any advice on how to achieve step 1, 2 and 5 programatically or by using command line tools?

In my humble opinion TFS is capable of doing everything that you listed without involving any additional tools. What you might need to do is to setup a Lab Environment and use specific Workflow build definition to achieve it. You need also Test controller and test agents.
The easiest way might be to setup Standard Lab environment which might act in this way -
Build - Deploy - Test workflow
Build got triggered, then got deployed into the lab environment (might be a bunch of either physical or VM machines with installed test agents on them and connected to the Test controller), after that all test are executed and the result is consolidated as a part of the build results.
Hope this helps a bit!
-Rado