We Keep Coding

dotnet publish with new implicit dockerfile feature using NuGet Microsoft.NET.Build.Containers fails in GitLab cicd

I am trying out the new NuGet package Microsoft.NET.Build.Containers which allows building docker image with dotnet publish.
Microsoft Docs: https://learn.microsoft.com/en-us/dotnet/core/docker/publish-as-container?view=vs-2022
It works fine on local machine pushing out an image locally as it should but fails in GitLab cicd.
Project (added following to a sln file):
# create a new project and move to its directory
dotnet new mvc -n my-awesome-container-app
cd my-awesome-container-app
# add a reference to a (temporary) package that creates the container
dotnet add package Microsoft.NET.Build.Containers
# publish your project for linux-x64
dotnet publish --os linux --arch x64 -c Release -p:PublishProfile=DefaultContainer
# run your app using the new container
docker run -it --rm -p 5010:80 my-awesome-container-app:1.0.0
Cicd pipeline:
image: mcr.microsoft.com/dotnet/sdk:7.0
stages: # List of stages for jobs, and their order of execution
- publish
build-job: # This job runs in the build stage, which runs first.
stage: publish
script:
- dotnet publish --os linux --arch x64 --configuration Release -p:PublishProfile=DefaultContainer
The error produced (omitted repo):
MSBuild version 17.4.1+9a89d02ff for .NET
Determining projects to restore...
Restored /builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj (in 827 ms).
my-awesome-container-app -> /builds/.../dotnetpublishdockerimage/my-awesome-container-app/bin/Release/net7.0/linux-x64/my-awesome-container-app.dll
my-awesome-container-app -> /builds/.../dotnetpublishdockerimage/my-awesome-container-app/bin/Release/net7.0/linux-x64/publish/
Building image 'my-awesome-container-app' with tags 1.0.0 on top of base image mcr.microsoft.com/dotnet/aspnet:7.0
/root/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: The "CreateNewImage" task failed unexpectedly. [/builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: System.AggregateException: One or more errors occurred. (An error occurred trying to start process 'docker' with working directory '/builds/.../dotnetpublishdockerimage/my-awesome-container-app'. No such file or directory) [/builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: ---> System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'docker' with working directory '/builds/.../dotnetpublishdockerimage/my-awesome-container-app'. No such file or directory [/builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec) [/builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo) [/builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) [/builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: at Microsoft.NET.Build.Containers.LocalDocker.Load(Image x, String name, String tag, String baseName) in D:\a\_work\1\s\Microsoft.NET.Build.Containers\LocalDocker.cs:line 19 [/builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: --- End of inner exception stack trace --- [/builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) [/builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) [/builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: at System.Threading.Tasks.Task.Wait() [/builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: at Microsoft.NET.Build.Containers.Tasks.CreateNewImage.Execute() in D:\a\_work\1\s\Microsoft.NET.Build.Containers\CreateNewImage.cs:line 243 [/builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj]
/root/.nuget/packages/microsoft.net.build.containers/0.2.7/build/Microsoft.NET.Build.Containers.targets(124,9): error MSB4018: at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/builds/.../dotnetpublishdockerimage/my-awesome-container-app/my-awesome-container-app.csproj]
Searching for a possible solution I came across and article for it for GitHub actions and it also works there. Also the example in there I have used for this questions project.
Article: https://devblogs.microsoft.com/dotnet/announcing-builtin-container-support-for-the-dotnet-sdk/
Also the GitLab runner is using Docker executor. Perhaps it doesn't work inside a container. The article for GitHub shows ubuntu-latest but I do not know if it is running in a container underneath or not.
Anyone know what is going on and have a solution?

Looking further into it I found the issue and a solution as well.
There are 2 problems here.
In order to build images (part of dotnet publish) docker-in-docker (dind) is required.
The .net sdk image does not provide dind support
To deal with the first problem there are a couple of changes needed.
First add dnid as a service in cicd:
services:
- docker:dind
Nex assign the docker host variable:
variables:
DOCKER_HOST: tcp://docker:2375/
And finally to enable privileged mode on docker runner (config.toml):
[[runners]]
...
[runners.docker]
...
privileged = true
...
Restart of docker runner may be required.
There can also be an issue with tls. It can be ignored (although not advised for privileged mode) by adding empty certs dir to cicd:
variables:
DOCKER_TLS_CERTDIR: ""
The deal with the second problem an alternative is to use the image 'docker' which provides dind support. From this there are 2 options:
Install the .net sdk inside:
before_script:
- apk add dotnet7-sdk
Official docs: https://learn.microsoft.com/en-us/dotnet/core/install/linux-alpine
Create a new docker image based off of docker with sdk installation added and then use this image in pipeline.
There may be other more elegant ways to solve this but it will do as a workaround.
A sample cicd pipeline including publishing to registry:
stages:
- publish
variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_TLS_CERTDIR: ""
services:
- docker:dind
docker_test:
stage: publish
image: docker
before_script:
- apk add dotnet7-sdk
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- dotnet publish --os linux --arch x64 --configuration Release -p:PublishProfile=DefaultContainer
- IMAGE_ID=$(docker images --format='{{.ID}}' | head -1)
- docker tag $IMAGE_ID $CI_REGISTRY_IMAGE/my-image:1.0.0
- docker push $CI_REGISTRY_IMAGE/my-image:1.0.0

Unreal Engine build crashes at C# trying to open Recycle Bin file

This boggles my mind, to the point that I wonder if isn't an SDK problem.
I share an Unreal Engine (4.26) game with a (non-technical) colleague through Git. Today he received his first code files from me, and to compile it, I had him install the SDKs(windows, .Net, C++ gaming, Visual Studio) and call Generate Visual Studio Project Files. It all generated just fine.
EDIT:
Emptying the recycle bin somehow fixed it, but now the problem repeats itself:
ERROR: Unhandled exception: System.UnauthorizedAccessException: Access to the path 'D:\System Volume Information' is denied.
So this entirely seems like a permission level problem to me, but I asked him to enable developer mode, to uncheck every read-only on Epic Games and Unreal Engine, and to run it in administrator mode. Nothing works. I don't see how we can get any more permissions than this.
Former message:
ERROR: Unhandled exception: System.UnauthorizedAccessException: Access to the path 'D:$RECYCLE.BIN\S-1-5-18' is denied.
Full logfile(minus some cleanup on my part):
Log file open, 04/17/21 12:00:08
LogInit: LLM is enabled
LogInit: LLM CsvWriter: off TraceWriter: off
LogInit: Display: Running engine for game: Naptin
LogPlatformFile: Not using cached read wrapper
LogTaskGraph: Started task graph with 5 named threads and 35 total threads with 3 sets of task threads.
LogStats: Stats thread started at 5.907298
LogD3D11RHI: Loaded GFSDK_Aftermath_Lib.x64.dll
LogICUInternationalization: ICU TimeZone Detection - Raw Offset: -5:00, Platform Override: ''
LogPluginManager: Mounting plugin MeshPainting
.....lots of LogPluginManager......
LogPluginManager: Mounting plugin SteamVR
LogXGEController: Cleaning working directory: C:/Users/adpar/AppData/Local/Temp/UnrealXGEWorkingDir/
LogXGEController: Cannot use XGE Controller as Incredibuild is not installed on this machine.
LogInit: Warning: Incompatible or missing module: Naptin
Running D:/.........UNREALENGINEFOLDER/UE_4.26/Engine/Binaries/DotNET/UnrealBuildTool.exe Development Win64 -Project="D:/....UNREAL/Naptin/Naptin/Naptin.uproject" -TargetType=Editor -Progress -NoEngineChanges -NoHotReloadFromIDE
Using 'git status' to determine working set for adaptive non-unity build (D:\....UNREAL\Naptin\Naptin).
Creating makefile for NaptinEditor(no existing makefile)
#progress push 5%
Parsing headers for NaptinEditor
Running UnrealHeaderTool "D:\....UNREAL\Naptin\Naptin\Naptin.uproject" "D:\....UNREAL\Naptin\Naptin\Intermediate\Build\Win64\NaptinEditor\Development\NaptinEditor.uhtmanifest" -LogCmds="loginit warning, logexit warning, logdatabase error" -Unattended -WarningsAsErrors -abslog="C:\Users\adpar\AppData\Local\UnrealBuildTool\Log_UHT.txt" -installed
Reflection code generated for NaptinEditor in 23.9260389 seconds
#progress pop
ERROR: Unhandled exception: System.UnauthorizedAccessException: Access to the path 'D:\$RECYCLE.BIN\S-1-5-18' is denied.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileSystemEnumerableIterator`1.CommonInit()
at System.IO.FileSystemEnumerableIterator`1..ctor(String path, String originalUserPath, String searchPattern, SearchOption searchOption, SearchResultHandler`1 resultHandler, Boolean checkHost)
at System.IO.DirectoryInfo.EnumerateFiles()
at Tools.DotNETCommon.FileFilter.FindMatchesFromDirectory(DirectoryInfo CurrentDirectory, String NamePrefix, Boolean bIgnoreSymlinks, List`1 MatchingFileNames) in D:\Build\++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\DotNETCommon\DotNETUtilities\FileFilter.cs:line 487
at Tools.DotNETCommon.FileFilter.FindMatchesFromDirectory(DirectoryInfo CurrentDirectory, String NamePrefix, Boolean bIgnoreSymlinks, List`1 MatchingFileNames) in D:\Build\++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\DotNETCommon\DotNETUtilities\FileFilter.cs:line 495
at Tools.DotNETCommon.FileFilter.FindMatchesFromDirectory(DirectoryInfo CurrentDirectory, String NamePrefix, Boolean bIgnoreSymlinks, List`1 MatchingFileNames) in D:\Build\++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\DotNETCommon\DotNETUtilities\FileFilter.cs:line 495
at Tools.DotNETCommon.FileFilter.ApplyToDirectory(DirectoryReference DirectoryName, Boolean bIgnoreSymlinks) in D:\Build\++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\DotNETCommon\DotNETUtilities\FileFilter.cs:line 408
at Tools.DotNETCommon.FilePattern.CreateMapping(HashSet`1 Files, FilePattern& SourcePattern, FilePattern& TargetPattern) in D:\Build\++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\DotNETCommon\DotNETUtilities\FilePattern.cs:line 340
at UnrealBuildTool.UEBuildBinary.PrepareRuntimeDependencies(List`1 RuntimeDependencies, Dictionary`2 TargetFileToSourceFile, DirectoryReference ExeDir) in D:\Build\++UE4\Sync\Engine\Source\Programs\UnrealBuildTool\Configuration\UEBuildBinary.cs:line 294
at UnrealBuildTool.UEBuildTarget.Build(BuildConfiguration BuildConfiguration, ISourceFileWorkingSet WorkingSet, Boolean bIsAssemblingBuild, List`1 SpecificFilesToCompile) in D:\Build\++UE4\Sync\Engine\Source\Programs\UnrealBuildTool\Configuration\UEBuildTarget.cs:line 1786
at UnrealBuildTool.BuildMode.CreateMakefile(BuildConfiguration BuildConfiguration, TargetDescriptor TargetDescriptor, ISourceFileWorkingSet WorkingSet) in D:\Build\++UE4\Sync\Engine\Source\Programs\UnrealBuildTool\Modes\BuildMode.cs:line 615
at UnrealBuildTool.BuildMode.Build(List`1 TargetDescriptors, BuildConfiguration BuildConfiguration, ISourceFileWorkingSet WorkingSet, BuildOptions Options, FileReference WriteOutdatedActionsFile, Boolean bSkipPreBuildTargets) in D:\Build\++UE4\Sync\Engine\Source\Programs\UnrealBuildTool\Modes\BuildMode.cs:line 257
at UnrealBuildTool.BuildMode.Execute(CommandLineArguments Arguments) in D:\Build\++UE4\Sync\Engine\Source\Programs\UnrealBuildTool\Modes\BuildMode.cs:line 226
at UnrealBuildTool.UnrealBuildTool.Main(String[] ArgumentsArray) in D:\Build\++UE4\Sync\Engine\Source\Programs\UnrealBuildTool\UnrealBuildTool.cs:line 550
LogInit: Warning: Still incompatible or missing module: Naptin
LogCore: Engine exit requested (reason: EngineExit() was called)
LogExit: Preparing to exit.
LogExit: Exiting.
Log file closed, 04/17/21 12:02:43
We have tried a few things, like generating again, or copypasting all of our assets to a new project: if we copy the assets, all good, if we copy the code, the same crash.
This happens only on his machine, so I'm kind of at a loss for words. He did not delete any files, the only difference between his machine and mine is that his SDK is more up-to-date by a few weeks.
I'd really appreciate any clue regarding why this file comes up in the build or how to stop it from crashing. The C++ code doesn't seem to be the issue at all, and he can't start Unreal Engine anymore.

Azure Devops - UI tests fail to run on Self hosted agent

While trying to run Coded UI tests on Azure self hosted agent, getting the below error
Agent machine has VS test platform installed and also contains VS2017 build tools to run Coded ui tests.
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.VisualStudio.TestPlatform.Core.ITestRunnerService.InitializeConnection(IEnumerable`1 pathToAdditionalExtensions, Boolean loadOnlyWellKnownExtensions)
at Microsoft.VisualStudio.TestPlatform.Client.TestRunnerServiceProxy.InitializeConnection(IEnumerable`1 pathToAdditionalExtensions, Boolean loadOnlyWellKnownExtensions)
at Microsoft.VisualStudio.TestPlatform.Client.TestRunnerServiceClient.TestConnection(Binding binding, EndpointAddress address, TestRunnerServiceProxy& proxy, Exception& exceptionDuringConnection)
--- End of inner exception stack trace ---
at Microsoft.VisualStudio.TestPlatform.Client.TestRunnerServiceClient.InitializeProxy()
at Microsoft.VisualStudio.TestPlatform.Client.TestRunnerServiceClient.Initialize_NoLock(Boolean forceX86Discoverer)
at Microsoft.VisualStudio.TestPlatform.Client.TestRunnerServiceClient.EnsureInitialized(Boolean forceX86Discoverer, Boolean shouldCheckForRelaunch)
at Microsoft.VisualStudio.TestPlatform.Client.TestRunnerServiceClient.InvokeTestRunnerServiceAction(Boolean forceX86Discoverer, Action`1 action)
at Microsoft.VisualStudio.TestPlatform.Client.TestRunnerServiceClient.DiscoverTests(IEnumerable`1 sources, String settings, Int64 frequencyOfDiscoveredTestEvent, ITestCaseDiscoveryLog testCaseDiscoveryLog, TimeSpan discoveredTestEventTimeout)
at Microsoft.VisualStudio.TestPlatform.Client.Rocksteady.RocksteadyDiscoveryManager.DiscoverTestsForTestRun(DiscoveryCriteria discoveryCriteria, ITestDiscoveryEventsHandler eventHandler)
at Microsoft.VisualStudio.TestPlatform.Client.DiscoveryRequest.Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.IDiscoveryRequest.DiscoverAsync()
at Microsoft.VisualStudio.TestService.VstestAdapter.DiscoverTests.PerformDiscovery(TestAutomationRunSlice sliceDetails, IAddTestCasesToTcmRunHandler addTestCasesToTcmRunHandler, ExecutionStateContext stateModelContext)
at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.FetchPreviousPhaseData(ExecutionStateContext stateModelContext)
at Microsoft.VisualStudio.TestService.VstestAdapter.ExecutionAndPublish.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken)
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()<---
##[error]TestExecution encountered unexpected failures System.AggregateException: One or more errors occurred. ---> Microsoft.VisualStudio.TestPlatform.Common.TestRunner.TestPlatformException: Failed to initialize client proxy: could not connect to test process vstest.discoveryengine.x86.exe. ---> System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at net.pipe://machineA/vstest.discoveryengine/15248 that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.
Specifically, the task that is failing is the Visual Studio Test task (VSTest#2). Immediately before that step, there is a Visual Studio test platform installer task (VisualStudioTestPlatformInstaller#1), which is set up to target the latest test platform available from nuget.
The CodedUI tests are being built using VS2017.
These are the yaml tasks:
- task: VisualStudioTestPlatformInstaller#1
displayName: 'Visual Studio Test Platform Installer'
inputs:
versionSelector: latestStable
- task: VSTest#2
displayName: 'CodedUI Tests'
inputs:
testSelector: 'testPlan'
testPlan: '8369'
testSuite: '8373'
testConfiguration: '13'
searchFolder: 'C:\Testing\bin\Tests'
uiTests: true
runSettingsFile: 'C:\Testing\bin\Tests\Solution Items\OurTestSettings.testsettings'
runTestsInIsolation: true
testRunTitle: 'Nightly Tests'

Changed cmd.exe to run in elevated privileges which always launches Agent.Listener as administrator solved the issue.

Visual Studio 2015 (Update 1) crashes when triggering IntelliSense in cshtml editor

Whenever I open a View file and type something that trigger IntelliSense (<, . or Ctrl + Space), VS immediately crashes without any message and suggest to restart itself.
I opened Event Viewer and see two Error records:
Application: devenv.exe Framework Version: v4.0.30319 Description: The
application requested process termination through
System.Environment.FailFast(string message). Message:
System.AggregateException: One or more errors occurred. --->
System.ArgumentOutOfRangeException: Specified argument was out of the
range of valid values. Parameter name: position at
Microsoft.CodeAnalysis.CSharp.CSharpSyntaxNode.FindToken(Int32
position, Boolean findInsideTrivia) at
Microsoft.CodeAnalysis.CSharp.CSharpSyntaxNode.FindToken(Int32
position, Func`2 findInsideTrivia) at
Microsoft.CodeAnalysis.CSharp.CSharpSyntaxNode.FindToken(Int32
position, Boolean findInsideTrivia) at
Microsoft.CodeAnalysis.CSharp.CSharpSyntaxNode.FindTokenCore(Int32
position, Boolean findInsideTrivia) at
Microsoft.CodeAnalysis.CSharp.Extensions.SyntaxTreeExtensions.IsEntirelyWithinStringLiteral(SyntaxTree
syntaxTree, Int32 position, CancellationToken cancellationToken) at
Microsoft.CodeAnalysis.CSharp.Extensions.SyntaxTreeExtensions.IsInNonUserCode(SyntaxTree
syntaxTree, Int32 position, CancellationToken cancellationToken) at
Microsoft.CodeAnalysis.CSharp.Completion.Providers.SpeculativeTCompletionProvider.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
The full log file is very long, if you need more, please see it here.
I ahve tried resetting user data and delete .vs folder in my project but it does not work. Can anyone suggest a fix? Thank you very much.

I ran into same issue after installing language pack for spanish.
Switching back to international settings to English allowed me to continue working on the cshtml. Must mention that if i switch back to spanish, condition reappears:
So go to:
1. Tools > Options > Environment > International Settings.
2. Select English at Language dropdown.
BTW my Visual Studio versión is the Community one, and haven't tried reinstalling the language pack.
Remember seeing a report a while on MS site (cant't find it lately), hope this helps to shed some light on this.

Continuous Integration build succeeds, Manual build fails

My Team Project consists of a Web Forms application and WCF Services, in two separate solutions (the WCF services are hosted on a server within the domain, the web app is in the DMZ).
I have two build definitions for my Team Project: a CI build and a manual build
Yesterday I merged a branch back into my trunk to prepare for a deployment. When I checked in my merge the CI build kicked off...and succeeded. So then I queued the manual build (the manual build is what ends up on prod server). The manual build failed. It fails everytime I run it now, however the CI build succeeds every time. The error from the build log is pasted below. I don't know how I broke this build, and I'm confused as to why the CI build succeeds but the manual build fails (same build definition except the drop location is different and the trigger is different).
Error:
Exception Message: Access to the path 'C:\Builds\1\My Web App\My Web
App\Sources\MyAppWcfServices\Services\Messages' is denied. (type
UnauthorizedAccessException) Exception Stack Trace: at
System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileSystemEnumerableIterator1.CommonInit() at
System.IO.FileSystemEnumerableIterator1..ctor(String path, String
originalUserPath, String searchPattern, SearchOption searchOption,
SearchResultHandler`1 resultHandler, Boolean checkHost) at
System.IO.Directory.InternalGetFileDirectoryNames(String path, String
userPathOriginal, String searchPattern, Boolean includeFiles, Boolean
includeDirs, SearchOption searchOption, Boolean checkHost) at
System.IO.Directory.InternalGetDirectories(String path, String
searchPattern, SearchOption searchOption) at
Microsoft.TeamFoundation.Common.FileSpec.DeleteDirectoryInternal(String
path) at
Microsoft.TeamFoundation.Common.FileSpec.DeleteDirectoryInternal(String
path) at
Microsoft.TeamFoundation.Common.FileSpec.DeleteDirectoryInternal(String
path) at
Microsoft.TeamFoundation.Common.FileSpec.DeleteDirectoryInternal(String
path) at
Microsoft.TeamFoundation.Common.FileSpec.DeleteDirectory(String path,
Boolean recursive) at
Microsoft.TeamFoundation.Build.Workflow.Activities.DeleteDirectory.Execute(CodeActivityContext
context) at
System.Activities.CodeActivity.InternalExecute(ActivityInstance
instance, ActivityExecutor executor, BookmarkManager bookmarkManager)
at
System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor
executor, BookmarkManager bookmarkManager, Location resultLocation)
Things I've read/tried:
TFS 2012 Build "Access to Path Denied"
Tool to find duplicate copies in a build (I didn't run the tool,
couldn't find a log file named like the example...confused)
I do not have my obj or bin directories in version control.
This build definition has been working great for months, up until yesterday. I'm not sure what happened when I merged that would've caused this. There were no conflicts in my merge, it was easy peasy....until I tried to build.

just go to the Build controller server and stop the "Visual Studio Team Foundation Build Service Host 2013" and delete the Files or folder which you have the error in your message and start the service again

Have you checked if the actual file is locked? try a tool like unlocker http://www.emptyloop.com/unlocker

I have seen this error happening when a particular folder/file is opened in iexplorer/cmd.exe etc (either by you or some other user). When TFS tries to clean up the directory and get the latest file, it will fail as it is used by another process.
If you have access to the server, then open the taskmanager -> process and close all the iexplorer.exe/cmd.exe process from other users. Hopefully that will fix it.