We Keep Coding

Two identical Azure deployments; one always throws 404 - how to find out what the difference is?

I am deploying a C# ASP.NET Core web service to Azure using Pulumi. I can deploy it in 3 ways:
Run it locally from Visual Studio, i.e., not using Azure at all.
Deploy it to Azure from my local developer computer.
Deploy it to Azure from Jenkins (whicn runs on a different computer).
I have this problem:
When I run it locally, I can call the service fine, e.g. from Postman or from a C# application. The web service returns what I expect.
When I deploy it to Azure from my local machine, I can also call it fine. The web service returns what I expect.
When I deploy it to Azure from Jenkins and then try to call the webservice, it returns "NotFound" to all calls no matter what I do. (This presumably means HTTP 404.)
The deployments in 2 and 3 should be exactly the same. My question is: How can I find out what the difference is between these two deployments in Azure?
The Jenkins-deployed webservice exhibits the following curious behaviour:
It does not log any exceptions (even when I wait several minutes for them to show up).
If I go to my resource group -> Application Insights -> Logs and search for "requests", it does list requests. Curiously, it says that it returned HTTP 200 to all the requests, even though what I get when calling them is 404.
The above is true even for web service calls that should never return 200 (they should return 201).
The above is true even for web service calls to methods that shouldn't even exist (i.e., when I deliberately corrupt the method URI before calling the service).
During deployment I authenticate with Azure using a service principal. My Jenkinsfile looks like this:
withVaultSecrets([
"path/to/secret/in/vault": [
"sp_name", "application_id", "object_id", "sp_secret"
]
]){
script {
env.PULUMI_CONFIG_PASSPHRASE = 'jenkinspassphrase'
env.ARM_CLIENT_ID = "${application_id}"
env.ARM_CLIENT_SECRET = "${sp_secret}"
env.ARM_TENANT_ID = "${azure_dev_tenant_id}"
env.ARM_SUBSCRIPTION_ID = "${azure_dev_subscription_id}"
env.AZURE_CLIENT_ID = "${application_id}"
env.AZURE_CLIENT_SECRET = "${sp_secret}"
env.AZURE_TENANT_ID = "${azure_dev_tenant_id}"
}//script
dir("./src/deploy/KmsStack"){
powershell "pulumi login --local";
powershell "pulumi stack init jenkinsfunctionaltest --secrets-provider=passphrase"
powershell "pulumi up --yes"
}//dir
}//withVaultSecrets
The script which I use to deploy locally looks like this, with the same service principal credentials:
cd $PSScriptRoot
cd webapi
dotnet publish /p:DisableGitVersionTask=true
cd ../deploy/KmsStack
$env:PULUMI_CONFIG_PASSPHRASE = 'jenkinspassphrase'
$env:ARM_CLIENT_ID = ...
$env:ARM_CLIENT_SECRET = ...
$env:ARM_TENANT_ID = ...
$env:ARM_SUBSCRIPTION_ID = ...
$env:AZURE_CLIENT_ID = ...
$env:AZURE_CLIENT_SECRET = ...
$env:AZURE_TENANT_ID = ...
pulumi logout
pulumi login --local
pulumi stack rm jenkinsfunctionaltest -y
pulumi stack init jenkinsfunctionaltest --secrets-provider=passphrase
pulumi stack select jenkinsfunctionaltest
pulumi up --yes
How can I find out why these two deployed services behave differently? The Azure portal GUI is rich and has lots of sections. Can you recommend me where to look? Might there be some security settings that differ? How can I find them?
Thanks in advance!

We found out what was wrong. It was not an Azure issue. The problem was that we were deploying a bad ZIP file. The ZIP file was missing web.config, which meant that the web application could not start up.
We were zipping our published web application by having this in the CSPROJ file:
<Target Name="ZipOutputPath" AfterTargets="Publish">
<ZipDirectory SourceDirectory="$(OutputPath)\publish" DestinationFile="$(MSBuildProjectDirectory)\kmswebapp.zip" Overwrite="true" />
</Target>
This turned out not to work because the compiler does things in a different order than we expected. At the time when it generated the ZIP file, web.config was not generated yet, so web.config never got packed into the ZIP file. Hence Azure could not start the application.
When we deployed from our local machines, it worked because we didn't clean the publish directory before each run, so there would be a web.config left over from the previous run, and this old (but unchanged) web.config would get packed into the ZIP file and deployed to Azure, so Azure would know how to start the application.
We solved it by removing the above from our CSPROJ file and doing (roughly) this in our Jenkinsfile:
powershell "dotnet publish ./src/webapi/WebAPI.csproj"
powershell "if (!(Test-Path('${publishDirectoryPath}/web.config'))){throw 'We need web.config to exist in the publish directory'}"
powershell "Compress-Archive -Path '${publishDirectoryPath}/*' -DestinationPath './src/webapi/kmswebapp.zip' -Force"
This generates a proper ZIP file including web.config, and Azure can now start our application so it can respond properly to requests.

Unable to start an NServiceBus Windows Service

Problem Description
I have a Windows service which is hosting an NServiceBus endpoint in NServiceBus.Host.exe.
The binaries are deployed to c:\inetpub\bus\services\myService folder on the server.
A DSC script makes sure the Windows service is created/exists on the server, with the "path to executable" property of the service set to "c:\inetpub\bus\services\myService´NServiceBus.Host.exe" -service NServicebus.Production.
Note! The -service switch is added when installing the service by using the built-in NServiceBus.Host.exe /install parameter, which is why I added it to the Windows service executable path in the DSC script.
Now, when I try to start the service manually on the server, it yields the following error message
Windows could not start the <service name> service on the Local Computer.
Error 1053: The service did not respond to the start or control request in a timely fashion.
Debugging Steps
I have looked through the event log and the following two error messages sticks out:
NServiceBust.Host.exe error:
Application: NServiceBus.Host.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info:
Topshelf.Exceptions.ConfigurationException
at Topshelf.Internal.Actions.RunAsServiceAction.Do(Topshelf.Configuration.IRunConfiguration)
at Topshelf.Runner.Host(Topshelf.Configuration.IRunConfiguration, System.String[])
at NServiceBus.Host.Program.Main(System.String[])
Local Activation permission error:
The application-specific permission settings do not grant Local
Activation permission for the COM Server application with CLSID
{D63B10C5-BB46-4990-A94F-E40B9D520160}
and APPID
{9CA88EE3-ACB7-47C8-AFC4-AB702511C276}
to the user <my_service_account_user> SID (<service_account_SID>) from
address LocalHost (Using LRPC) running in the application container
Unavailable SID (Unavailable). This security permission can be modified
using the Component Services administrative tool.`
Note! The error above only occurs once, i.e. the first time I try to start the service. It does not appear again in the event log for any subsequent attempts of starting the service.
What I have done so far:
Tried the suggestions in a closely related post here on SO, none of which were working.
Tried to install the service by using using the NServiceBus.Host.exe /install parameter. In this case, the service name is created with its name on the following format: MyService.EndpointConfig_v1.0.0.0. Using this approach, the service starts successfully without any error message
Stopping the service and then try to start the service created by the DSC script (with a different name) => success
Removing the service created by NServiceBus and then trying to start the DSC-created service again => failure
Tried granting the service account used for logon when running the service various privileges (neither of which yielded any success), among others:
Membership in the Administrators group
Membership in the Performance Log Users group
Full DCOM permissions via "Launch and Activation Permissions" in dcomcnfg
Tried running c:\inetpub\bus\services\myService´NServiceBus.Host.exe NServicebus.Production from the CLI => success
Code
My Init() method for the service looks like this:
namespace MyService
{
public class EndpointConfig : IConfigureThisEndpoint, AsA_Server, IWantCustomLogging, IWantCustomInitialization
{
public void Init()
{
Directory.SetCurrentDirectory(System.AppDomain.CurrentDomain.BaseDirectory);
SetLoggingLibrary.Log4Net(() => XmlConfigurator.Configure(File.OpenRead(#"log4net.config")));
GlobalContext.Properties["Hostname"] = Dns.GetHostName();
GlobalContext.Properties["Service"] = typeof(EndpointConfig).Namespace;
var container = new WindsorContainer(new XmlInterpreter());
Configure.With()
.CastleWindsorBuilder(container)
.XmlSerializer()
.MsmqTransport()
.IsTransactional(true)
.PurgeOnStartup(false)
.IsolationLevel(System.Transactions.IsolationLevel.RepeatableRead);
var connectionString = ConfigurationManager.ConnectionStrings["<some_conn_string>"].ConnectionString;
container.Register(
Component.For<IDatabaseProvider>()
.ImplementedBy<DatabaseProvider>()
.DependsOn(Property.ForKey("connectionString").Eq(connectionString)));
}
}
}
Theory
When installing the service using /install I assume that NServiceBus.Host.exe does some black magic under the hood - e.g. grants some necessary permissions - to make the service able to start.
Note! On the server, the latest version of NServiceBus is installed (v6.x). However, in my solution/service project version 2.x is used (please, do not ask if I can upgrade - unfortunately, that is not an option).
Appreciate any help I can get, as I am running out of ideas.
EDIT 1
I was asked why I can't just use the /install parameter of NServiceBus and be happy with that. The answer to that is that I could (and, actually, I currently am).
The reason I have still posted this question is split:
I wish to understand why one of two seemingly equivalent approaches fails
I am not completely happy with using the /install parameter. The reason? It boils down to a "chicken or the egg" problem. I use Powershell DSC to provision servers in Azure and I believe that ensuring that Windows Services exists on the server is the responsibility of DSC. However, the first time a server is provisioned the services cannot exist unless I script their creation with DSC, and point the executable path to where the service binaries will be deployed whenever that happens. The other alternative is to skip service creation in DSC, and run the NServiceBus.Host.exe /install as a part of the service/application deployment script instead. Obviously, deployment cannot happen until after a server has been provisioned. Thus, it requires the Windows Service part of the DSC script being stripped down to e.g. merely ensuring the service exist - a verification which will fail until a first time deployment of the application has been performed.

TFS WorkItemStore throws COMException in ASP.NET MVC app when using Authentication

I'm completely stumped by this and endless google/stackoverflow searches haven't helped. I'm using the 2012 Visual Studio SDK to connect to TFS 2012 and query the work item store. The code below works perfectly fine in both a console application and an ASP.NET MVC app that does not use authentication, or in any scenario that's run on my local machine. However, I get a COMException when I try to instantiate a WorkItemStore from within an MVC app that's been deployed to the server and uses (Windows) authentication.
It makes no difference if I have the <authentication mode="Windows" /> element in my web.config or not; as long as there is an [Authorize] attribute on my controller or any of its action methods, I get an exception as soon as the last line of code below is invoked. If I remove the [Authorize] attribute, the exception does not occur. If I call the code below at some point before calling code decorated with [Authorize], the exception does not occur. Somehow, using the AuthorizeAttibute is inducing this exception.
Any ideas of how to resolve this? Or at least for more exactly identifying the real root problem? I'd really like to understand what is going on here.
Uri tfsAddress = new Uri("http://tfs-address:8080/tfs/DefaultCollection");
var myCreds = new NetworkCredential("userName", "password", "domain");
var tfsCreds = new TfsClientCredentials(new WindowsCredential(myCreds), false);
var defaultCollection = new TfsTeamProjectCollection(tfsAddress, tfsCreds);
defaultCollection.EnsureAuthenticated();
var store = defaultCollection.GetService<WorkItemStore>(); // <-- EXCEPTION
Stack trace:
[COMException (0x80004005): Error HRESULT E_FAIL has been returned from a call to a COM component.]
Microsoft.TeamFoundation.WorkItemTracking.Client.DataStore.DataStoreNative.BeginDataStoreInit(IntPtr handle, String defaultCachePath, String instanceId, Int32 cacheVersion) +0
Microsoft.TeamFoundation.WorkItemTracking.Client.DataStore.Datastore.BeginDataStoreInit(String defaultCachePath, String instanceId, Int32 cacheVersion) +56
Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.InitializeInternal() +598
Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore.Microsoft.TeamFoundation.Client.ITfsTeamProjectCollectionObject.Initialize(TfsTeamProjectCollection teamProjectCollection) +23
Microsoft.TeamFoundation.Client.TfsTeamProjectCollection.InitializeTeamFoundationObject(String fullName, Object instance) +43
Microsoft.TeamFoundation.Client.TfsConnection.CreateServiceInstance(Assembly assembly, String fullName) +91
Microsoft.TeamFoundation.Client.TfsConnection.GetServiceInstance(Type serviceType, Object serviceInstance) +200
Microsoft.TeamFoundation.Client.TfsTeamProjectCollection.GetServiceInstance(Type serviceType, Object serviceInstance) +439
Microsoft.TeamFoundation.Client.TfsConnection.GetService(Type serviceType) +241
Microsoft.TeamFoundation.Client.TfsConnection.GetService() +58

The solution was to change the identity of the application's AppPool to either a domain account or to LocalSystem.
I can't say I fully understand what is going on behind the scenes here, but it seems there was some kind of conflict between the authenticated identity of the user browsing the site and the AppPool identity, which was originally the default ApplicationPoolIdentity. If the root cause of the problem is unauthorized access or something similar, it seems to me that I should have gotten the same exception for anonymous users as well as authenticated ones. I'd also expect an actual Unauthorized Access Exception, not an 'undefined' COMException, so I really don't know what the real root cause is. If someone comes across this question and cares to elaborate, please do so.

For me this error appears under certain situations, like when I created a new site in local IIS which uses the TFS API. I even followed the comments here to no avail, I had the AppPool Identity set to LocalSystem. I had a separate app pool. I had 32bit applications enabled for the TFS API.
For me the problem ceased when I changed the app pool from framework v4.0 to v2.0 and then back to v4.0.
I don't know why it fixed the issue, but it worked for me.

The accepted answer worked for me, but didn't seem optimal. I was able to fix this by also deleting the below cache directories and/or giving the application pool user permissions to them.
Windows 7:
C:\ProgramData\Microsoft Team Foundation
Windows 8.1:
C:\ProgramData\Microsoft\Team Foundation
These folders seem to be created whenever my applications utilize the TFS SDK.

Check this response
http://social.msdn.microsoft.com/Forums/vstudio/en-US/e4c84467-309a-45bd-9da8-973732897440/problem-with-microsoftteamfoundationworkitemtrackingclientdatastore?forum=tfsgeneral (Read All)
http://blogs.msdn.com/b/buckh/archive/2012/02/01/listing-the-work-items-associated-with-changesets-for-a-path.aspx

how to host silverlight project local

i want to debug my program but i can't host my silverlight project in VS 2010
here is the message that the VS send to me:
"the Silverlight project you are about to debug uses web service. Call to the web service will fail unless the Silverlight is host and launched from the same web project contains the web service"
when I search about it in the web i saw that the problem is that I'm not doing it local
so when I tried to change this line
endpoint address="http://xxx.xx.x.x/WebService/Service1.svc"
to
endpoint address="http://localhost/WebService/Service1.svc"
on this file:
ServiceReferences.ClientConfig
update!!
i tried now to do it like this:
endpoint address="http://localhost:54502/Service1.svc"
and i got and error that says:
the remote server returned an error:NotFound
here:
public int EndAddParticipant(System.IAsyncResult result) {
object[] _args = new object[0];
int _result = ((int)(base.EndInvoke("AddParticipant", _args, result)));
return _result;
what should i do to change it?
i saw that i need to turn on the debug in web.config file but it was already on.

You didn't explain well so here are a bunch of answers:
When you created your solution, you should have told it to host in a web site.
You can do this after the fact but it is a lot of work (for a new guy), you're better off starting over and copying your code from your silverlight project.
If you did but you are trying to connect to another web site as your web service, you will have to implement a Cross Domain configuration file. This opens a big security hole though.
If you have the 2 projects, but your issue is just the "ServiceReferences.ClientConfig" file not pointing the the right server, you need to include the PORT when debugging locally. It should look something like this:
http://localhost:12345/Service1.svc
Where 12345 is the port of your local web service. You can find this out by looking at the "Web" tab of your project properties.
Hope one of these does it. If not, please provide a little more info, like all the important stuff behind the phrase, "it didn't work"!

How do I properly host a WCF Data Service that connects to SQLServer in IIS? Why am I getting errors?

I'm playing around with WCF Data Services (ADO.NET Data Services). I have an entity framework model pointed at the AdventureWorks database.
When I debug my svc file from within Visual Studio, it works great. I can say /awservice.svc/Customers and get back the ATOM feed I expect.
If I publish the service (hosted in an ASP.NET web application) to IIS7, the same query string returns a 500 fault. The root svc page itself works as expected and successfully returns ATOM. The /Customers path fails.
Here is what my grants look like in the svc file:
public class AWService : DataService<AWEntities>
{
public static void InitializeService( DataServiceConfiguration config )
{
config.SetEntitySetAccessRule( "*", EntitySetRights.All );
config.SetServiceOperationAccessRule( "*", ServiceOperationRights.All );
config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
}
}
Update: I enabled verbose errors and get the following in the XML message:
<innererror>
<message>The underlying provider failed on Open.</message>
<type>System.Data.EntityException</type>
<stacktrace>
at System.Data.EntityClient.EntityConnection.OpenStoreConnectionIf(
...
...
<internalexception>
<message>
Login failed for user 'IIS APPPOOL\DefaultAppPool'.
</message>
<type>System.Data.SqlClient.SqlException</type>
<stacktrace>
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, ...

It looks to me like this is a SQL authentication error, IIS is running its appPool under a user that does not have access to your SQL server, when you ruin in Visual Studio (locally) it will be a different user. Check the user that the IIS on the server is using and make sure it has rights to do what you want in SQL.

Try to change the connection string attribute Integrated security to False

Quick solution with IIS Express
Create a firewall exception to allow HTTP requests through the firewall on the port that IIS Express is using.
Get the IP address of the development computer, if necessary, by running ipconfig.
Find the IIS Express configuration file, applicationhost.config, in the folder %USERPROFILE%\Documents\IISExpress\config. The USERPROFILE environment variable typically has a value of C:\Users\.
Open applicationhost.config with Notepad or another text editor and make the following changes.
Find the site element for the web service, WebServiceForTesting.
If you don’t see the site element for the web service, you have to deploy the service at least one time to create the element.
Within the bindings section of the site element, copy the binding element and paste a copy directly below the existing binding element to create a second binding.
In the new binding element, replace localhost with the computer’s IP address.
Save the changes.
Run Visual Studio as administrator and open the Visual Studio solution.
In the phone app project, remove the service reference to the service if you have previously added it.
Add a new service reference to the reconfigured web service. In the Add Service Reference dialog box, in the Address box, replace localhost with the IP address of your development computer. Click Go.
The second binding for the service in the WCF project is discovered and displayed. Click OK.
A new service reference that uses the IP address of the development computer is added to the Windows Phone project.