We Keep Coding

Two identical Azure deployments; one always throws 404 - how to find out what the difference is?

I am deploying a C# ASP.NET Core web service to Azure using Pulumi. I can deploy it in 3 ways:
Run it locally from Visual Studio, i.e., not using Azure at all.
Deploy it to Azure from my local developer computer.
Deploy it to Azure from Jenkins (whicn runs on a different computer).
I have this problem:
When I run it locally, I can call the service fine, e.g. from Postman or from a C# application. The web service returns what I expect.
When I deploy it to Azure from my local machine, I can also call it fine. The web service returns what I expect.
When I deploy it to Azure from Jenkins and then try to call the webservice, it returns "NotFound" to all calls no matter what I do. (This presumably means HTTP 404.)
The deployments in 2 and 3 should be exactly the same. My question is: How can I find out what the difference is between these two deployments in Azure?
The Jenkins-deployed webservice exhibits the following curious behaviour:
It does not log any exceptions (even when I wait several minutes for them to show up).
If I go to my resource group -> Application Insights -> Logs and search for "requests", it does list requests. Curiously, it says that it returned HTTP 200 to all the requests, even though what I get when calling them is 404.
The above is true even for web service calls that should never return 200 (they should return 201).
The above is true even for web service calls to methods that shouldn't even exist (i.e., when I deliberately corrupt the method URI before calling the service).
During deployment I authenticate with Azure using a service principal. My Jenkinsfile looks like this:
withVaultSecrets([
"path/to/secret/in/vault": [
"sp_name", "application_id", "object_id", "sp_secret"
]
]){
script {
env.PULUMI_CONFIG_PASSPHRASE = 'jenkinspassphrase'
env.ARM_CLIENT_ID = "${application_id}"
env.ARM_CLIENT_SECRET = "${sp_secret}"
env.ARM_TENANT_ID = "${azure_dev_tenant_id}"
env.ARM_SUBSCRIPTION_ID = "${azure_dev_subscription_id}"
env.AZURE_CLIENT_ID = "${application_id}"
env.AZURE_CLIENT_SECRET = "${sp_secret}"
env.AZURE_TENANT_ID = "${azure_dev_tenant_id}"
}//script
dir("./src/deploy/KmsStack"){
powershell "pulumi login --local";
powershell "pulumi stack init jenkinsfunctionaltest --secrets-provider=passphrase"
powershell "pulumi up --yes"
}//dir
}//withVaultSecrets
The script which I use to deploy locally looks like this, with the same service principal credentials:
cd $PSScriptRoot
cd webapi
dotnet publish /p:DisableGitVersionTask=true
cd ../deploy/KmsStack
$env:PULUMI_CONFIG_PASSPHRASE = 'jenkinspassphrase'
$env:ARM_CLIENT_ID = ...
$env:ARM_CLIENT_SECRET = ...
$env:ARM_TENANT_ID = ...
$env:ARM_SUBSCRIPTION_ID = ...
$env:AZURE_CLIENT_ID = ...
$env:AZURE_CLIENT_SECRET = ...
$env:AZURE_TENANT_ID = ...
pulumi logout
pulumi login --local
pulumi stack rm jenkinsfunctionaltest -y
pulumi stack init jenkinsfunctionaltest --secrets-provider=passphrase
pulumi stack select jenkinsfunctionaltest
pulumi up --yes
How can I find out why these two deployed services behave differently? The Azure portal GUI is rich and has lots of sections. Can you recommend me where to look? Might there be some security settings that differ? How can I find them?
Thanks in advance!

We found out what was wrong. It was not an Azure issue. The problem was that we were deploying a bad ZIP file. The ZIP file was missing web.config, which meant that the web application could not start up.
We were zipping our published web application by having this in the CSPROJ file:
<Target Name="ZipOutputPath" AfterTargets="Publish">
<ZipDirectory SourceDirectory="$(OutputPath)\publish" DestinationFile="$(MSBuildProjectDirectory)\kmswebapp.zip" Overwrite="true" />
</Target>
This turned out not to work because the compiler does things in a different order than we expected. At the time when it generated the ZIP file, web.config was not generated yet, so web.config never got packed into the ZIP file. Hence Azure could not start the application.
When we deployed from our local machines, it worked because we didn't clean the publish directory before each run, so there would be a web.config left over from the previous run, and this old (but unchanged) web.config would get packed into the ZIP file and deployed to Azure, so Azure would know how to start the application.
We solved it by removing the above from our CSPROJ file and doing (roughly) this in our Jenkinsfile:
powershell "dotnet publish ./src/webapi/WebAPI.csproj"
powershell "if (!(Test-Path('${publishDirectoryPath}/web.config'))){throw 'We need web.config to exist in the publish directory'}"
powershell "Compress-Archive -Path '${publishDirectoryPath}/*' -DestinationPath './src/webapi/kmswebapp.zip' -Force"
This generates a proper ZIP file including web.config, and Azure can now start our application so it can respond properly to requests.

Unable to start an NServiceBus Windows Service

Problem Description
I have a Windows service which is hosting an NServiceBus endpoint in NServiceBus.Host.exe.
The binaries are deployed to c:\inetpub\bus\services\myService folder on the server.
A DSC script makes sure the Windows service is created/exists on the server, with the "path to executable" property of the service set to "c:\inetpub\bus\services\myService´NServiceBus.Host.exe" -service NServicebus.Production.
Note! The -service switch is added when installing the service by using the built-in NServiceBus.Host.exe /install parameter, which is why I added it to the Windows service executable path in the DSC script.
Now, when I try to start the service manually on the server, it yields the following error message
Windows could not start the <service name> service on the Local Computer.
Error 1053: The service did not respond to the start or control request in a timely fashion.
Debugging Steps
I have looked through the event log and the following two error messages sticks out:
NServiceBust.Host.exe error:
Application: NServiceBus.Host.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info:
Topshelf.Exceptions.ConfigurationException
at Topshelf.Internal.Actions.RunAsServiceAction.Do(Topshelf.Configuration.IRunConfiguration)
at Topshelf.Runner.Host(Topshelf.Configuration.IRunConfiguration, System.String[])
at NServiceBus.Host.Program.Main(System.String[])
Local Activation permission error:
The application-specific permission settings do not grant Local
Activation permission for the COM Server application with CLSID
{D63B10C5-BB46-4990-A94F-E40B9D520160}
and APPID
{9CA88EE3-ACB7-47C8-AFC4-AB702511C276}
to the user <my_service_account_user> SID (<service_account_SID>) from
address LocalHost (Using LRPC) running in the application container
Unavailable SID (Unavailable). This security permission can be modified
using the Component Services administrative tool.`
Note! The error above only occurs once, i.e. the first time I try to start the service. It does not appear again in the event log for any subsequent attempts of starting the service.
What I have done so far:
Tried the suggestions in a closely related post here on SO, none of which were working.
Tried to install the service by using using the NServiceBus.Host.exe /install parameter. In this case, the service name is created with its name on the following format: MyService.EndpointConfig_v1.0.0.0. Using this approach, the service starts successfully without any error message
Stopping the service and then try to start the service created by the DSC script (with a different name) => success
Removing the service created by NServiceBus and then trying to start the DSC-created service again => failure
Tried granting the service account used for logon when running the service various privileges (neither of which yielded any success), among others:
Membership in the Administrators group
Membership in the Performance Log Users group
Full DCOM permissions via "Launch and Activation Permissions" in dcomcnfg
Tried running c:\inetpub\bus\services\myService´NServiceBus.Host.exe NServicebus.Production from the CLI => success
Code
My Init() method for the service looks like this:
namespace MyService
{
public class EndpointConfig : IConfigureThisEndpoint, AsA_Server, IWantCustomLogging, IWantCustomInitialization
{
public void Init()
{
Directory.SetCurrentDirectory(System.AppDomain.CurrentDomain.BaseDirectory);
SetLoggingLibrary.Log4Net(() => XmlConfigurator.Configure(File.OpenRead(#"log4net.config")));
GlobalContext.Properties["Hostname"] = Dns.GetHostName();
GlobalContext.Properties["Service"] = typeof(EndpointConfig).Namespace;
var container = new WindsorContainer(new XmlInterpreter());
Configure.With()
.CastleWindsorBuilder(container)
.XmlSerializer()
.MsmqTransport()
.IsTransactional(true)
.PurgeOnStartup(false)
.IsolationLevel(System.Transactions.IsolationLevel.RepeatableRead);
var connectionString = ConfigurationManager.ConnectionStrings["<some_conn_string>"].ConnectionString;
container.Register(
Component.For<IDatabaseProvider>()
.ImplementedBy<DatabaseProvider>()
.DependsOn(Property.ForKey("connectionString").Eq(connectionString)));
}
}
}
Theory
When installing the service using /install I assume that NServiceBus.Host.exe does some black magic under the hood - e.g. grants some necessary permissions - to make the service able to start.
Note! On the server, the latest version of NServiceBus is installed (v6.x). However, in my solution/service project version 2.x is used (please, do not ask if I can upgrade - unfortunately, that is not an option).
Appreciate any help I can get, as I am running out of ideas.
EDIT 1
I was asked why I can't just use the /install parameter of NServiceBus and be happy with that. The answer to that is that I could (and, actually, I currently am).
The reason I have still posted this question is split:
I wish to understand why one of two seemingly equivalent approaches fails
I am not completely happy with using the /install parameter. The reason? It boils down to a "chicken or the egg" problem. I use Powershell DSC to provision servers in Azure and I believe that ensuring that Windows Services exists on the server is the responsibility of DSC. However, the first time a server is provisioned the services cannot exist unless I script their creation with DSC, and point the executable path to where the service binaries will be deployed whenever that happens. The other alternative is to skip service creation in DSC, and run the NServiceBus.Host.exe /install as a part of the service/application deployment script instead. Obviously, deployment cannot happen until after a server has been provisioned. Thus, it requires the Windows Service part of the DSC script being stripped down to e.g. merely ensuring the service exist - a verification which will fail until a first time deployment of the application has been performed.

ASP.NET Web Application - on deploy, System.Speech.dll object not getting set to an instance of an object

I have an ASP.NET web application that uses System.Speech to transform text to a WAV file. It works fine locally but when I deploy it to the server, I get the below error message. This is using Windows Server 2012, ASP.NET 4.5, and IIS 8.5:
Object reference not set to an instance of an object.
System.Speech
at System.Speech.Internal.ObjectTokens.RegistryDataKey..ctor(String fullPath, RegistryDataKey copyKey)
at System.Speech.Internal.ObjectTokens.SAPICategories.DefaultDeviceOut()
at System.Speech.Internal.Synthesis.VoiceSynthesis..ctor(WeakReference speechSynthesizer)
at System.Speech.Synthesis.SpeechSynthesizer.get_VoiceSynthesizer()
at QuinnSDS.handlerTransform.<>c__DisplayClass6.<ProcessRequest>b__1()
The code which is generating this error message runs on the server:
if (context.Request.ContentLength > 0)
{
string line = new StreamReader(context.Request.InputStream).ReadToEnd();
// ********* generate wav file voicing the response *****************
// Using Microsoft voices
// initiate new instance of speech synthesizer
Thread t = new Thread(() =>
{
try
{
// The object creation works fine
System.Speech.Synthesis.SpeechSynthesizer synth = new System.Speech.Synthesis.SpeechSynthesizer();
if (synth != null)
{
// The code breaks at synth.GetInstalledVoices() below. It will break any time I try to do anything with the synth object
foreach (System.Speech.Synthesis.InstalledVoice voice in synth.GetInstalledVoices())
{
System.Speech.Synthesis.VoiceInfo info = voice.VoiceInfo;
string voiceName = info.Name;
ws.WriteLine(voiceName);
}
}
}
catch (Exception e)
{
ws.WriteLine(e.Message);
ws.WriteLine(e.Source);
ws.WriteLine(e.StackTrace);
}
//... code continues...
It does not break when the Speech Synthesis object is created; it breaks whenever I try to use that object in any way.
I'm not sure if it's an access issue but I'm pretty new to ASP.NET and IIS and I can't figure out how to give the web app access to the GAC or if that's even what the problem is. I tried changing the property Local Copy for the System.Speech reference to True in Visual Studio, before I deploy the app, but that hasn't worked. I searched online and while the "object reference not set to an instance of an object" seems fairly common, I cannot find any similar issues where it is because of a .NET framework class library...I have run the text-to-speech code locally on the server and it ran fine. I have not run the entire app locally on the server because the web app requires speech input and there is not a microphone on the server.
Any ideas of anything to try would be most welcome!

What user account is the code running under when executed from ASP.NET? If the Speech API is touching the registry like the call stack suggests, it possibly has different permissions than the account you used to run the code manually.
If you can't just make the application pool for your site run with the same account you log into the machine with, I've had some success using Process Monitor to track down this kind of problem before. Basically, execute the code that fails while Process Monitor is running and look for 'ACCESS DENIED' in the 'Result' column (or anything else that looks suspicious). Quickly switching the application pool to use your standard user account will be the fastest way to rule out security or permission related problems though.

.Net's Directory Services throws a strange exception

I have a small C# solution used to check users credentials. It works fine for two of my teammates, but on my PC I get an exception.
The relevant code:
PrincipalContext context = new PrincipalContext(ContextType.Domain);
if (context.ValidateCredentials(System.Environment.UserDomainName + "\\" + usr, pwd))
return true;
else
return false;
And the exception is:
DirectoryOperationException, "The server cannot handle directory requests.".
I tried creating context with the explicit server name and the 636 port number, but this didn't help as well.
Any ideas?

I had this problem too using IIS Express and VS 2010. What fixed it for me was a comment on another thread.
Validate a username and password against Active Directory?
but i'll save you the click and search... :) Just add ContextOpations.Negotiate to you Validate Credentials call like below.
bool valid = context.ValidateCredentials(user, pass, ***ContextOptions.Negotiate***);

I had this issue: things were working on my dev machine but didn't work on the server. Turned out that IIS on the server was set up to run as LocalMachine. I changed it to NetworkService (the default) and things started working.
So basically check the user of the app pool if this is running on IIS.

I had to just create a new app pool and assign it .NET 2.0, then assign the new app pool to our web app, and it started working. We had .NET 3.5 SP2, so the hotfix wasn't ideal for us. Since the WWW service is usually Local System, I questioned that too. But since it was .NET and security related, I gave a shot at the app pool first and it worked.

Perhaps you need the hotfix?
FIX: DirectoryOperationException exception
And you are an Admin or the id that your service is running under is an Admin on your PC right?
I take it you already looked into this:
System.DirectoryServices.Protocols
"You may receive a less than helpful DirectoryOperationException(“The server cannot handle directory requests.”) what isn’t quite so amusing about this is that it didn’t even try to communicate with the server. The solution was to add the port number to the server. So instead of passing “Server” to open the LdapConnection, I passed “server:636”. By the way, LDAPS is port 636 – rather than the 389 port used by LDAP."
Good point, I wouldn't expect that Win7/.NET 3.5 would need that patch. How about the info provided in this question:
Setting user's password via System.DirectoryServices.Protocols in AD 2008 R2

Control IIS 7 server from Windows 2003 server programmatically

We run various jobs using a Windows 2003 server. Some of these jobs send app pool commands to web servers running IIS 6 (recycle, start, stop). Now we have a Windows 2008 web server running IIS 7, and we want to send the same commands. This is all done using C#.
This is the code we use to send commands for IIS 6:
var methodToInvoke = "Stop"; // could be "Stop", "Start", or "Recycle"
var co = new ConnectionOptions
{
Impersonation = ImpersonationLevel.Impersonate,
Authentication = AuthenticationLevel.PacketPrivacy
};
var objPath = string.Format("IISApplicationPool.Name='W3SVC/AppPools/{0}'", appPoolName);
var scope = new ManagementScope(string.Format(#"\\{0}\root\MicrosoftIISV2", machineName), co);
using (var mc = new ManagementObject(objPath))
{
mc.Scope = scope;
mc.InvokeMethod(methodToInvoke, null, null);
}
This code doesn't work for IIS 7 due to underlying changes, so we're currently trying this:
using (ServerManager serverManager = ServerManager.OpenRemote(machineName))
{
var appPool = serverManager.ApplicationPools[appPoolName];
if (appPool != null)
{
appPool.Stop(); // or app.Start() or app.Recycle()
serverManager.CommitChanges();
}
}
The above code works fine on my workstation, which runs Windows 7 (and, thus, IIS 7.5). However, it does not work when I deploy this code to our application server. It get this error:
System.InvalidCastException:
Unable to cast COM object of type 'System.__ComObject' to interface type
'Microsoft.Web.Administration.Interop.IAppHostWritableAdminManager'.
This operation failed because the QueryInterface call on the COM component for the
interface with IID '{FA7660F6-7B3F-4237-A8BF-ED0AD0DCBBD9}' failed due to the following error:
Interface not registered (Exception from HRESULT: 0x80040155).
From my research, this is due to the fact that IIS 7 is not available on the Windows Server 2003 server. (I did include the Microsoft.Web.Administration.dll file.)
So my questions are:
Is it possible for the above code for IIS 7 to work at all from a Windows 2003 server?
If no to #1, is there a better way of doing this?

From reading around it doesn't appear to be possible to do what you're looking for. It's not enough to include the dll files.
According to http://forums.iis.net/t/1149274.aspx..
In order to use Microsoft.Web.Administration you need to have IIS installed, at the bare minimum you need to install the Configuration API's which are brought through installing the Management Tools.
Unfortunately there is no SDK that enables this and it has several dependencies on other components that wouldn't let you just take it to another machine and make it work (such as COM objects, DLL's, etc).
I'd be interested in knowing if you've found a way round this.
Thanks

Try controlling the IIS pool with DirectoryEntry instead.
See this topic:
Check the status of an application pool (IIS 6) with C#

Microsoft.Web.Administration, it relies on System.Web.dll which was provided by framework 4, not client profile.