We Keep Coding

DefaultAzureCredential().GetTokenAsync fails when using custom scope when running in Azure

I'm trying to get service-to-service authentication working in Azure AD. I have it working when the two services (webapps) are running locally against an Azure AD app registration. But when the webapps are running in Azure (against the same Azure AD app registration), it doesn't work.
I've created two Azure App Service webapps (which are ASP.NET Core Web API projects) - one acting as the client and the other the server. The bit that breaks is the client webapp, where it requests an access token from AzureAD. It only breaks when requested a token with a custom scope. This same 'custom scope' token request works fine from a local webapp though.
In Azure, I've created an 'App Registration', and given it a custom scope called 'bob'...
In the client ASP.NET Core webapi app, I've added the Azure.Identity nuget package, and am getting a token like this...
var tokenRequestContext = new TokenRequestContext(new[] {scope});
token = await new DefaultAzureCredential().GetTokenAsync(tokenRequestContext, cancellationToken);
(where the scope variable matches the name from the screenshot above)
Locally, this works fine and returns a token.
In Azure, I get this exception:
Azure.Identity.AuthenticationFailedException: DefaultAzureCredential failed to retrieve a token from the included credentials.
- ClientSecretCredential authentication failed: AADSTS70011: The provided request must include a 'scope' input parameter. The provided value for the input parameter 'scope' is not valid. The scope api://<guid>/bob is not valid.
(I've removed the GUID from that snippet)
When running this locally and in Azure, I'm setting the AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID environment variables to the same value. I don't understand why it would work differently locally than in Azure, given these environment variables are the same.
Note that my intention is to use a Managed Service Identity instead of setting those environment variables. But for the time being, by using those environment variables both locally and in Azure - both should be using the same "ClientSecretCredential" authentication - so I'd expect the same results.

There are 2 parts of the answer:
Azure.Identity is currently meant to be ONLY for the native Azure Services SDK using new Azure.* client libraries. In your case, you are trying to use it for your custom web app auth with custom scope which is not officially supported or documented. For regular auth flow (like in this case Client Credential Grant for example) You should be still using MSAL, OR for easier integration with asp.net core apps, Microsoft Identity Web.
NOTE: Specifically for Client Credential Flow, it only allows .default scope. There is an open UserVoice and another thread for workaround.
Now coming to your problem of why it's is not working in Azure.Identity DefaultAzureCredential (regardless of the fact it's not officially supported). I suspect at your deployed Web App, it's probably one or more context missing (IDENTITY_ENDPOINT or AZURE_AUTHORITY_HOST would be my wild guess looking at code). But no idea unless we do little trial and error (again, I would only want to do it for experimental purpose, not in a real production application due to lack of official support). The DefaultAzureCredential attempts to authenticate via the following mechanisms in order. In your local machine, it might be able to get the required context from one of mechanisms down in the chain (like VS or CLI) whereas in the deployed app, those are simply non-existent.

While explicitly stated - you can use the .default claim as hinted at here: https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-permissions-and-consent#the-default-scope.
This will let you generate an access token for your API but it will not have any other scopes for you to verify.

Using IdentityServer4 as Class Library instead of Hosting Solution

I have been exploring IdentityServer4 for a couple of days. It performs hosting internally generating a connect/token endpoint which internally validates and generates an access token.
Even to call it a discovery URL is needed ultimately requiring a web app template for hosting.
Is it possible to use IdentityServer4 as a library that just allows access tokens to be generated?
**Updated Based on input from Bryan
What I am trying to achieve here is GSMA Compliance that requires both OpenID Connect and OAuth. I need to generate an Access Token as well as Identity_Token. But the caveat here is that IdentityServer4 hosts its internal endpoint where I do have custom endpoints. Also it require http based calling and I do require a sort of library that I can integrate and internally call to generate Access token as well as Identity Token.
** Some more update
We have hosted a custom GSMA compliant Web API Project and have hosted several endpoints as per required by GSMA ultimately validating and sanitizing the request landing on our endpoints. Internally we do require an Access token and Identity Token to be generated and shared back on the callback of the calling party. Currently, I have used an Identity Server template which is MVC based project ultimately requiring us to use at least two ports/project one for our Custom GSMA compliant APIs and one for Identity Server. Calling identity server via URL add a new hop and an HTTP request needs to be sent out to Identity Server whereas I was thinking to use IdentityServer as a class library somewhat allowing me to pass in input parameter and do get an access token/ identity toke.
Upon dissecting server Identity Server project like
Identity Server 4
Identity Mode
Identity Storage
I got the impression that it's very difficult to exact a library out of these projects that can help me generate access token/identity token easily.

IdentityServer4 is an OpenID Connect (and OAuth) identity service -- it's not just about tokens. Those technologies are primarily HTTP-based. Thus IS4 is built on top of ASP.Net Core's web tech. If you just need tokens (JWT, I assume), it's pretty simple to roll your own and use them however you see fit. A few Google searches (or an SO search) will provide you with the code needed to create your own JWTs (for example: https://houseofcat.io/tutorials/csharp/identity/createjwt). Without more information about how you want to use them, I can't elaborate on this answer.

Stateless .NET JWT based oauth2 authentication server

First of all excuse my bad english.
I'm a newby in oauth2 and I'm trying to setup an authorization server with JWT tokens.
The server is running in a Owin self-hosted environment.
At this moment I've setup the token generation and it seems to work properly, I've also setup the refresh token provider and it seems to work (but it's just a minimal implementation); I didn't change the RefreshTokenFormat in the OAuthAuthorizationServerOptions leaving it to default (I don't know what is the default format), so the access-token is clearly a JWT token and the refresh token is something else.
All of this seems to work correctly, but I will need to run this authorization server in a microservices environment, so the authorization server service might be moved at any moment from one machine to another, so it needs to be stateless or at least to save needed informations on a shared storage (DB) or replicated local storage (I'm on Service Fabric) or some kind of distributed cache.
Testing I noticed that if I generate a refresh-token on a machine and then I try to use this refresh-token on another instance of the authorization server (on another machine) to get a new access-token, it fail with a generic invalid_grant error. My guess is that the auth server just keeps some token-related information in memory but I don't know exactly what and how.
I also would like someone to point me in the right direction to solve this issue.
Thank you very much.

I solved the situation using a JWT token also for the refresh token.
This way the authentication server seems to be completly stateless.

Azure Active Directory Authentication with Azure Mobile Services Failed

I followed this Microsoft Azure Mobile Service Authentication Tutorial, to try to add a server authentication function for my Windows Store C# app. However, after completing every step, when I run my app, it showed that the application cannot connect to the service.
I found this useful blog tallking about troubleshooting Azure Authentication issues in Azure Mobile Service. To troubleshoot, I type the link in the firefox web browser: myServiceUrl/login/aad, but I receive the error response:
Authorization has been denied for this request.
I also followed the same tutorial to test with Google Log in. It turns out to work properly. And when I type the link: myServiceUrl/login/google, the web browser directed me to the google log in page, unlike the Unauthorization error message when I type in myServiceUrl/login/aad.
Although google log in works out fine, but it is desired for us to use Azure Active Directory authentication. Could anyone tell us what could possibly be wrong? Any troubleshooting suggestions are also appreciated. Thank you.

The "Application cannot connect to the service" error comes from whenever the Web Authentication Broker in Windows receives an error response from the resource it is trying to reach. There are a couple of issues that can cause this, and I'll try and address the most common ones.
I noticed the tutorial you linked to is for the .NET backend. If you are using the Mobile Services .NET backend, there is an extra step required to configure the AAD server flow, and it's a common cause of the issue being described. In the tutorial, it's under the title "Configure your .NET mobile service for AAD login." On the backend project, you will need to install the Mobile Services .NET Backend Security Extension NuGet package. Then, in WebApiConfig.cs, you will need to include
options.LoginProviders.Remove(typeof(AzureActiveDirectoryLoginProvider));
options.LoginProviders.Add(typeof(AzureActiveDirectoryExtendedLoginProvider));
This allows the runtime to use the server flow in addition to the client flow (leveraging the Active Directory Authentication Library) which was first released with the .NET backend.
Pending that, or in the case of the Node runtime, the next thing to do is check the AAD configuration. In the AAD portal, make sure that your application registration uses your mobile service's /login/aad endpoint for the resource URI. It must match exactly the value provided in the Mobile Services portal. This should also be one of the redirect URIs if you are using the Node backend. For .NET, you would use the /signin-aad endpoint for the redirect URI instead.
You should also check that you have copied the Client ID from the AAD registration and pasted it into the Mobile Services portal's Identity Tab. For completeness, the "Allowed Tenants" field should also be filled out, but I don't believe it is the cause of this issue.
Lastly, if your AAD tenant is federated with ADFS, then there is a wide range of issues that could lead to this. The biggest case comes from the WAB needing to be configured for Enterprise Authentication. This typically only causes problems when the device is domain joined / on the corporate network. That behavior is a known bug for the Mobile Services Windows Store SDK, but there is a workaround available. Glad to provide that if needed.

How to get started with OAuth to secure a Web API application?

I have a Web API application and I've understood OAuth would be the standard security model for APIs where an Authentication Server would become responsible to generate Authorization Tokens so that the user can send to our server and consume the services.
I'm very new to this but I understand the roles involved:
Resource Owner
Client
Resource Server
Authorization Server
But what is OAuth exactly in practice, not in theory? Is it a .NET library? Is it a service provided by a separate Company? Is it something I can configure on my local development machine and see how it works?
How to get started with OAuth to secure a Web API application?

OAuth is a protocol; the current version is OAuth 2.0. More to your question, that link lists several implementations of the protocol in various technologies. For use with the .NET Web API you're probably interested in DotNetOpenAuth which provides implementations of both OAuth 1 and OAuth 2.
I'm using DotNetOpenAuth in an app I'm working on now to secure a .NET Web API. I've got an OAuth2Handler which extends DelegatingHandler which is inserted into the Web API pipeline before incoming requests reach any controllers. OAuth2Handler does the following:
Instantiates a DotNetOpenAuth ResourceServer
Calls ResourceServer.GetPrincipal() which reads and decrypts an access
token (issued elsewhere by the AuthorizationServer and returns an
OAuthPrincipal (In my case I'm reading additional data that the DotNetOpenAuth implementation allows you to pass and creating a ClaimsPrincipal.)
Assigning the IPrincipal containing the user information read from the access token to the User property of the thread and current HTTP context so it is available from the ApiController.User property in the service controllers: httpContext.User = Thread.CurrentPrincipal = principal;
Honestly, getting this all working (e.g. setting up the authorization server, resource server, certificates, etc.) isn't trivial. Unfortunately there didn't seem to be a good guide on the DotNetOpenAuth site. Here's a few other tasks you'll have ahead of you if you go this route:
Implement IAuthorizationServer - This is the interface provided by
DotNetOpenAuth that allows you to plug in to the library and use
their implementation to issue OAuth2 access tokens. You'll also need to implement INonceStore and ICryptoKeyStore which I did using an EntityFramework context for storage.
Configure Certificates - The AuthorizationServer and ResourceServer each use certificates to encrypt/decrypt the access token ensuring they are only accessible to each other. I built some custom configuration so I could manage this configuration in the web.config files of my authorization server app and my Web API services (resource server).
Manage Refresh Token - When first requesting an access token from the authorization server you'll get back (depending on your configuration) both an OAuth2 refresh token and an access token. The services use the access token which should be short-lived. The refresh token is used to get more access tokens. The refresh token should be kept secret (whatever that means in your scenario). For me it means the refresh token is never exposed to client-side javascript in my web app.
I hope that helps give you a high level idea of how to get started with OAuth and .NET Web API. Here's a blog post demonstrating some of these steps. This SO answer gives a few more high level details of the client side of the picture.
(The DotNetOpenAuth online docs appear to be down right now... sorry for no links to them; Apparently it has happened before).