I'm currently setting up a multiple publisher/single subscriber architecture, using Azure Storage Queues to buffer events and Azure Functions as the subscriber.
Publisher -> Queue -> Function
Creating the function is no problem, the part I am trying to work out is how to set up a development and a production deployment of the same function. I have created the function in Visual Studio, and the connection is a constant string literal:
public static class FooAdded
{
[FunctionName("FooAdded")]
public static void Run([QueueTrigger("tracker-readings", Connection = "https://foo.queue.core.windows.net/?sv=...")]string myQueueItem, TraceWriter log)
{
log.Info($"C# Queue trigger function processed: {myQueueItem}");
}
}
How can I provide a different value for the connection, depending on whether I deploy to the development environment or the live environment?
To set up local debug environment
You can use local.settings.json file to define the local settings. The prerequisite for using Azure storage locally is that you need to have Azure Storage Emulator running on your machine. In the local.settings.json file define the Azure Storage Account connection string as UseDevelopmentStorage=true. The file should look something like this:
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "UseDevelopmentStorage=true",
"AzureWebJobsDashboard": "UseDevelopmentStorage=true"
}
}
By default, if no value is provided for the Connection property of the QueueTrigger attribute, it will use the AzureWebJobsStorage setting:
public static class FooAdded
{
[FunctionName("FooAdded")]
public static void Run([QueueTrigger("tracker-readings")]string myQueueItem, TraceWriter log)
{
log.Info($"C# Queue trigger function processed: {myQueueItem}");
}
}
Alternatively, if you want to explicitly specify a connection string, then you can set the Connection to the name of the connection string (not the connection string itself), and add an entry for it in the Values section of the configuration file:
QueueTrigger("tracker-readings", Connection = "CustomConnection")
In the local.settings.json file:
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "UseDevelopmentStorage=true",
"AzureWebJobsDashboard": "UseDevelopmentStorage=true",
"CustomConnection": "Some custom connection string"
}
}
For more details refer to the official documentation:
Develop and run Azure functions locally
App settings reference
To set up an Azure environment:
The values in local.settings.json do not get published to Azure, and so a value for each setting needs to be defined in the Azure environment that you are deploying to. Note that the values for AzureWebJobsStorage and AzureWebJobsDashboard are set automatically based off the storage account you choose when creating the function.
The easiest way to define settings is through the Azure portal. Go to Application Settings in the Function App and define the actual value of the storage account connection string. You do not need to make any code changes to the azure function, it will automatically pick up the connection string from application settings.
Alternatively, you can use Azure Resource Manager templates, to deploy and update the environment settings programmatically.
Each function app you create in Azure has its own environment, so after providing values for the relevant settings, you can deploy your function to multiple environments in Azure (Dev/Test/Prod etc.) as well as debug your code locally without changing connection strings everytime.
Have the connection string in an environment variable(or an app setting in app services). You can then have the app setting set to to different values in different environments.
I am trying to work out is how to set up a development and a production deployment of the same function.
I am not clear why you want to create a function both as production and development.
Actually, Azure billing is based on your app service plan, so you create one more azure function may not cost more. You could create two function to distinguish them more clearly.
If you still want to use a single function, as Travis said, you could set two connection string in app settings for different environment.
This is a well known scenario and fully supported. All you need to do manage you deployment with the ARM Template and a Parameter File.
Deploy resources with Resource Manager templates and Azure PowerShell
The connection string, along with any other Sub/Resource Group dependent settings are just parameters.
You can set Connection to a config value (something like Connection=AppSettingsKey.DatabaseConnectionString) and then have different settings for different platforms (dev, staging, prod).
Depending on your tooling for build and deployment, you can inject config values for connection strings during one of these steps.
Another option would be to use a Keyvault secret that stores the connection string.
Related
How do I move sensitive information below into the 'The Secret Manager Tool'?
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
=> optionsBuilder.UseSqlServer("Server=Test; column encryption setting=enabled;Database=Test;user id=User1;password='Password1';Trust Server Certificate=true");
I know I can right click on the solution name and select "Manage User Secrets", which then generates the secret Json file, but what I am pasting into this file?
And when I move this application over to the production server, do I copy & paste over the secret.json as well?
Thanks in advance.
You need to take a small step back and consider tools ASP.NET Core/.NET provides to work with configuration.
From DbContext Lifetime, Configuration, and Initialization doc you can see that one of the common pattern is to use dependency injection and setup the the connection string on application startup. This will require adding constructor to the context (and modifying/removing OnConfiguring overload - docs):
public class ApplicationDbContext : DbContext
{
public ApplicationDbContext(DbContextOptions<ApplicationDbContext> options)
: base(options)
{
}
}
And:
builder.Services.AddDbContext<ApplicationDbContext>(
options => options.UseSqlServer("ConnectionStringHere")); // or AddDbContextFactory
Next step is to read the connection string from the settings:
builder.Services.AddDbContext<ApplicationDbContext>(
options => options.UseSqlServer(builder.Configuration.GetConnectionString("ConnStringName"))); // or AddDbContextFactory
Which will require connection string to be read from the configuration, for example from appsettting.json:
{
"ConnectionStrings": {
"ConnStringName": "ACTUAL_CONNECTION_STRING"
},
}
Also you can move connection string to any of the supported configuration providers, for example - Secret Manager (note, it is for development environment only, for other environments is better to use either environment variables or secured key storages).
In the production server you need to create a configuration key with the same name with the production connection string.
Set the ASPNETCORE_ENVIRONMENT variable to "Production"
When environment "ASPNETCORE_ENVIRONMENT" is in "Production" the secrets.json source is ignored. You can check this in Default application configuration sources
According to ASP.NET Core documentation
WebApplication.CreateBuilder initializes a new instance of the WebApplicationBuilder class with preconfigured defaults. The initialized WebApplicationBuilder (builder) provides default configuration for the app in the following order, from highest to lowest priority:
Command-line arguments using the Command-line configuration provider.
Non-prefixed environment variables using the Non-prefixed environment variables configuration provider.
User secrets when the app runs in the Development environment.
appsettings.{Environment}.json using the JSON configuration provider. For example, appsettings.Production.json and appsettings.Development.json.
appsettings.json using the JSON configuration provider.
A fallback to the host configuration described in the next section.
When running production apps in AWS, GCP, or Azure, we keep all secrets in the cloud provider's respective secret manager (e.g. AWS Secrets Manager) and use scripting when the runtime environment is created to inject those secrets into environment variables.
Note, by Secrets Manager, I don't mean the .NET Secret Manager, which merely stores secrets in a file on the development machine.
.NET can read configuration from the environment, including connection strings.
With this approach, only trusted DevOps personnel ever have the ability to view a connection string. It is never in a file, and certainly never in source control.
Best that you can do for the connection string in the production server is to configure windows authentication.
There must be a network user configured. This user should be given permissions in the SQL Server.
When you configure your Web server pool, run this pool under identity created in #1
At this point your connection string will look like this Server=MyServerName;Database=MyDbName;Trusted_Connection=SSPI;Encrypt=false;TrustServerCertificate=true. As you see - no password or user
With this setup, unless attacker knows password to the account under which your pool runs, this is plenty secure.
If you insist in using Sql Authentication, you need to store your credentials in the secure secrets manager (even if homegrown) and then acquire them and build connection string at runtime.
To preface, I know that the connection string comes from the Configuration on the Azure Portal / local.settings.json, and that it's an environment variable you can access with the following:
Environment.GetEnvironmentVariable("Name", EnvironmentVariableTarget.Process);
I am trying to set the ServiceBusTrigger connection string from the appsettings.json.
I have added an appsettings.json file following this blog on how to do it. I am able to access settings from the appsettings.json file, however in the Program.cs when I try to use the following to set the environment variable:
.ConfigureAppConfiguration(c =>
{
var config = c.SetBasePath(Environment.CurrentDirectory)
.AddJsonFile("appsettings.json", true, true)
.AddEnvironmentVariables()
.Build();
// Setting the Environment Variable for the connection string
Environment.SetEnvironmentVariable(config["ConnectionString:Name"], config["ConnectionString:Value"]);
})
where "Name" is what's passed to the Connection in the function, and "Value" is the connection string, I get the following error:
Microsoft.Azure.WebJobs.ServiceBus: Microsoft Azure WebJobs SDK ServiceBus connection string 'Name' is missing or empty.
The Connection I'm talking about:
public async Task Run(ServiceBusTrigger("myqueue", Connection = "Name")] string myQueueItem, FunctionContext) { // Some Implementation }
I am trying to avoid setting the connection string "Name" in the App Configuration on the Azure portal. Is there a way to do it programmatically?
When you set the variable by Environment.SetEnvironmentVariable, it will not show in application setting. But we can use it by Environment.GetEnvironmentVariable as expected.
SetEnvironmentVariable(String,String) Creates, modifies, or deletes an environment variable stored in the current process.
Although the solution you mentioned is not so good, but it can implement your requirement. The adverse effect is when you restart the function app, the variables will be lost.
For more information on SetEnvironmentVariable, please refer this MSFT documentation
We have Azure Functions (V2) that have been created with the Service Bus Trigger.
[FunctionName("MyFunctionName")]
public static async Task Run(
[ServiceBusTrigger("%MyQueueName%", Connection = "ServiceBusConnectionString")]
byte[] messageBytes,
TraceWriter log)
{
// code to handle message
}
The queue name is defined in the local.settings.json file:
{
"Values": {
...
"MyQueueName": "local-name-of-my-queue-in-azure",
...
}
}
This works quite well as when deployed we can set the environment variables to be dev-queue-name, live-queue-name etc for the various deployed environments that we have.
However, when more than one developer is connected locally, given that the local.settings.json file is in source control and needs to be to properly maintain the environment variables, then the local function app runners will all connect to the same queue, and it is random as to which developer's application will pick up and process the messages.
What we need is for each developer to have their own queue, but we do not want to have to remove the JSON config file from source control so that we can maintain a different file (as it contains other pieces of information that need updating).
How can we get each developer / computer running our application to have a unique queue name (but known so that we can create the service bus queues in the cloud)?
You can override the setting value via Environment variables. Settings specified as a system environment variable take precedence over values in the local.settings.json file. Just define an Environment variable called MyQueueName.
Having said that, I think that committing local.settings.json to source control is generally not recommended. I suppose you also store your Service Bus connection string there, which means you store your secrets in source control.
Note that default .gitignore file has it listed out.
If you need it in source control, I would commit a version of local.settings.json with all variables with fake values, and then make each developer setup the proper values locally and then ignore the changes on commit (set assume-unchanged).
I am adding a webjob project to a website project. Webjob project requires AzureWebJobsStorage and AzureWebJobsDashboard connection strings in the environment when deployed to Azure. Is it possible to change the name of these connection strings, i.e. MyAzureWebJobsDashboard and MyAzureWebJobsDashboard?
The connection string names themselves are not configurable. They are settable via the JobHostConfiguration.StorageConnectionString and JobHostConfiguration.DashboardConnectionString properties, which would allow you to read and set them however you choose.
However, that won't work for the WebJobs Dashboard itself - it reads the settings directly from the app settings set in the portal and expects these well known names.
I created a new Azure WebJob project in Visual Studio 2015 using .NET Framework 4.6.
In the app.config, I set three connection strings:
AzureWebJobsDashboard
AzureWebJobsStorage
MyDatabaseConnectionString
The AzureWebJobsDashboard and AzureWebJobsStorage connection strings are identical and they're both pointing to my storage account. I'm including one of the connection strings -- since they're both identical, except the "name".
<add name="AzureWebJobsDashboard" connectionString="DefaultEndpointsProtocol=https;AccountName=mystorageaccountname;AccountKey=thisIsTheLongPrimaryKeyICopiedFromAzurePortalForMyStorageAccount" />
Everything looks right to me but I'm getting the following error:
The configuration is not properly set for the Microsoft Azure WebJobs
Dashboard. In your Microsoft Azure Website configuration you must set
a connection string named AzureWebJobsDashboard by using the following
format DefaultEndpointsProtocol=https;AccountName=NAME;AccountKey=KEY
pointing to the Microsoft Azure Storage account where the Microsoft
Azure WebJobs Runtime logs are stored.
By the way, I know the app.config is being read by the web job because my code is able to connect to my database and update some records.
Any idea what I'm doing wrong?
You need to set the AzureWebJobsDashboard connection string in the portal in your Web App Application Settings blade (steps to do that here). The Dashboard runs as a separate site extension and doesn't have access to app.config. Add the connection string to the connection strings section on the settings blade.
You can add your other connection strings there as well (e.g. AzureWebJobsStorage) rather than storing in app.config if you wish for security/consistency, however the WebJob can read AzureWebJobsStorage from app.config.
The change needs to be done in App Services settings in Azure Portal
For that
Open the Azure (Management) Portal at https://portal.azure.com
Goto Home > App Services
Select the App service that is hosting your WebJob
Goto Settings > Choose Application settings
Scroll down to Connection strings
Add a new connection string with Name as 'AzureWebJobsDashboard' and Value as . Choose Type as 'Custom'
Press Save Button ( at the Page Top)
All done! Check your Webjobs Dashboard - the Warning and Error message at top should be gone now.
I was having this problem too.
My storageaccount kind is StorageV2 (general purpose V2)
I had both AzureWebJobsDashboard and AzureWebJobsStorage correctly set in the App Service Configuration.
But, the storage account had a Minimum TLS version set to 1.2
I found that changing this to 1.0 was needed for the WebJobs Dashboard to display correctly and for the WebJobs to run ok.
Right now (version of Web Job Tools is 15.0.31201.0) no necessary to configure any connection strings from azure portal, enough to set it in app.config file of web job