I'm trying to find what I'm doing wrong regarding an Azure WebJobs QueueTrigger method that should be triggered from an Azure Storage Queue.
I've read a couple of documents (as in blog posts / msdn articles). But I'm still not clear.
Main question / misunderstood aspect:
What should be the name of the connection string for Azure storage console app App.config or Windows Azure Configuration (portal). So far I have the following name set at both places.
AzureJobsStorage
AzureWebJobsStorage
AzureJobsRuntime
AzureJobsDashboard
AzureJobsData
Here's my WebJobs console app code.
static void Main()
{
JobHost host = new JobHost();
host.RunAndBlock();
}
public static void CreateLeague([QueueTrigger("temp")] string msg)
{
var task = JsonConvert.DeserializeObject<QueueTask>(msg);
if (task.TaskType == QueueTask.TaskTypes.Pdf)
RenderPdf(task.Id);
}
This console app is continuously running on my Azure Website.
I can access its "debug" page where I can toggle output and I see it is started / running.
My code to add queue (from my ASP.NET MVC app):
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings["StorageConnectionString"]);
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
CloudQueue queue = queueClient.GetQueueReference("temp");
queue.CreateIfNotExists();
Common.QueueTask task = new Common.QueueTask();
task.TaskType = Common.QueueTask.TaskTypes.Pdf;
task.Id = p.Id;
CloudQueueMessage msg = new CloudQueueMessage(JsonConvert.SerializeObject(task) );
queue.AddMessage(msg);
This code is executed, and queue are added to my Storage Account. But they did not get "dequeue" or read from the WebJobs.
Hmm, the WebJobs class had to be public.
using Microsoft.Azure.WebJobs;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using Newtonsoft.Json;
using Proceed.Common;
using System;
using System.Configuration;
using System.IO;
public class WebJobsTask {
public static void Main()
{
JobHost host = new JobHost();
host.RunAndBlock();
}
public static void CreateLeague([QueueTrigger("temp")] string msg)
{
var task = JsonConvert.DeserializeObject<QueueTask>(msg);
if (task.TaskType == QueueTask.TaskTypes.Pdf)
RenderPdf(task.Id);
}
}
Also found a quick way to explore my queues: https://azurestorageexplorer.codeplex.com/.
In my case, I had assumed that QueueTrigger was referring to Service Bus Queues instead of Azure Queues, and I actually needed to use ServiceBusTrigger.
You can use the server explorer in VS to explore the content of the Storage queues.
The queue triggers for the WebJobs SDK will exponentially back off if there is no work to do. There might be a delay between the moment a message is put in a queue and the moment when it is picked up. You can configure the maximum back off through the JobHostConfiguration.Queues.MaxPollingInterval property.
For the latest SDK you need two storage connection strings AzureWebJobsStorage and AzureWebJobsDashboard
This is a great place for more resources: https://learn.microsoft.com/en-us/azure/app-service-web/websites-webjobs-resources
Related
We want to create an azure function in c# that retrieve the list of azure web app contained in the subscription (basically we want to call dynamically, for each webapp, the same API endpoint changing the subdomain of the api).
It's possible with c# retrieve the list of the web app contained in the same azure function subscriptions?
Usually we connect to the master database, we query the sys.databases to collect the dbname and understand the webapp names. But we are searching for a smartest way.
If you're in C# land, I'd look at using the ArmClient class to retrieve what you're looking for.
Install these (I've got a few others installed but start with that and see how you go, there may be a couple of others needed) Nuget packages ...
Azure.Identity;
Azure.ResourceManager;
Azure.ResourceManager.AppService
... and from there, using the DefaultCredential approach (if you've never used it, read up on it here -> https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/identity/Azure.Identity/README.md) you can query your subscriptions webApps ...
using Azure.Identity;
using Azure.ResourceManager;
using Azure.ResourceManager.AppService;
using System;
using System.Threading.Tasks;
namespace AzureManagement
{
internal class Program
{
static void Main(string[] args)
{
GetAzureResources().Wait();
}
static async Task GetAzureResources()
{
var credential = new DefaultAzureCredential();
var armClient = new ArmClient(credential);
var subscription = await armClient.GetDefaultSubscriptionAsync();
var webSitesEnumerator = subscription.GetWebSitesAsync().GetAsyncEnumerator();
try
{
while (await webSitesEnumerator.MoveNextAsync())
{
var webSite = webSitesEnumerator.Current;
Console.WriteLine($"Web App Name ........ {webSite.Data.Name}");
Console.WriteLine($"Default Host Name ... {webSite.Data.DefaultHostName}\n");
}
}
finally
{
await webSitesEnumerator.DisposeAsync();
}
Console.ReadLine();
}
}
}
The above is obviously not a function app but the core code will still work for you and can be ported as need be.
Note: I could be telling you how to suck eggs, but, once deployed to Azure, you'll need to do the necessary work to ensure that the function app has the required access to retrieve all of the resource information you're looking for.
If you're unfamiliar with that, read up on the managed identity concept. It's very easy to setup -> https://learn.microsoft.com/en-us/azure/app-service/overview-managed-identity
Yes, one easy way is to use HttpClient and send a request to Azure Rest API:
GET https://management.azure.com/subscriptions/{subscriptionId}/providers/Microsoft.Web/sites?api-version=2022-03-01
https://learn.microsoft.com/en-us/rest/api/appservice/web-apps/list
PS: you first need to acquire an authentication token.
https://www.youtube.com/watch?v=6b1J03fDnOg&t=329s
I am trying to get application insights to pick up on custom ActivitySource in a library, however the documentation is unclear on how to achieve this.
Currently I have the following:
...
public static readonly ActivitySource Source = new ActivitySource("MyCompany.Library");
...
In the library it is used like this:
using(var activity = Source.StartActivity("Action"))
{
...
}
And in my startup I've added the following:
services.ConfigureTelemetryModule<DependencyTrackingTelemetryModule>(
(m, o) => m.IncludeDiagnosticSourceActivities.Add("MyCompany.Library")
);
services.AddApplicationInsightsTelemetryWorkerService();
However, these activities are not being picked up by application insights.
Is there something else required to make application insights aware of these activities ?
I'd rather not 'pollute' these libraries with application insights code
ApplicationInsights SDKs does not support reporting telemetry from custom ActivitySource.
There is a preview version which supports ActivitySource based telemetry. (Its called OpenTelemetry AzureMonitorExporter)
https://learn.microsoft.com/azure/azure-monitor/app/opentelemetry-enable?tabs=net
Follow the below steps to operate:
Add the OpenTelemetry.Exporter.Console NuGet package.
dotnet add package OpenTelemetry.Exporter.Console
Update Program.cs with additional OpenTelemetry using directives
using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using System;
using System.Diagnostics;
using System.Threading.Tasks;
Update Main() to create the OpenTelemetry TracerProvider:
public static async Task Main()
{
using var tracerProvider = Sdk.CreateTracerProviderBuilder()
.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("MySample"))
.AddSource("Sample.DistributedTracing")
.AddConsoleExporter()
.Build();
await DoSomeWork();
Console.WriteLine("Example work done");
}
Now the app collects distributed trace information and displays it to the console:
> dotnet run
You will get the result required.
Follow the below link for further reference:
https://learn.microsoft.com/en-us/dotnet/core/diagnostics/distributed-tracing-collection-walkthroughs
I understand
User already has more than 'max_user_connections' active connections
has already many answers. But this is in regard to Hangfire.
I am using Hangfire for background processing.
It worked fine for the first time and only when I started my application. But now its not. SO now when I am trying to open the database to see what's happening it gives me this error.
My idea is that it cannot execute any more items in the queue because it cannot connect to the database.
Also, I cannot open the Hangfire dashboard. It gives 404 error.
I have created the OWIN startup.cs as
using Hangfire;
using Microsoft.Owin;
using Owin;
[assembly: OwinStartup(typeof(MyApp.Startup))]
namespace MyApp
{
public class Startup
{
public void Configuration(IAppBuilder app)
{
app.UseHangfireDashboard();
}
}
}
Also, I have put the code as required in Application_Start in Global.asax.cs as:
var options = new Hangfire.MySql.MySqlStorageOptions
{
QueuePollInterval = TimeSpan.FromSeconds(15), // Default value
PrepareSchemaIfNecessary = false
};
Hangfire.MySql.MySqlStorage storage = new Hangfire.MySql.MySqlStorage("hangfire2", options);
GlobalConfiguration.Configuration.UseStorage(storage);
_backgroundJobServer = new BackgroundJobServer();
Is there any way I can avoid max_user_connection error ? Or actually, run the background call.
Here it is how I am calling it:
BackgroundJob.Enqueue(() => MyMethod());
UPDATE:
After timeout, I saw the database and found that Second operation was not inserted into database, however, the third one was inserted but gives the state of the job as this:
Retry attempt 1 of 10: Exception has been thrown by the target of an inv...
I want to write a code, similar to the code at the bottom of this link (https://azure.microsoft.com/en-us/blog/automating-azure-analysis-services-processing-with-azure-functions/) in Visual Studio and building a DLL file. However instead of using the connection string, i would like to use an existing Linked Service from my Azure portal.
The goal is to create a DLL that refreshes my Cube, while at the same time using an existing Linked Service which is already in my Azure Portal.
Is this possible?
Thanks.
#r "Microsoft.AnalysisServices.Tabular.DLL"
#r "Microsoft.AnalysisServices.Core.DLL"
#r "System.Configuration"
using System;
using System.Configuration;
using Microsoft.AnalysisServices.Tabular;
public static void Run(TimerInfo myTimer, TraceWriter log)
{
log.Info($"C# Timer trigger function started at: {DateTime.Now}");
try
{
Microsoft.AnalysisServices.Tabular.Server asSrv = new Microsoft.AnalysisServices.Tabular.Server();
var connStr = ConfigurationManager.ConnectionStrings["AzureASConnString"].ConnectionString; // Change this to a Linked Service connection
asSrv.Connect(connStr);
Database db = asSrv.Databases["AWInternetSales2"];
Model m = db.Model;
db.Model.RequestRefresh(RefreshType.Full); // Mark the model for refresh
//m.RequestRefresh(RefreshType.Full); // Mark the model for refresh
m.Tables["Date"].RequestRefresh(RefreshType.Full); // Mark only one table for refresh
db.Model.SaveChanges(); //commit which will execute the refresh
asSrv.Disconnect();
}
catch (Exception e)
{
log.Info($"C# Timer trigger function exception: {e.ToString()}");
}
log.Info($"C# Timer trigger function finished at: {DateTime.Now}");
}
So I guess you're using the Data Factory and you want to process your analysis services model from your pipeline. I don't see what your question actually has to do with the Data lake store.
To trigger Azure Functions from the Data Factory (v2 only), you'll have to use a web activity. It is possible to pass a Linked Service as part of your payload, as shown in the documentation. It looks like this:
{
"body": {
"myMessage": "Sample",
"linkedServices": [{
"name": "MyService1",
"properties": {
...
}
}]
}
However, there is no Analysis services linked service in the Data Factory, at least, I didn't hear of such a thing. Passing in a connectionstring from the pipeline seems like a good idea however. You could pass it as a pipeline parameter in your body of the webrequest.
Create a parameter in your pipeline
Add it to your Web Activity Payload
{
"body": {
"AzureASConnString": "#pipeline().parameters.AzureASConnString"
}
You can retrieve this value from functions like described here
I have a Continuous Azure Webjob that checks DocumentDb on each message from QueueTrigger. Use DocumentDB SDK 1.10 for .NET. I keep getting errors like this from time to time:
Only one usage of each socket address (protocol/network address/port) is normally permitted :443
at Microsoft.Azure.Documents.Client.GatewayServiceConfigurationReader.d__0.MoveNext()
I have dependency injection (StructureMap) through the JobActivator and use non-static method to process queue triggers within Functions class. use
DocumentClient is wrapped in a static class so it should be one instance per WebJob.
Also some web apps (ASP.NET MVC) use their own document db connection with same connection string from time to time. I could not find any information about this issue. What could be go wrong here?
Changed internal code:
class Program
{
public static IReliableReadWriteDocumentClient Client { get; set; }
static void Main()
{
Client = // initialize client with settings from app config
}
}
public class Functions
{
public async Task ProcessQueueMessage([QueueTrigger("myqueuename")] CustomerRequest customerRequest,
TextWriter dashboardLogger)
{
// usage from some place which runs on QueueTrigger
var customerRepository = new DocumentDbRepository<Customer>(Program.Client, databaseId,
() => "Customers");
// additional logic
}
}
Repository class is taken from here: https://github.com/Crokus/documentdb-repo