We Keep Coding

Is it possible that 2 azure functions can get triggered by one eventhub?

Is it possible that 2 azure functions can get triggered by one eventhub? One azure function will write its data to database1 and the other azure function writes its data to database2
[FunctionName("EventToDB1")]
public async System.Threading.Tasks.Task Run([EventHubTrigger("eventhub", Connection = "Debezium")]
EventData[] events, ILogger log)
{
{
[FunctionName("EventToDB2")]
public async System.Threading.Tasks.Task Run([EventHubTrigger("eventhub", Connection = "Debezium")]
EventData[] events, ILogger log)
{
{
answer on the possibility of having 2 azure functions get triggered by one eventhub

Yes that is possible by using different consumer groups. Because you specified the same connection to the Event Hub, being "Debezium", I Assume you want both funtions to process the same message. You have to create a new consumer group and specify the name using the ConsumerGroup property of the EventHubTrigger attribute (The default consumergroup is $Default):
public class EventToDB1
{
[FunctionName("EventToDB1")]
public async System.Threading.Tasks.Task Run(
[EventHubTrigger("eventhub",
Connection = "Debezium",
ConsumerGroup = "CG1")]
EventData[] events, ILogger log)
{
}
}
public class EventToDB2
{
[FunctionName("EventToDB2")]
public async System.Threading.Tasks.Task Run(
[EventHubTrigger("eventhub",
Connection = "Debezium",
ConsumerGroup = "CG2")]
EventData[] events, ILogger log)
{
}
}
Each consumer group receives the same messages from the Event Hub.

I do agree with #peter bons, you need to create two consumer groups for that and you can create two consumers by below process:
You can also use logic apps to work with event hubs.

How to efficiently count HTTP Calls in asp.net core?

I have an abstract class called HttpHelper it has basic methods like, GET, POST, PATCH, PUT
What I need to achieve is this:
Store the url, time & date in the database each time the function is called GET, POST, PATCH, PUT
I don't want to store directly to the database each time the functions are called (that would be slow) but to put it somewhere (like a static queue-memory-cache) which must be faster and non blocking, and have a background long running process that will look into this cache-storage-like which will then store the values in the database.
I have no clear idea how to do this but the main purpose of doing so is to take the count of each calls per hour or day, by domain, resource and url query.
I'm thinking if I could do the following:
Create a static class which uses ConcurrentQueue<T> to store data and call that class in each function inside HttpHelper class
Create a background task similar to this: Asp.Net core long running/background task
Or use Hangfire, but that might be too much for simple task
Or is there a built-in method for this in .netcore?

Both Hangfire and background tasks would do the trick as consumers of the queue items.
Hangfire was there before long running background tasks (pre .net core), so go with the long running tasks for net core implementations.
There is a but here though.
How important is to you that you will not miss a call? If it is, then neither can help you.
The Queue or whatever static construct you have will be deleted the time your application crashes/machine restarts or just plain recycling of the application pools.
You need to consider some kind of external Queuing mechanism like rabbit mq with persistence on.
You can also append to a file, but that might also cause some delays as read/write.

I do not know how complex your problem is but I would consider two solutions.
First is calling Async Insert Method which will not block your main thread but will start task. You can return response without waiting for your log to be appended to database. Since you want it to be implemented in only some methods, I would do it using Attributes and Middleware.
Simplified example:
public IActionResult SomePostMethod()
{
LogActionAsync("This Is Post Method");
return StatusCode(201);
}
public static Task LogActionAsync(string someParameter)
{
return Task.Run(() => {
// Communicate with database (X ms)
});
}
Better solution is creating buffer which will not communicate with database each time but only when filled or at interval. It would look like this:
public IActionResult SomePostMethod()
{
APILog.Log(new APILog.Item() { Date = DateTime.Now, Item1 = "Something" });
return StatusCode(201);
}
public partial class APILog
{
private static List<APILog.Item> _buffer = null;
private cont int _msTimeout = 60000; // Timeout between updates
private static object _updateLock = new object();
static APILog()
{
StartDBUpdateLoopAsync();
}
private void StartDBUpdateLoopAsync()
{
// check if it has been already and other stuff
Task.Run(() => {
while(true) // Do not use true but some other expression that is telling you if your application is running.
{
Thread.Sleep(60000);
lock(_updateLock)
{
foreach(APILog.Item item in _buffer)
{
//Import into database here
}
}
}
});
}
public static void Log(APILog.Item item)
{
lock(_updateLock)
{
if(_buffer == null)
_buffer = new List<APILog.Item>();
_buffer.Add(item);
}
}
}
public partial class APILog
{
public class Item
{
public string Item1 { get; set; }
public DateTime Date { get; set; }
}
}
Also in this second example I would not call APILog.Log() each time but use Middleware in combination with Attribute

Logging Hangfire jobs to Application Insights and correlating activity to an Operation Id

I feel like this should be a lot simpler than it's turning out to be, or I am just over thinking it too much.
I have a .NET Core 3.1 Web API application, which is using HangFire to process some jobs in the background. I have also configured Application Insights to log Telemetry from the .NET Core API.
I can see logging events and dependency telemetry data logged in Application Insights. However, each event/log/dependency is recorded against a unique OperationId and Parent Id.
I am trying to determine how to ensure that any activity which is logged, or any dependencies which are used in the context of the background job are logged against the OperationId and/or Parent Id of the original request which queued the background job.
When I queue a job, I can get the current OperationId of the incoming HTTP request, and I push that into the HangFire queue with the job. When the job is then performed, I can get back that OperationId. What I then need to do is make that OperationID available throughout the context/lifetime of the job execution, so that it is attached to any Telemetry sent to Application Insightd.
I thought I could create a IJobContext interface, which could be injected into the class which performs the job. Within that context I could push the OperationID. I could then create a ITelemetryInitializer which would also take the IJobContext as a dependency. In the ITelemetryInitializer I could then set the OperationID and ParentId of the telemetry being sent to Application Insights. Here's some simple code:
public class HangFirePanelMessageQueue : IMessageQueue
{
private readonly MessageProcessor _messageProcessor;
private readonly IHangFireJobContext _jobContext;
private readonly TelemetryClient _telemetryClient;
public HangFirePanelMessageQueue(MessageProcessor panelMessageProcessor,
IIoTMessageSerializer iotHubMessageSerialiser,
IHangFireJobContext jobContext, TelemetryClient telemetryClient)
{
_messageProcessor = panelMessageProcessor;
_jobContext = jobContext;
_telemetryClient = telemetryClient;
}
public async Task ProcessQueuedMessage(string message, string operationId)
{
var iotMessage = _iotHubMessageSerialiser.GetMessage(message);
_jobContext?.Set(iotMessage.CorrelationID, iotMessage.MessageID);
await _messageProcessor.ProcessMessage(iotMessage);
}
public Task QueueMessageForProcessing(string message)
{
var dummyTrace = new TraceTelemetry("Queuing message for processing", SeverityLevel.Information);
_telemetryClient.TrackTrace(dummyTrace);
string opId = dummyTrace.Context.Operation.Id;
BackgroundJob.Enqueue(() =>
ProcessQueuedMessage(message, opId));
return Task.CompletedTask;
}
}
The IJobContext would look something like this:
public interface IHangFireJobContext
{
bool Initialised { get; }
string OperationId { get; }
string JobId { get; }
void Set(string operationId, string jobId);
}
And then I would have an ITelemetryInitializer which enriches any ITelemetry:
public class EnrichBackgroundJobTelemetry : ITelemetryInitializer
{
private readonly IHangFireJobContext jobContext;
public EnrichBackgroundJobTelemetry(IHangFireJobContext jobContext)
{
this.jobContext = jobContext;
}
public void Initialize(ITelemetry telemetry)
{
if (!jobContext.Initialised)
{
return;
}
telemetry.Context.Operation.Id = jobContext.OperationId;
}
}
The problem I have however is that the ITelemetryInitializer is a singleton, and so it would be instantiated once with a IHangFireJobContext which would then never update for any subsequent HangFire job.
I did find the https://github.com/skwasjer/Hangfire.Correlate project, which extends https://github.com/skwasjer/Correlate. Correlate creates a correlation context which can be accessed via a ICorrelationContextAccessor which is similar to the IHttpContextAccessor.
However, the footnotes for Correlate state "Please consider that .NET Core 3 now has built-in support for W3C TraceContext (blog) and that there are other distributed tracing libraries with more functionality than Correlate." which lists Application Insights as one of the alternatives for more Advanced distributed tracing.
So can anyone help me understand how I can enrich any Telemetry going to Application Insights when it is created within the context of a HangFire job? I feel the correct answer is to use an ITelemetryInitializer and populate the OperationId on that ITelemetry item, however, I am not sure what dependancy to inject into the ITelemetryInitialzer in order to get access to the HangFire Job Context.

When I queue a job, I can get the current OperationId of the incoming HTTP request, and I push that into the HangFire queue with the job.
So, am I correct to say that you have a controller action that pushes work to hangfire? If so What you can do is inside the controller method get the operation id and pass it to the job. Use that operation id to start a new operation using the operation Id. That operation, together with all the telemetry generated during that operation, will be linked to the original request.
I have no hangfire integration but the code below shows the general idea: some work is queued to be done in the background and should be linked to the request regarding the telemetry:
[HttpGet("/api/demo5")]
public ActionResult TrackWorker()
{
var requestTelemetry = HttpContext.Features.Get<RequestTelemetry>();
_taskQueue.QueueBackgroundWorkItem(async ct =>
{
using(var op = _telemetryClient.StartOperation<DependencyTelemetry>("QueuedWork", requestTelemetry.Context.Operation.Id))
{
_ = await new HttpClient().GetStringAsync("http://blank.org");
await Task.Delay(250);
op.Telemetry.ResultCode = "200";
op.Telemetry.Success = true;
}
});
return Accepted();
}
The full example can be found here.

Working from Peter Bons' example I did it like this:
Code originally triggered from a controller action:
// Get the current ApplicationInsights Id. Could use .RootId if
// you only want the OperationId, but I want the ParentId too
var activityId = System.Diagnostics.Activity.Current?.Id;
_backgroundJobClient.Enqueue<JobDefinition>(x =>
x.MyMethod(queueName, otherMethodParams, activityId));
In my JobDefinition class:
// I use different queues, but you don't need to.
// otherMethodParams is just an example. Have as many as you need, like normal.
[AutomaticRetry(OnAttemptsExceeded = AttemptsExceededAction.Delete, Attempts = 10)]
[QueueNameFromFirstParameter]
public async Task MyMethod(string queueName, string otherMethodParams,
string activityId)
{
var (operationId, parentId) = SplitCorrelationIdIntoOperationIdAndParentId(
activityId);
// Starting this new operation will initialise
// System.Diagnostics.Activity.Current.
using (var operation = _telemetryClient.StartOperation<DependencyTelemetry>(
"JobDefinition.MyMethod", operationId, parentId))
{
try
{
operation.Telemetry.Data = $"something useful here";
// If you have other state you'd like in App Insights logs,
// call AddBaggage and they show up as a customDimension,
// e.g. in any trace logs.
System.Diagnostics.Activity.Current.AddBaggage("QueueName", queueName);
// ... do the real background work here...
operation.Telemetry.Success = true;
}
catch (Exception)
{
operation.Telemetry.Success = false;
throw;
}
}
}
// Splits full value from System.Diagnostics.Current.Activity.Id
// like "00-12994526f1cb134bbddd0f256e8bc3f0-872b3bd78c345a46-00"
// into values ( "12994526f1cb134bbddd0f256e8bc3f0", "872b3bd78c345a46" )
private static (string, string) SplitCorrelationIdIntoOperationIdAndParentId(string activityId)
{
if (string.IsNullOrEmpty(activityId))
return (null, null);
var splits = activityId.Split('-');
// This is what should happen
if (splits.Length >= 3)
return (splits[1], splits[2]);
// Must be in a weird format. Try to return something useful.
if (splits.Length == 2)
return (splits[0], splits[1]);
return (activityId, null);
}
I'm not sure using the OperationId and ParentId is quite right here, e.g. it does tie the background job to the originating request's OperationId, but if the originating Request has a ParentId then this background job should really have its ParentId set as the Request, not as the Request's ParentId. Anyone know?

Azure Functions QueueTrigger with variable name, QueueTriggerAttribute restrictions

Assume the following typical Queue Trigger function:
public void Run([QueueTrigger("queue1")]object data, ILogger log)
{
// Do something with data
}
My problem is that "queue1" has to be a constant field, so it has to be defined at compile time.
Also, I'd want to have a base class for Queue Triggers, that could work like this:
public abstract class QueueBase<TModel>
{
public void Run([QueueTrigger("queueName")]TModel data, ILogger log)
{
// Do something with data, log something etc.
OnRunExecuted(data);
// Do something with data, log something etc.
}
public abstract void OnRunExecuted(TModel data);
}
with this, I could write own classes which inherit from QueueBase but can even live inside a library which doesn't have Microsoft.Azure.WebJobs dependency:
public class MyQueueHandler : QueueBase<MyModel>
{
public void OnRunExecuted(MyModel data) => ...;
}
But it's impossible to pass in a Queue name... is it?

See binding expressions:
In short, you can pass in a variable queue name as "%queue-name-variable%"
[FunctionName("QueueTrigger")]
public static void Run(
[QueueTrigger("%queue-name-variable%")]string myQueueItem,
ILogger log)
{
log.LogInformation($"C# Queue trigger function processed: {myQueueItem}");
}
Where input-queue-name is defined in your configuration like
{"queue-name-variable": "queue-name-in-current-env"}

As i remember attribute QueueTrigger accept only const string, so you can try make some tricks using environment variables like in post how to pass dynamic queue name

Test Brokered Message

Now i'm working at writing unit test on azure service bus trigger function
It's highly needed to mock somehow BrokeredMessage object that pass around into function. Function declaration is given below:
public static void Run(
[ServiceBusTrigger("saas01.queue.dbmigration", AccessRights.Manage, Connection = "connection")]BrokeredMessage message)
Unfortunately, i can't find any applicable way to mock it. It hardly mocking du to this class is sealed and i can't event create wrapper around it. Do you have some ideas about it?
Thanks for helping
,

One solution is to create a wrapper around BrokeredMessage you can test, as is done here. Here's also a MSDN post to the ServiceBus team that talks about using a wrapper too.
Note that Azure Functions V2 uses the Message class, which is public and not sealed.
[FunctionName("ServiceBusFunc")]
public static void Run([ServiceBusTrigger("myqueue", AccessRights.Manage, Connection = "ServiceBus")]BrokeredMessage myQueueItem, TraceWriter log)
{
var message = new MyBrokeredMessage(myQueueItem);
BusinessLogic(message, log);
}
public static void BusinessLogic(MyBrokeredMessage myMessage, TraceWriter log)
{
var stream = myMessage.GetBody<Stream>();
var reader = new StreamReader(stream);
log.Info($"C# ServiceBus queue trigger function processed message: '{reader.ReadToEnd() }'");
}
public class MyBrokeredMessage
{
private BrokeredMessage _msg;
public MyBrokeredMessage(BrokeredMessage msg) => _msg = msg;
public T GetBody<T>()
{
return _msg.GetBody<T>();
}
}