We Keep Coding

How to efficiently count HTTP Calls in asp.net core?

I have an abstract class called HttpHelper it has basic methods like, GET, POST, PATCH, PUT
What I need to achieve is this:
Store the url, time & date in the database each time the function is called GET, POST, PATCH, PUT
I don't want to store directly to the database each time the functions are called (that would be slow) but to put it somewhere (like a static queue-memory-cache) which must be faster and non blocking, and have a background long running process that will look into this cache-storage-like which will then store the values in the database.
I have no clear idea how to do this but the main purpose of doing so is to take the count of each calls per hour or day, by domain, resource and url query.
I'm thinking if I could do the following:
Create a static class which uses ConcurrentQueue<T> to store data and call that class in each function inside HttpHelper class
Create a background task similar to this: Asp.Net core long running/background task
Or use Hangfire, but that might be too much for simple task
Or is there a built-in method for this in .netcore?

Both Hangfire and background tasks would do the trick as consumers of the queue items.
Hangfire was there before long running background tasks (pre .net core), so go with the long running tasks for net core implementations.
There is a but here though.
How important is to you that you will not miss a call? If it is, then neither can help you.
The Queue or whatever static construct you have will be deleted the time your application crashes/machine restarts or just plain recycling of the application pools.
You need to consider some kind of external Queuing mechanism like rabbit mq with persistence on.
You can also append to a file, but that might also cause some delays as read/write.

I do not know how complex your problem is but I would consider two solutions.
First is calling Async Insert Method which will not block your main thread but will start task. You can return response without waiting for your log to be appended to database. Since you want it to be implemented in only some methods, I would do it using Attributes and Middleware.
Simplified example:
public IActionResult SomePostMethod()
{
LogActionAsync("This Is Post Method");
return StatusCode(201);
}
public static Task LogActionAsync(string someParameter)
{
return Task.Run(() => {
// Communicate with database (X ms)
});
}
Better solution is creating buffer which will not communicate with database each time but only when filled or at interval. It would look like this:
public IActionResult SomePostMethod()
{
APILog.Log(new APILog.Item() { Date = DateTime.Now, Item1 = "Something" });
return StatusCode(201);
}
public partial class APILog
{
private static List<APILog.Item> _buffer = null;
private cont int _msTimeout = 60000; // Timeout between updates
private static object _updateLock = new object();
static APILog()
{
StartDBUpdateLoopAsync();
}
private void StartDBUpdateLoopAsync()
{
// check if it has been already and other stuff
Task.Run(() => {
while(true) // Do not use true but some other expression that is telling you if your application is running.
{
Thread.Sleep(60000);
lock(_updateLock)
{
foreach(APILog.Item item in _buffer)
{
//Import into database here
}
}
}
});
}
public static void Log(APILog.Item item)
{
lock(_updateLock)
{
if(_buffer == null)
_buffer = new List<APILog.Item>();
_buffer.Add(item);
}
}
}
public partial class APILog
{
public class Item
{
public string Item1 { get; set; }
public DateTime Date { get; set; }
}
}
Also in this second example I would not call APILog.Log() each time but use Middleware in combination with Attribute

Logging Hangfire jobs to Application Insights and correlating activity to an Operation Id

I feel like this should be a lot simpler than it's turning out to be, or I am just over thinking it too much.
I have a .NET Core 3.1 Web API application, which is using HangFire to process some jobs in the background. I have also configured Application Insights to log Telemetry from the .NET Core API.
I can see logging events and dependency telemetry data logged in Application Insights. However, each event/log/dependency is recorded against a unique OperationId and Parent Id.
I am trying to determine how to ensure that any activity which is logged, or any dependencies which are used in the context of the background job are logged against the OperationId and/or Parent Id of the original request which queued the background job.
When I queue a job, I can get the current OperationId of the incoming HTTP request, and I push that into the HangFire queue with the job. When the job is then performed, I can get back that OperationId. What I then need to do is make that OperationID available throughout the context/lifetime of the job execution, so that it is attached to any Telemetry sent to Application Insightd.
I thought I could create a IJobContext interface, which could be injected into the class which performs the job. Within that context I could push the OperationID. I could then create a ITelemetryInitializer which would also take the IJobContext as a dependency. In the ITelemetryInitializer I could then set the OperationID and ParentId of the telemetry being sent to Application Insights. Here's some simple code:
public class HangFirePanelMessageQueue : IMessageQueue
{
private readonly MessageProcessor _messageProcessor;
private readonly IHangFireJobContext _jobContext;
private readonly TelemetryClient _telemetryClient;
public HangFirePanelMessageQueue(MessageProcessor panelMessageProcessor,
IIoTMessageSerializer iotHubMessageSerialiser,
IHangFireJobContext jobContext, TelemetryClient telemetryClient)
{
_messageProcessor = panelMessageProcessor;
_jobContext = jobContext;
_telemetryClient = telemetryClient;
}
public async Task ProcessQueuedMessage(string message, string operationId)
{
var iotMessage = _iotHubMessageSerialiser.GetMessage(message);
_jobContext?.Set(iotMessage.CorrelationID, iotMessage.MessageID);
await _messageProcessor.ProcessMessage(iotMessage);
}
public Task QueueMessageForProcessing(string message)
{
var dummyTrace = new TraceTelemetry("Queuing message for processing", SeverityLevel.Information);
_telemetryClient.TrackTrace(dummyTrace);
string opId = dummyTrace.Context.Operation.Id;
BackgroundJob.Enqueue(() =>
ProcessQueuedMessage(message, opId));
return Task.CompletedTask;
}
}
The IJobContext would look something like this:
public interface IHangFireJobContext
{
bool Initialised { get; }
string OperationId { get; }
string JobId { get; }
void Set(string operationId, string jobId);
}
And then I would have an ITelemetryInitializer which enriches any ITelemetry:
public class EnrichBackgroundJobTelemetry : ITelemetryInitializer
{
private readonly IHangFireJobContext jobContext;
public EnrichBackgroundJobTelemetry(IHangFireJobContext jobContext)
{
this.jobContext = jobContext;
}
public void Initialize(ITelemetry telemetry)
{
if (!jobContext.Initialised)
{
return;
}
telemetry.Context.Operation.Id = jobContext.OperationId;
}
}
The problem I have however is that the ITelemetryInitializer is a singleton, and so it would be instantiated once with a IHangFireJobContext which would then never update for any subsequent HangFire job.
I did find the https://github.com/skwasjer/Hangfire.Correlate project, which extends https://github.com/skwasjer/Correlate. Correlate creates a correlation context which can be accessed via a ICorrelationContextAccessor which is similar to the IHttpContextAccessor.
However, the footnotes for Correlate state "Please consider that .NET Core 3 now has built-in support for W3C TraceContext (blog) and that there are other distributed tracing libraries with more functionality than Correlate." which lists Application Insights as one of the alternatives for more Advanced distributed tracing.
So can anyone help me understand how I can enrich any Telemetry going to Application Insights when it is created within the context of a HangFire job? I feel the correct answer is to use an ITelemetryInitializer and populate the OperationId on that ITelemetry item, however, I am not sure what dependancy to inject into the ITelemetryInitialzer in order to get access to the HangFire Job Context.

When I queue a job, I can get the current OperationId of the incoming HTTP request, and I push that into the HangFire queue with the job.
So, am I correct to say that you have a controller action that pushes work to hangfire? If so What you can do is inside the controller method get the operation id and pass it to the job. Use that operation id to start a new operation using the operation Id. That operation, together with all the telemetry generated during that operation, will be linked to the original request.
I have no hangfire integration but the code below shows the general idea: some work is queued to be done in the background and should be linked to the request regarding the telemetry:
[HttpGet("/api/demo5")]
public ActionResult TrackWorker()
{
var requestTelemetry = HttpContext.Features.Get<RequestTelemetry>();
_taskQueue.QueueBackgroundWorkItem(async ct =>
{
using(var op = _telemetryClient.StartOperation<DependencyTelemetry>("QueuedWork", requestTelemetry.Context.Operation.Id))
{
_ = await new HttpClient().GetStringAsync("http://blank.org");
await Task.Delay(250);
op.Telemetry.ResultCode = "200";
op.Telemetry.Success = true;
}
});
return Accepted();
}
The full example can be found here.

Working from Peter Bons' example I did it like this:
Code originally triggered from a controller action:
// Get the current ApplicationInsights Id. Could use .RootId if
// you only want the OperationId, but I want the ParentId too
var activityId = System.Diagnostics.Activity.Current?.Id;
_backgroundJobClient.Enqueue<JobDefinition>(x =>
x.MyMethod(queueName, otherMethodParams, activityId));
In my JobDefinition class:
// I use different queues, but you don't need to.
// otherMethodParams is just an example. Have as many as you need, like normal.
[AutomaticRetry(OnAttemptsExceeded = AttemptsExceededAction.Delete, Attempts = 10)]
[QueueNameFromFirstParameter]
public async Task MyMethod(string queueName, string otherMethodParams,
string activityId)
{
var (operationId, parentId) = SplitCorrelationIdIntoOperationIdAndParentId(
activityId);
// Starting this new operation will initialise
// System.Diagnostics.Activity.Current.
using (var operation = _telemetryClient.StartOperation<DependencyTelemetry>(
"JobDefinition.MyMethod", operationId, parentId))
{
try
{
operation.Telemetry.Data = $"something useful here";
// If you have other state you'd like in App Insights logs,
// call AddBaggage and they show up as a customDimension,
// e.g. in any trace logs.
System.Diagnostics.Activity.Current.AddBaggage("QueueName", queueName);
// ... do the real background work here...
operation.Telemetry.Success = true;
}
catch (Exception)
{
operation.Telemetry.Success = false;
throw;
}
}
}
// Splits full value from System.Diagnostics.Current.Activity.Id
// like "00-12994526f1cb134bbddd0f256e8bc3f0-872b3bd78c345a46-00"
// into values ( "12994526f1cb134bbddd0f256e8bc3f0", "872b3bd78c345a46" )
private static (string, string) SplitCorrelationIdIntoOperationIdAndParentId(string activityId)
{
if (string.IsNullOrEmpty(activityId))
return (null, null);
var splits = activityId.Split('-');
// This is what should happen
if (splits.Length >= 3)
return (splits[1], splits[2]);
// Must be in a weird format. Try to return something useful.
if (splits.Length == 2)
return (splits[0], splits[1]);
return (activityId, null);
}
I'm not sure using the OperationId and ParentId is quite right here, e.g. it does tie the background job to the originating request's OperationId, but if the originating Request has a ParentId then this background job should really have its ParentId set as the Request, not as the Request's ParentId. Anyone know?

Hangfire .NET Core - Get enqueued jobs list

Is there a method in the Hangfire API to get an enqueued job (probably by a Job id or something)?
I have done some research on this, but I could not find anything.
Please help me.

I have found the answer in the official forum of Hangfire.
Here is the link:
https://discuss.hangfire.io/t/checking-for-a-job-state/57/4
According to an official developer of Hangfire, JobStorage.Current.GetMonitoringApi() gives you all the details regarding Jobs, Queues and the configured servers too!
It seems that this same API is being used by the Hangfire Dashboard.
:-)

I ran into a case where I wanted to see ProcessingJobs, EnqueuedJobs, and AwaitingState jobs for a particular queue. I never found a great way to do this out of the box, but I did discover a way to create a "set" of jobs in Hangfire. My solution was to add each job to a set, then later query for all items in the matching set. When the job reaches a final state, remove the job from the set.
Here's the attribute to create the set:
public class ProcessQueueAttribute : JobFilterAttribute, IApplyStateFilter
{
private readonly string _queueName;
public ProcessQueueAttribute()
: base() { }
public ProcessQueueAttribute(string queueName)
: this()
{
_queueName = queueName;
}
public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
{
if (string.IsNullOrEmpty(context.OldStateName))
{
transaction.AddToSet(_queueName, context.BackgroundJob.Id);
}
else if (context.NewState.IsFinal)
{
transaction.RemoveFromSet(_queueName, context.BackgroundJob.Id);
}
}
public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction) { }
}
You decorate your job this way:
[ProcessQueue("queueName")]
public async Task DoSomething() {}
Then you can query that set as follows:
using (var conn = JobStorage.Current.GetConnection())
{
var storage = (JobStorageConnection)conn;
if (storage != null)
{
var itemsInSet = storage.GetAllItemsFromSet("queueName");
}
}

Use events to forward exceptions for producer/consumer using Dataflow

I’m trying to implement a producer/consumer queue using Dataflow for HTTP requests towards a web service. I found an excellent post from Stephen Cleary, which is covering exactly this scenario. However, in contrast to Stephen’s post, I cannot mark the producer queue as complete since clients shall be able to enqueue requests throughout the entire lifetime of the application. The idea behind this approach that the client can constantly produce requests and the consumer is able to handle requests differently if more than 1 request is pending (which is required).
This requirement leads also to the fact that the consumption of the requests cannot be started after the production was finished, but have to be started the first request was enqueued. This also requires me to start the consumption in a non-blocking way (otherwise it would lead to a deadlock). I’ve done this via an async-call which is not awaited, which unfortunately hampers the exception handling. Exceptions occurring during the consumption (implementing the HTTP requests) cannot bubble up since the call of the consume-function is not awaited. I’ve introduced and event to deal with this kind of problem, but this leads me to the following questions:
Is it a good idea to use an event to forward exceptions from the consumer to the client of the producer?
Is this a good idea to implement the producer/consumer pattern in that fashion for my use case?
Are there potentially other approaches, which are more beneficial under the given circumstances?
To make it a more explicit, I’ve prepared a code example illustrating the problem I described above:
public class HttpConnector
{
private BufferBlock<RequestPayload> queue;
public delegate void ConnectorExceptionHandler(object sender, Exception e);
public event ConnectorExceptionHandler ConnectorExceptionOccured;
public Task<bool> ProduceRequest(RequestPayload payload)
{
if(this.queue == null)
{
this.queue = new BufferBlock<RequestPayload>();
this.ConsumeRequestsAsync(queue); //this call cannot be awaited since it would lead to a deadlock
//however, by not awaiting this call all exceptions raised in
//ConsumeRequestsAsync will be lost
}
return await queue.SendAsync(payload)
}
public Task ConsumeRequestsAsync(BufferBlock<RequestPayload> queue)
{
while(await queue.OutputAvailableAsync())
{
try
{
var payload = await queue.ReceiveAsync();
//do the HTTP request...
}
catch (Exception e)
{
ConnectorExceptionOccured(this, e); //fire event to forward the exception to the client
}
}
}
}
public class Client
{
private HttpConnector connector = new HttpConnector();
public Task<bool> UpdateDataAsync()
{
connector += (object sender, Exception e ) //register an event handler to receive exceptions occur
//during the consumption of the requests
{
//handle exception or re-throw
};
connector.ProduceRequest(new RequestPayload()); //produce a request
}
}

Forwarding exceptions via an event has some severe drawbacks:
Natural exception handling is not possible. If developers are aware of this mechanism, they won't catch any exception.
You cannot use AppDomain#UnhandledException for unhandled exceptions during the application runtime. In fact, if you don't have a subscription to the 'Exception'-event, the exception is completely lost.
If you have only one event to subscribe to, your exception object needs a lot of context information in order to figure out which operation caused the exception.
For our problem it turned out that it is better to use TaskCompletionSource, which is a standard technique to synchronize different threads. An instance of TaskCompletionSource class is provided by each RequestPayload object. After the consumption the TaskCompletionSource.Task is completed (either with the result or with an exception). The producer doesn't return the Task for queue.SendAsync(payload) but payload.CompletionSource.Task:
public class RequestPayload
{
public IModelBase Payload { get; set; }
public TaskCompletionSource<IResultBase> CompletionSource { get; private set; }
}
public class HttpConnector
{
private BufferBlock<RequestPayload> queue;
public Task ProduceRequest(RequestPayload payload)
{
if(this.queue == null)
{
this.queue = new BufferBlock<RequestPayload>();
this.ConsumeRequestsAsync(queue);
}
await queue.SendAsync(payload);
return await payload.CompletionSource.Task;
}
public Task ConsumeRequestsAsync(BufferBlock<RequestPayload> queue)
{
while(await queue.OutputAvailableAsync())
{
try
{
var payload = await queue.ReceiveAsync();
//do the HTTP request...
payload.CompletionSource.TrySetResult(null);
}
catch (Exception e)
{
payload.CompletionSource.TrySetException(e)
}
}
}
}
public class Client
{
private HttpConnector connector = new HttpConnector();
public Task UpdateDataAsync()
{
try
{
await connector.ProduceRequest(new RequestPayload());
}
catch(Exception e) { /*handle exception*/ }
}
}

Suggest a design pattern for implementing fail over mechanism

I have an application(say App1) which is connected to another application (App2) via .net remoting. App2 acts as a server.. If App2 goes down App1 will not be able to pull data from App2. We are planning to run an instance of App2(say App2a) in another machine so that if App2 goes down App1 automatically takes the data from App2a. When App2 runs again.. App1 will need to take the data from App2. The fail over mechanism is not implemented yet... Please suggest a design pattern so that in future any number of server instances can be added for App1 to pull data.
Thanks

The closest design pattern that I can think of is the Chain of Responsibility pattern.
The idea is that:
You build a chain of objects (servers)
Let the object (server) handle the request
If it is unable to do so, pass the request down the chain
Code:
// Server interface
public interface IServer
{
object FetchData(object param);
}
public class ServerProxyBase: IServer
{
// Successor.
// Alternate server to contact if the current instance fails.
public ServerBase AlternateServerProxy { get; set; }
// Interface
public virtual object FetchData(object param)
{
if (AlternateServerProxy != null)
{
return AlternateServerProxy.FetchData(param);
}
throw new NotImplementedException("Unable to recover");
}
}
// Server implementation
public class ServerProxy : ServerProxyBase
{
// Interface implementation
public override object FetchData(object param)
{
try
{
// Contact actual server and return data
// Remoting/WCF code in here...
}
catch
{
// If fail to contact server,
// run base method (attempt to recover)
return base.FetchData(param);
}
}
}
public class Client
{
private IServer _serverProxy;
public Client()
{
// Wire up main server, and its failover/retry servers
_serverProxy = new ServerProxy("mainserver:2712")
{
AlternateServerProxy = new ServerProxy("failover1:2712")
{
AlternateServerProxy = new ServerProxy("failover2:2712")
}
};
}
}
This example wires up a chain of 3 servers (mainserver, failover1, failover2).
The call the FetchData() will always attempt to go to mainserver.
When it fails, it'll then attempt failover1, followed by failover2, before finally throwing an exception.
If it were up to me, I wouldn't mind using something quick and dirty such as:
public class FailoverServerProxy: IServer
{
private readonly List<ServerProxy> _servers;
public FailoverServerProxy RegisterServer(Server server)
{
_servers.Add(server);
return this;
}
// Implement interface
public object FetchData(object param)
{
foreach(var server in _servers)
{
try
{
return server.FetchData(param);
}
catch
{
// Failed. Continue to next server in list
continue;
}
}
// No more servers to try. No longer able to recover
throw new Exception("Unable to fetch data");
}
}
public class Client
{
private IServer _serverProxy;
public Client()
{
// Wire up main server, and its failover/retry servers
_serverProxy = new FailoverServerProxy()
.RegisterServer("mainserver:2712")
.RegisterServer("failover1:2712")
.RegisterServer("failover2:2712");
}
}
I think it borrows ideas from other patterns such as Facade, Strategy and Proxy.
But my motivations are simply to:
Make the least impact on existing classes (ie, No extra property in the Server class)
Separation of concerns:
Central class for the server's failover/recovery logic.
Keep the failover/recovery's implementation hidden from the Client/Server.