I am trying to use mass transit for request response handling. Most examples for mass transit are for console application or web application and I don't know how to start or stop bus on producer when I use it in class library.
Because in examples for web application bus started on application start but for class library there are no such a thing like startup.cs.
My question is where to start bus or stop when I use class library for connecting to bus?
My producer code looks like
IBusControl busControl = CreateBus();
TaskUtil.Await(() => busControl.StartAsync());
IRequestClient<IAccountingRequest, IAccountingResponse> client = CreateRequestClient(busControl);
IAccountingResponse response = null;
AccountingRequest accountingRequest = MapToAccountingRequest(accountingIntegration);
Task.Run(async () =>
{
response = await client.Request(accountingRequest);
}).Wait();
busControl.Stop();
But I think starting and stopping bus for every request is not good.
You should provide methods in your class library to Start/Stop the bus. You can abstract them however you want, but allow the developer the ability to startup and shutdown the bus. Many other libraries to this, typically via a method to Stop, Shutdown, Close, etc.
The fact that you're also hiding the Task in the above example and blocking/waiting makes me thing you're stuck within something super legacy that you can't get avoid. In this case, well, I had to say this, but manage the reference to the bus in some static singleton (yuck), and Start it the first time it's used (double yuck), then, try to find a hook on application exit to stop it clean (good luck).
The best solution is to give the developers a call into your library to shut it down so they can free the connection and resources.
Related
I'm trying to implement a basic pub/sub system with dynamic subscribers. I need to dynamically register a topic subscriber in my .NET APIs, but it seems like I can only do that manually from the Azure Portal. When my program starts, I want to be able to register a subscriber to a topic in the format of subscribername-{timestamp} because I want to be able to deploy as many staging/dev versions as I want without having to manually create these subscribers each time.
I feel like this is a fundamental feature that I'm just blindly missing. I can do this when working with queues, but if I try to do the same with a topic, I get continuous errors of that subscriber path not found. I have searched the internet to no end and while I have found SOME solutions, they are very old and often not compatible with .NET 5 or the package is deprecated. I'm feeling like I'm going against the grain and missing something with what I'm coming up with, so I'd like to get some input on what the proper practice is for this.
I'm using Azure.Messaging.ServiceBus for publishing and subscribing currently. Below is some code -
var processor = ServiceBusClient.CreateProcessor(TopicName, $"DynamicSubscriber-{DateTime.Now}");
try
{
processor.ProcessErrorAsync += ErrorHandler;
processor.ProcessMessageAsync += MessageHandler;
await processor.StartProcessingAsync();
}
catch (Exception e)
{
await processor.DisposeAsync();
await ServiceBusClient.DisposeAsync();
}
finally
{
Console.WriteLine("Press a key to exit.");
Console.ReadLine();
}
Thank You #PeterBons! Yes, when updating/creating/fetching/deleting the Service Bus entities, ServiceBusAdministrationClient is the client Class to be used.
Also, There are few error details given this article when using the method of Queue with ServiceBusAdministrationClient and this SO Thread.
The ServiceBusTopicSubscription class is used to setup the Azure Service bus subscription. The class uses the ServiceBusClient to set up the message handler, the ServiceBusAdministrationClient is used to implement filters and add or remove these rules. The Azure.Messaging.ServiceBus Nuget package is used to connect to the subscription.
Sorry about the vague title, it's rather hard to explain. I have the following setup:
I'm running a .NET Core 2.2 Web API hosted in Service Fabric.
Part of this API's responsibilities is to monitor an external FTP storage for new incoming files.
Each file will trigger a Mediator Command to be invoked with processing logic.
I've implemented a hybrid solution based on https://learn.microsoft.com/en-us/dotnet/architecture/microservices/multi-container-microservice-net-applications/background-tasks-with-ihostedservice and https://blog.maartenballiauw.be/post/2017/08/01/building-a-scheduled-cache-updater-in-aspnet-core-2.html. In essence this is an IHostedService implementation that is registered in the Startup.cs of this API. Its basically a background service running in-process.
As for the problem. The solution above works fine on a 1-node cluster, but causes "duplicates" to be processed when running on a 5-node cluster. The problem lies in the fact that on a 5-node cluster, there are ofcourse 5 identical ScheduledTasks running and will all access the same file on the FTP at the same time.
I've realised this is caused somewhat by improper separation of concerns - aka the API shouldn't be responsible for this, rather a completely separate process should handle this.
This brings me to the different services supported on Service fabric (Stateful, Stateless, Actors and Hosted Guest Exe's). The Actor seems to be the only one that runs single-threaded, even on a 5-node cluster. Additionally, an Actor doesn't seem to be well suited for this kind of scenario, as it needs to be triggered. In my case, I basically need a daemon that runs all the time on a schedule. If I'm not mistaken, the other stateful/stateless services will run with 5 "clones" as well and just cause the same issue as I currently have.
I guess my question is: how can I do efficient background processing with Service Fabric and avoid these multi-threaded/duplicate issues? Thanks in advance for any input.
In service farbic you have 2 options with actors:
Reliable actor timers
Reliable actor reminders
You can use the state to determine if the actor has processed your ftp file.
Have a look at this blog post, to see how they used a reminder to run every 30 seconds.
It's important that the code in your actor allows reantrancy.
Basically because the actors are reliable, your code might get executed multiple times and be canceled in the middle of an execution.
Instead of doing this:
public void Method()
{
_ftpService.Process(file);
}
Consider doing this:
public void Method(int fileId)
{
if (_ftpService.IsNotProcessed(fileId))
{
_ftpService.Process(file);
_ftpService.SetProcessed(fileId);
}
}
If your actor has trouble disposing, you might want to check if you are handling cancelationtokens in your code. I never had this issue, but we are using autofac, with Autofac.ServiceFabric to register our actors with RegisterActor<T>() and we have cancelationtokens in most of our logic. Also the documentation of CancellationTokenSource can help you.
Example
public Ctor()
{
_cancelationTokenSource = new CancellationTokenSource();
_cancellationToken= _cancelationTokenSource.Token;
}
public async Task SomeMethod()
{
while(/*condition*/)
{
_cancellationToken.ThrowIfCancellationRequested();
/*Other code*/
}
}
protected override async Task OnDeactivateAsync()
{
_cancelationTokenSource.Cancel();
}
Problem: If the DB is offline when this service is started, this service will not start as it fails inside this line: var container = new BootStrapper().Container; on start.
private static void Main(string[] args)
{
Logger.Info("Engine Service is bootstrapping...");
AppDomain.CurrentDomain.UnhandledException += UncaughtExceptions.DomainException;
Directory.SetCurrentDirectory(AppDomain.CurrentDomain.BaseDirectory);
var container = new BootStrapper().Container;
var controller = container.Resolve<EngineController>();
ServiceBase.Run(controller.MainView as ServiceBase);
container.Dispose();
}
The reason it fails there is that it runs this code where it adds the nhibernate facility container.AddFacility<NHibernateFacility>(); and fails with a connection timeout.
public void Install(IWindsorContainer container, IConfigurationStore store)
{
var isAutoTxFacilityRegistered = container.Kernel.GetFacilities().Any(f => f is AutoTxFacility);
if (!isAutoTxFacilityRegistered) container.AddFacility<AutoTxFacility>();
container.Register(
Component.For<INHibernateInstaller>().ImplementedBy<CieFluentInstaller>().IsDefault().LifestyleTransient(),
Classes.FromThisAssembly().Pick().WithService.DefaultInterfaces().LifestyleTransient()
);
var isNHibernateFacilityRegistered = container.Kernel.GetFacilities().Any(f => f is NHibernateFacility);
if (!isNHibernateFacilityRegistered) container.AddFacility<NHibernateFacility>();
}
If the windows service start up takes longer than 30 seconds (which it may if updates or backups are being conducted on the DB) the app service fails to start.
I'm using FluentNhibernate, NHibernate, Castle Windsor with NHibernateFacility.
Things I've tried:
Can't do it from the service start event because it fails before it
gets to the view or controller. The view and controller have no
direct access to the IoC container, only via an injected IoCFactory
as per Castle Windsor recommendations.
I've tried to spawn a thread in the main and start it off there with
a retry loop but because the service "waits" inside the
ServiceBase.Run method, I can't seem to get the correct returns to
make it "fake start" while in a retry loop.
Investigated lengthening the service start timeout, but can't access
the servicebase/view since it fails before then and a system wide
change at hundreds of production sites is not an option.
Question: How can I make it so that the windows service "starts" when DB is offline given the design?
You need to divide your startup actions into two categories:
Actions that must happen fairly immediately and/or won't fix themselves
in case of failure. Things such as a mandatory configuration file
missing, for which administrator intervention would be required.
Actions that we're OK to delay, or - more importantly - actions that can
fail due to transient errors. Such errors can be network failure or that
we happened to start somewhat faster than the database server after a
reboot.
You service OnStart code should follow this basic structure:
OnStart:
Perform the immediate category 1 tasks and exit if any of these fail.
Launch the main application thread.
One approach to the "main application thread" is to follow this basic
structure:
ManualResetEvent shutdownRequestedEvent = new ManualResetEvent()
RealMain:
while (!shutdownRequestedEvent.WaitOne(0) && !bootstrapPerformed)
{
try
{
PerformBootstrap()
bootstrapPerformed = true
}
catch (Exception ex)
{
LogError(ex)
}
if (!bootstrapPerformed)
shutdownRequestedEvent.WaitOne(some timeout)
}
Second bootstrap action similar to above, etc.
Third bootstrap action similar to above, etc.
Eventually, start performing real work, while listening to
the shutdownRequestedEvent.
The services OnShutdown would signal the shutdownRequestedEvent and then
wait for the RealMain thread to exit.
If the RealMain thread serves no purpose other then setup, it should perhaps
be allowed to exit when it's done with all bootstrap tasks.
Another thing to be careful about is to make sure your service, during normal operation, can withstand the temporary loss of access to a resource due to transient errors. For example, your service shouldn't crash just because someone reboots the database server. It should just wait patiently and retry forever.
An alternative approach that can work in some cases is to handle the bootstrapping as a dependency of whatever the real task is. For instance, launch the real task, the real task will request a database session, to get that we must have the session factory, if we don't yet have the session factory, launch the session factory initialization. If the session factory
cannot be created, exception bubbles up and the whole task fails. The remaining
work is now to wait a little while and then retry the task. Repeat forever.
Turned out to be a bug in NHibernate that prevented doing any of the above. Between Nibernate 2.0 and 3.0 you have to add the following to the NHibernate v3.0+ config (or in this case the FluentNHibernate):
cfg.SetProperty("hbm2ddl.keywords", "none");
This allows NHibernate to properly bootstrap itself and get to the controller now without error.
My project was standalone application then I decided to split it as client & server because I need powerful CPU usage and portability at the same time. Now multiple clients can connect to one server.
It was easy when 1 by 1 processing did the job. Now I need to call the same function & scope area again & again at the same time -via client requests-
Please can anyone give me some clue how should I handle these operations, I need to know how can I isolate clients' processes from each other at the server side? My communication is asynchronous, server receives a request and starts a new thread. I think I pass a parameter which one carries the client information, and another parameter as job id -to help client back, client may ask for multiple jobs and some jobs finish quicker than others-
Should I instantiate the class Process on each call? Can I use a static method, etc, any explanation will be of great help!
Below is the part of my code to need modification
class static readonly Data
{
public variable listOfValues[]
}
class Process
{
local variable bestValue
function findBestValue(from, to)
{
...
if(processResult > bestValue) bestValue = processResult
...
}
...
for(i=0;i<10;i++) startThread(findBestValue(i*1000,i*1000+999));
...
}
EDIT: I think I have to instantiate a
new Process class and call the
function for each client and ignore
the same client for same job since job is already running.
Not getting into your application design, since you didn't talk much about it, I think that your problem is ideal for using WCF WebServices. You get client isolation by design because every request will start in it's own thread. You can create WCF host as standalone application/windows service.
You can wrap your communication with WCF service and configure it to be PerCall service (meaning each request will be processed separately from others).
So you'll clean up your buisness logic from syncronization stuff. That's the best way, because managing and creating threads is not difficult to implement, but it is difficult to implement correctly and optimized for resources consumption.
I have a console application that starts up, hosts a bunch of services (long-running startup), and then waits for clients to call into it. I have integration tests that start this console application and make "client" calls. How do I wait for the console application to complete its startup before making the client calls?
I want to avoid doing Thread.Sleep(int) because that's dependent on the startup time (which may change) and I waste time if the startup is faster.
Process.WaitForInputIdle works only on applications with a UI (and I confirmed that it does throw an exception in this case).
I'm open to awkward solutions like, have the console application write a temp file when it's ready.
One option would be to create a named EventWaitHandle. This creates a synchronization object that you can use across processes. Then you have your 'client' applications wait until the event is signalled before proceeding. Once the main console application has completed the startup it can signal the event.
http://msdn.microsoft.com/en-us/library/41acw8ct(VS.80).aspx
As an example, your "Server" console application might have the following. This is not compiled so it is just a starting point :)
using System.Threading;
static EventWaitHandle _startedEvent;
static void main()
{
_startedEvent = new EventWaitHandle(false, EventResetMode.ManualReset, #"Global\ConServerStarted");
DoLongRunnningInitialization();
// Signal the event so that all the waiting clients can proceed
_startedEvent.Set();
}
The clients would then be doing something like this
using System.Threading;
static void main()
{
EventWaitHandle startedEvent = new EventWaitHandle(false, EventResetMode.ManualReset, #"Global\ConServerStarted");
// Wait for the event to be signaled, if it is already signalled then this will fall throught immediately.
startedEvent.WaitOne();
// ... continue communicating with the server console app now ...
}
What about setting a mutex, and removing it once start up is done. Have the client app wait until it can grab the mutex before it starts doing things.
Include an is ready check in the app's client interface, or have it return a not ready error if called before it's ready.
Create a WCF service that you can use for querying the status of the server process. Only start this service if a particular command is passed on the command line. The following traits will ensure a very fast startup of this service:
Host this service as the first operation of the client application
Use the net.tcp or net.pipe binding because they start very quickly
Keep this service as simple as possible to ensure that as long as the console application doesn't terminate, it will remain available
The test runner can attempt to connect to this service. Retry the attempt if it fails until the console application terminates or a reasonably short timeout period expires. As long as the console application doesn't terminate unexpectedly you can rely on this service to provide any additional information before starting your tests in a reasonably short period of time.
Since the two(the console application, and integration test app that makes client calls - as I understand) are separate application, so there should be a mechanism - a bridge - that would tell play as a mediator(socket, external file, registry, etc).
Another possibility could be that you come up with an average time the console takes to load the services and use that time in your test app; well, just thinking out loud!