I have a rabbitmq setup with a producer and many consumers.
What would the best practice way to tell the consumers that the producer isn't able to send due to crash or some other failure?
In case of a failure in the producer I'd like to notify and show a fitting message to all consumers.
There is not an automatic way to do that, but in general, the messages systems are designed to decoupling the producers and the consumers. The basic idea is that the consumers don't know anything about producers.
Said, that you should handle the producers crashes and maybe adopt policies as publish confirm you want more control about your producers
I know it's kind of late to answer your question but here is my method for letting know the consumers that the producer is alive : you can add a ping message for example inside a Task which will publish to RabbitMQ each X seconds.
This solution works with ACKS back from RMQ and it works when you have a big number of messages coming in. This does not affect your performance by using ACKS
For example, taking the code in C# :
...
m_mainTimer = new System.Timers.Timer();
m_mainTimer.Interval = 10000; // every 10 secs
m_mainTimer.Elapsed += m_mainTimer_Elapsed;
m_mainTimer.AutoReset = false; // makes it fire only once
m_mainTimer.Start(); // Start
...
void m_mainTimer_Elapsed(object sender, System.Timers.ElapsedEventArgs e){
try {
// send to RMQ
sendMessageToRabbitMQ("PING", "error");
m_timerTaskSuccess = true;
} catch (Exception ex) {
m_timerTaskSuccess = false;
} finally {
if (m_timerTaskSuccess) {
m_mainTimer.Start();
}
}
}
The actual message in RMQ:
{
"Message": "PING",
"Timestamp": 1620303014184
}
If you don't get this message in less than 11 seconds you know there is a problem.
I hope it helps the others as well.
Related
I have a WPF application in which i want to return list of data or any data when user call it. Also i need to call WCF service to get data. What if service is down for any reason and i want to fixed broken service or wait for service alive and return the data. Let me show you what i am doing:
public List<MyData> GetMyData()
{
try
{
var data =GetOrCreateChannel().GetMyData(); //GetOrCreateChannel method create WCF service channel
return data;
}
catch(Exception ex)
{
_log.error(ex);
FixedBrokenService()
return GetMyData(); //Call again this method.
}
}
In above method, if service is not running, it will go to catch block and again call the same method until unless service is down. Whenever service get alive, it will return the data. I want to know is this approach is fine or not? What if service is down for 2 to 3 hour it wil recursivly call method and the stack size in memory will increasing. Is there any other approach?
What if service is down for 2 to 3 hour it wil recursivly call method and the stack size in memory will increasing. Is there any other approach?
I think you're asking because you already sense there might be some other way to improve what you've got so far; my guess is you're looking for some standard.
If so, I'd recommend Google's Exponential backoff guideline, here applied to Google Maps calls.
The idea is to introduce a delay between subsequent calls to the web service, increasing it in case of repeated failures.
A simple change would be:
public List<MyData> GetMyData()
{
List<MyData> data = null;
int delayMilliseconds = 100;
bool waitingForResults = true;
while (waitingForResults)
{
try
{
data = GetOrCreateChannel().GetMyData();
waitingForResults = false; // if this executes, you've got your data and can exit
}
catch (Exception ex)
{
_log.error(ex);
FixedBrokenService();
Thread.Sleep(delayMilliseconds); // wait before retrying
delayMilliseconds = delayMilliseconds * 2; // increase your delay
}
}
return data;
}
This way you won't have to deal with recursion either; don't forget to add
using System.Threading; to the top.
Since you mentioned WPF, we might want to take Jeroen's suggestion and wait in another thread: this means that your WPF GUI won't be frozen while you try reconnecting, but it will be enabled and perhaps show a spinner, a wait message or something like that (e.g. "Reconnecting in x seconds").
This requires changing the second to last line, i.e. Thread.Sleep(delayMilliseconds); to Wait(delayMilliseconds); and adding these two methods below GetMyData:
private async static Task Wait(int delayMilliseconds)
{
await WaitAsync(delayMilliseconds);
}
private static Task WaitAsync(int delayMilliseconds)
{
Thread.Sleep(delayMilliseconds);
return new Task(() => { });
}
Try using a wcf client with ClientBase (there are tons of examples). You can register to an event of the InnerChannel named InnerChannel.Faulted. When that event is called it means the service has failed somehow.
Instead if immediately retrying to connect in the catch you can write a separate thread which retries to connect with the client when the service has gone down.
I am creating a consumer that runs in an infinite loop to read messages from the queue. I am looking for advice/sample code on how to recover abd continue within my infinite loop even if there are network disruptions. The consumer has to stay running as it will be installed as a WindowsService.
1) Can someone please explain how to properly use these settings? What is the difference between them?
NetworkRecoveryInterval
AutomaticRecoveryEnabled
RequestedHeartbeat
2) Please see my current sample code for the consumer. I am using the .Net RabbitMQ Client v3.5.6.
How will the above settings do the "recovery" for me?
e.g. will consumer.Queue.Dequeue block until it is recovered?
That doesn't seem right
so...
Do I have to code for this manually? e.g. will consumer.Queue.Dequeue throw an exception for which I have to detect and manually re-create my connection, channel, and consumer? Or just the consumer, as "AutomaticRecovery" will recover the channel for me?
Does this mean I should move the consumer creation inside the while loop? what about the channel creation? and the connection creation?
3) Assuming I have to do some of this recovery code manually, are there event callbacks (and how do I register for them) to tell me that there are network problems?
Thanks!
public void StartConsumer(string queue)
{
using (IModel channel = this.Connection.CreateModel())
{
var consumer = new QueueingBasicConsumer(channel);
const bool noAck = false;
channel.BasicConsume(queue, noAck, consumer);
// do I need these conditions? or should I just do while(true)???
while (channel.IsOpen &&
Connection.IsOpen &&
consumer.IsRunning)
{
try
{
BasicDeliverEventArgs item;
if (consumer.Queue.Dequeue(Timeout, out item))
{
string message = System.Text.Encoding.UTF8.GetString(item.Body);
DoSomethingMethod(message);
channel.BasicAck(item.DeliveryTag, false);
}
}
catch (EndOfStreamException ex)
{
// this is likely due to some connection issue -- what am I to do?
}
catch (Exception ex)
{
// should never happen, but lets say my DoSomethingMethod(message); throws an exception
// presumably, I'll just log the error and keep on going
}
}
}
}
public IConnection Connection
{
get
{
if (_connection == null) // _connection defined in class -- private static IConnection _connection;
{
_connection = CreateConnection();
}
return _connection;
}
}
private IConnection CreateConnection()
{
ConnectionFactory factory = new ConnectionFactory()
{
HostName = "RabbitMqHostName",
UserName = "RabbitMqUserName",
Password = "RabbitMqPassword",
};
// why do we need to set this explicitly? shouldn't this be the default?
factory.AutomaticRecoveryEnabled = true;
// what is a good value to use?
factory.NetworkRecoveryInterval = TimeSpan.FromSeconds(5);
// what is a good value to use? How is this different from NetworkRecoveryInterval?
factory.RequestedHeartbeat = 5;
IConnection connection = factory.CreateConnection();
return connection;
}
RabbitMQ features
The documentation on RabbitMQ's site is actually really good. If you want to recover queues, exchanges and consumers, you're looking for topology recovery, which is enabled by default. Automatic Recovery (which is enabled by default) includes:
Reconnect
Restore connection listeners
Re-open channels
Restore channel listeners
Restore channel basic.qos setting, publisher confirms and transaction settings
The NetworkRecoveryInterval is the amount of time before a retry on an automatic recovery is performed (defaults to 5s).
Heartbeat has another purpose, namely to identify dead TCP connections. There are more to read about that at RabbitMQ's site.
Code sample
Writing reliable code for recovery is tricky. The EndOfStreamException is (as you suspect) most likely due to network problems. If you use the management plugin, you can reproduce this by closing the connection from there and see that the exception is triggered. For production-like applications, you might want to have a set of brokers that you alternate between in case of connection failure. If you have several RabbitMQ brokers, you might also want to guard yourself against long-term server failure on one or more of the servers. You might want to implement error strategies, like requeuing the message, or using a dead letter exchange.
I've been thinking a bit of these things and written a thin client, RawRabbit, that handles some of these things. Maybe it could be something for you? If not, I would suggest that you change the QueueingBasicConsumer to an EventingBasicConsumer. It is event driven, rather than thread blocking.
var eventConsumer = new EventingBasicConsumer(channel);
eventConsumer.Received += (sender, args) =>
{
var body = args.Body;
eventConsumer.Model.BasicAck(args.DeliveryTag, false);
};
channel.BasicConsume(queue, false, eventConsumer);
If you have topology recovery activated, the consumer will be restored by the RabbitMQ Client and start receiving messages again.
For more granular control, hook up event handlers for ConsumerCancelled and Shutdown to detect connectivity problems and Registered to know when the consumer can be used again.
I'm looking at using MassTransit to publish a event from one process to all other processes that care about that event. In my environment, several processes that are subscribed to that event could be running on the same box.
Based on the MassTransit's example code, here's a VERY simple proof-of-concept that does not attempt to deal with multi-cast (processes will also be distributed across many machines). Consider the "Message" below as an "Event", and that I want every instance of this process to get that message:
using System;
using MassTransit;
namespace BasicPubSub
{
public class Message
{
public String Text { get; set; }
}
public class Program
{
static void Main(string[] args)
{
//initialize the bus
var bus = ServiceBusFactory.New(sbc =>
{
sbc.UseMsmq();
//cause the DTC and MSMQ to get installed/fixed if needed.
sbc.VerifyMsDtcConfiguration();
sbc.VerifyMsmqConfiguration();
//sbc.UseMulticastSubscriptionClient();//Doesn't behave for a private queue.
sbc.ReceiveFrom("msmq://localhost/test_queue");
//Listen for interesting events.
sbc.Subscribe(subs =>
{
subs.Handler<Message>(msg => Console.WriteLine(msg.Text));
});
});
//stay open until the user types "exit"
var message = "";
do
{
message = Console.ReadLine();
//broadcast the message
bus.Publish(new Message { Text = message });
} while (message != "exit");
}
}
}
If I run multiple instances of this C# app, only one of the instances receives the message, but I need all of the instances to receive it. This behavior is consistent with the classical notion of a queue, so I'm fine with that.
What I am looking to know is whether I can use MSMQ/MassTransit to publish a message that all subscribers will receive, as opposed to just a single subscriber dequeuing it and the others not receiving it. Perhaps I'm trying to use a hammer and I need a paintbrush?
Just to be clear here, I am not looking at sharing a single queue for different kinds of messages, I'm looking to setup "peer-to-peer" pub-sub for a specific type of message, using MT and MSMQ.
I'm not sure about hammers or paintbrushes, but I think you're heading in the right direction.
To start with, each instance of a bus requires it's own queue to read from. If you are running multiple versions of this same program, make sure they are reading off different queues. Attempting to read from the same queue leads to Bad Stuff(TM).
Join the mailing list if you have more questions. https://groups.google.com/forum/#!forum/masstransit-discuss.
When publishing, all consumers should recieve the message once they have been subscribed. It might take a second or two for the subscription data to pass to all processes when starting up.
I would also suggest taking a look at RabbitMQ. Unless you absolutely need DTC, RabbitMQ is a much more robust messaging platform.
will the call to SQL in my multi threaded application cause the app to be bottleneck?
Is the logic i am using valid and conform to best practice?
static void Main(string[] args)
{
try
{
TcpListener listener = new TcpListener(IPAddress.Any, 8000);
TcpClient client;
listener.Start();
while (true) // Add your exit flag here
{
client = listener.AcceptTcpClient();
ThreadPool.QueueUserWorkItem(ThreadProc, client);
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.ReadLine();
}
}//Main
private static void ThreadProc(object obj)
{
var client = (TcpClient)obj;
// Do your work here...
NetworkStream ns = client.GetStream();
using (StreamWriter writer = new StreamWriter(ns))
{
writer.WriteLine("220 SMTP server ready.");
writer.Flush();
using (StreamReader reader = new StreamReader(ns))
{
//parse + persist smtp message here...
//save email message parts to a Email table in SQL SVR(from, to, body...)
}
}
}//ThreadProc
The way I would approach this is to have a Queue which you add the stuff you're going to parse and persist and then have that serviced by a pool of threads, or even just one thread, it depends on the expected throughput. The key here is the queue is serviced as soon as possible, but if there is a backlog it doesn't affect adding more items.
Edit:
There are a bunch of unknowns in your scenario, so my solution was equally vague, but I should of explained that.
The problem with your problem is I don't know how long your sql will run, or what action should be taken if it fails/times out, or how many of these sql statements you're expecting to queue up or run.
My solution would be to first extract the public facing interface for a) queuing an operation and b) processing an operation. this gives you the opportunity to, at a later date, or even now, switch out the queuing mechanism for something more appropriate.
So, take the case where by you are queuing work items, and the database stops responding. Suddenly nothing is being saved. Does that affect what your application does? Is there any sort of transactional issue here? Would it be better perhaps to be writing the work item to something like MSMQ? SQL Server also has a broker service which could be pretty spot on for what you want.
So I'm not going to add any code, but think in these terms:
1) Specify interfaces interface:
public interface QueueWorkItem
{
void AddWorkItemToQueue(WorkItem item);
}
public interface ConsumeWorkItem
{
bool ItemsInQueue {get;}
WorkItem GetNextItem();
}
You don't have to use these, it's just examples.
2) Build a class that implements the interfaces
This is the fun bit, you have to worry about threading and blocking (Look for producer consumer pattern) You could always use the threadpool and have the ConsumeWorkItem do nothing really.
Finally, lets talk about the threadpool. It's fine, but you don't have much control over it, and there's no guarantee that all the threads running in it are yours.
Hope that helps.
Needed:
A Windows Service That Executes Jobs from a Job Queue in a DB
Wanted:
Example Code, Guidance, or Best Practices for this type of Application
Background:
A user will click on an ashx link that will insert a row into the DB.
I need my windows service to periodically poll for rows in this table, and it should execute a unit of work for each row.
Emphasis:
This isn't completely new terrain for me.
EDIT: You can assume that I know how to create a Windows Service and basic data access.
But I need to write this service from scratch.
And I'd just like to know upfront what I need to consider.
EDIT: I'm most worried about jobs that fail, contention for jobs, and keeping the service running.
Given that you are dealing with a database queue, you have a fair cut of the job already done for you due to the transactional nature of databases. Typical queue driven application has a loop that does:
while(1) {
Start transction;
Dequeue item from queue;
process item;
save new state of item;
commit;
}
If processing crashes midway, the transaction rolls back and the item is processed on the next service start up.
But writing queues in a database is actually a lot trickier than you believe. If you deploy a naive approach, you'll find out that your enqueue and dequeue are blocking each other and the ashx page becomes unresponsive. Next you'll discover the dequeue vs. dequeue are deadlocking and your loop is constantly hitting error 1205. I strongly urge you to read this article Using Tables as Queues.
Your next challenge is going to be getting the pooling rate 'just right'. Too aggressive and your database will be burning hot from the pooling requests. Too lax and your queue will grow at rush hours and will drain too slowly. You should consider using an entirely different approach: use a SQL Server built-in QUEUE object and rely on the magic of the WAITFOR(RECEIVE) semantics. This allows for completely poll free self load tuning service behavior. Actually, there is more: you don't need a service to start with. See Asynchronous Procedures Execution for an explanation on what I'm talking about: launching processing asynchronously in SQL Server from a web service call, in a completely reliable manner. And finally, if the logic must be in C# process then you can leverage the External Activator, which allows the processing to be hosted in standalone processes as opposed to T-SQL procedures.
First you'll need to consider
How often to poll for
Does your service just stop and start or does it support pause and continue.
Concurrency. Services can increase the likelihood of a encountering a problem
Implementation
Use a System.Timers.Timer not a Threading.Timer
Maker sure you set the Timer.AutoReset to false. This will stop the reentrant problem.
Make sure to include execution time
Here's the basic framework of all those ideas. It includes a way to debug this which is a pain
public partial class Service : ServiceBase{
System.Timers.Timer timer;
public Service()
{
timer = new System.Timers.Timer();
//When autoreset is True there are reentrancy problme
timer.AutoReset = false;
timer.Elapsed += new System.Timers.ElapsedEventHandler(DoStuff);
}
private void DoStuff(object sender, System.Timers.ElapsedEventArgs e)
{
Collection stuff = GetData();
LastChecked = DateTime.Now;
foreach (Object item in stuff)
{
try
{
item.Dosomthing()
}
catch (System.Exception ex)
{
this.EventLog.Source = "SomeService";
this.EventLog.WriteEntry(ex.ToString());
this.Stop();
}
TimeSpan ts = DateTime.Now.Subtract(LastChecked);
TimeSpan MaxWaitTime = TimeSpan.FromMinutes(5);
if (MaxWaitTime.Subtract(ts).CompareTo(TimeSpan.Zero) > -1)
timer.Interval = MaxWaitTime.Subtract(ts).TotalMilliseconds;
else
timer.Interval = 1;
timer.Start();
}
protected override void OnPause()
{
base.OnPause();
this.timer.Stop();
}
protected override void OnContinue()
{
base.OnContinue();
this.timer.Interval = 1;
this.timer.Start();
}
protected override void OnStop()
{
base.OnStop();
this.timer.Stop();
}
protected override void OnStart(string[] args)
{
foreach (string arg in args)
{
if (arg == "DEBUG_SERVICE")
DebugMode();
}
#if DEBUG
DebugMode();
#endif
timer.Interval = 1;
timer.Start();
}
private static void DebugMode()
{
Debugger.Break();
}
}
EDIT Fixed loop in Start()
EDIT Turns out Milliseconds is not the same as TotalMilliseconds
You may want to have a look at Quartz.Net to manage scheduling the jobs. Not sure if it will fit your particular situation, but it's worth a look.
Some things I can think of, based on your edit:
Re: job failure:
Determine whether a job can be retried and do one of the following:
Move the row to an "error" table for logging / reporting later OR
Leave the row in the queue so that it will be reprocessed by the job service
You could add a column like WaitUntil or something similar to delay retrying the job after a failure
Re: contention:
Add a timestamp column such as "JobStarted" or "Locked" to track when the job was started. This will prevent other threads (assuming your service is multithreaded) from trying to execute the job simultaneously.
You'll need to have some cleanup process that goes through and clears stale jobs for re-processing (in the event the job service fails and your lock is never released).
Re: keeping the service running
You can tell windows to restart a service if it fails.
You can detect previous failure upon startup by keeping some kind of file open while the service is running and deleting it upon successful shutdown. If your service starts up and that file already exists, you know the service previously failed and can alert an operator or perform the necessary cleanup operations.
I'm really just poking around in the dark here. I'd strongly suggest prototyping the service and returning with any specific questions about the way it functions.