As everything fail one day or the other. Are there any recommendations/best practices on how to handle errors when publishing messages to Amazon SQS?
I am running the Amazon .NET SDK and send a couple of 1000 SQS messages a day. It hasnt come to my attention that publishing has failed but that could be that any problem hasent surfaced.
However, how should I handle an error in the following basic code (pretty much a straight forward usage example from the SDK documentation):
public static string sendSqs(string data)
{
IAmazonSQS sqs = AWSClientFactory.CreateAmazonSQSClient(RegionEndpoint.EUWest1);
SendMessageRequest sendMessageRequest = new SendMessageRequest();
CreateQueueRequest sqsRequest = new CreateQueueRequest();
sqsRequest.QueueName = "mySqsQueue";
CreateQueueResponse createQueueResponse = sqs.CreateQueue(sqsRequest);
sendMessageRequest.QueueUrl = createQueueResponse.QueueUrl;
sendMessageRequest.MessageBody = data;
SendMessageResponse sendMessageresponse = sqs.SendMessage(sendMessageRequest);
return sendMessageresponse.MessageId;
}
First (kinda unrelated) I would recommend separating the client from the send message:
public class QueueStuff{
private static IAmazonSQS SQS;
//Get only one of these
public QueueStuff(){
SQS = AWSClientFactory.CreateAmazonSQSClient(RegionEndpoint.EUWest1);
}
//...use SQS elsewhere...
Finally to answer your question: check the Common Errors and SendMessage (in your case) pages and catch relevant exceptions. What you do will depend on your app and how it should handle losing messages. An example might be:
public static string sendSqs(string data)
{
SendMessageRequest sendMessageRequest = new SendMessageRequest();
CreateQueueRequest sqsRequest = new CreateQueueRequest();
sqsRequest.QueueName = "mySqsQueue";
CreateQueueResponse createQueueResponse = sqs.CreateQueue(sqsRequest);
sendMessageRequest.QueueUrl = createQueueResponse.QueueUrl;
sendMessageRequest.MessageBody = data;
try{
SendMessageResponse sendMessageresponse = SQS.SendMessage(sendMessageRequest);
catch(InvalidMessageContents ex){ //Catch or bubble the exception up.
//I can't do anything about this so toss the message...
LOGGER.log("Invalid data in request: "+data, ex);
return null;
} catch(Throttling ex){ //I can do something about this!
//Exponential backoff...
}
return sendMessageresponse.MessageId;
}
Exceptions like Throttling or ServiceUnavailable are ones commonly overlooked but can be handled properly. Its commonly recommended that for things like these you implement an Exponential Backoff. When you're throttled you backoff until the service is available again. An example of implementation and usage in Java: https://gist.github.com/alph486/f123ea139e6ea56e696f .
You shouldn't need to do much of your own error handling at all; the AWS SDK for .NET handles retries for transient failures under the hood.
It will automatically retry any request that fails if:
your access to the AWS service is being throttled
the request times out
the HTTP connection fails
It uses an exponential backoff strategy for multiple retries. On the first failure, it sleeps for 400 ms, then tries again. If that attempt fails, it sleeps for 1600 ms before trying again. If that fails, it sleeps for 6400 ms, and so on, to a maximum of 30 seconds.
When the configured maximum number of retries is reached, the SDK will throw. You can configure the maximum number of retries like this:
var sqsClient = AWSClientFactory.CreateAmazonSQSClient(
new AmazonSQSConfig
{
MaxErrorRetry = 4 // the default is 4.
});
If the API call ends up throwing, it means that something is really wrong, like SQS has gone down in your region, or your request is invalid.
Source: The AWS SDK for .NET Source Code on GitHub.
Related
I am trying to implement exponential backoff with azure service bus.
Basically i have a catch block and if any error currently what i am doing is i am asking it to retry after every 1 second and i am awaiting that.
Goal :
I want to use exponential delay. So basically after each retry i want exponentially increase the seconds and i dont want it to wait. Till then it can process other messages.
Current catch block looks like :
catch (Exception ex)
{
_logger.Error(ex, $"Failed to process request {requestId}");
totalAttempts++;
if (totalAttempts == MaxAttempts)
return new Response { Error = ex.ToString() };
await Task.Delay(TimeSpan.FromSeconds(1));
}
I tried the below but it is not exponentially increasing the time. I am using this for the first time .
while (true)
try
{
executemethod();
}
catch (Exception ex)
{
_logger.Error(ex, $"Failed to process request {requestId}");
totalAttempts++;
if (totalAttempts == MaxAttempts)
return new Response { Error = ex.ToString() };
queueClient.RetryPolicy = new RetryExponential(TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(30), 10);
}
I am not very much sure if i am doing correct.
Goal : need to retry with a back-off strategy and make the processing thread available for other messages while waiting for the next retry.Consider exponential back-off with dead-lettering after reaching max delivery attempts.
Many Azure-oriented .NET libraries implement Retry internally. Service Bus client has it built-in as well. Check out Retry guidance for Azure services. As per documentation:
When using the built-in RetryExponential implementation, do not
implement a fallback operation as the policy reacts to Server Busy
exceptions and automatically switches to an appropriate retry mode.
Other from that, you shall set policy before you make a request and not in the process.
For retry policies, you can also use an external library like Polly.
I am creating an application to connect to multiple ActiveMQ servers and get the total number of messages withing their different queues.
I am using a slightly modified version of the code found in this link ActiveMQ with C# and Apache NMS - Count messages in queue
to count the messages withing the queue.
The problem I am having is that if the queue contains more than 400 messages this code stops counting at 400.
public static int GetMessageCount(string server, string user, string pw) {
int messageCount = 0;
var _server = $"activemq:ssl://{server}:61616?transport.acceptInvalidBrokerCert=true";
IConnectionFactory factory = new NMSConnectionFactory(_server);
using (IConnection connection = factory.CreateConnection(user, pw)) {
connection.Start();
using (ISession session = connection.CreateSession(AcknowledgementMode.AutoAcknowledge)) {
IDestination requestDestination = session.GetQueue(QueueRequestUri);
IQueueBrowser queueBrowser = session.CreateBrowser((IQueue)requestDestination);
IEnumerator messages = queueBrowser.GetEnumerator();
while (messages.MoveNext()) {
IMessage message = (IMessage)messages.Current;
messageCount++;
}
connection.Close();
session.Close();
connection.Close();
}
}
return messageCount;
}
How do I get the actual number of messages in the queue?
Why is this behavior?
Is this an issue with IEnumerator interface or is it an issue with the Apache.NMS.ActiveMQ API?
Normally there is no guarantee that the browser will return all messages from the queue. It provides a snapshot of the messages but may not return all of them. ActiveMQ has a limit for overhead reduction. You can increase the limits, see maxBrowsePageSize, however there is still no guarantee.
maxBrowsePageSize - 400 - The maximum number of messages to page in from the store at one time for a browser.
Those APIs are not designed for counting messages and you shouldn't do it. Just process the messages without counting them. If you want to get metrics then use some kind of admin libraries. JMX (yes I know you use C#) could be helpful as well.
I am reading from a REST service and need to handle "Wait and retry" for a heavily used service that will give me an error:
Too many queries per second
or
Server Busy
Generally speaking, since I have many REST services to call, how can I generically handle backoff logic that would occur when an exception occurs?
Is there any framework that has this built in? I'm just looking to write clean code that doesn't worry too much about plumbing and infrastructure.
You can wrap the attempt up within a method that handles the retry logic for you. For example, if you're using WebClient's async methods:
public async Task<T> RetryQuery<T>(Func<Task<T>> operation, int numberOfAttempts, int msecsBetweenRetries = 500)
{
while (numberOfAttempts > 0)
{
try
{
T value = await operation();
return value;
}
catch
{
// Failed case - retry
--numberOfAttempts;
}
await Task.Delay(msecsBetweenRetries);
}
throw new ApplicationException("Operation failed repeatedly");
}
You could then use this via:
// Try 3 times with 500 ms wait times in between
string result = await RetryQuery(async () => webClient.DownloadStringTaskAsync(url), 3);
Try and determine how many active requests can be active at a time and use a Semaphore.
It is a way to handle resource locking where the are multiple identical resources, but only a limited number of them.
Here's the MSDN documentation on semaphores
I recommend you look into the Transient Fault Handling Application Block, part of the Enterprise Library.
In the past, the EL has IMO been over-engineered and not that useful, but they've taken steps to address that; the TFHAB is one of the newer blocks that follows better design guidelines (again, IMO).
I have to periodically check messages in a queue within Websphere MQ. I haven't found better approach rather than try getting a message and handle 2033 reason code (which is NO_MSG_AVAILABLE) like this:
try
{
// ...
inQueue.Get(message);
}
catch (MQException exception)
{
if (exception.ReasonCode != 2033)
throw;
}
Is there better way to get message from queue? I think that there might be some openOptions flag that I'm not aware of, that wouldn't throw exception when no message available, but return null instead.
There are three ways to avoid or reduce this polling mechanism.
Here they are in oder of elegance(the higher the better):
MQGET with wait interval UNLIMITED and MQGMO_FAIL_IF_QUIESCING
Get your application be triggered by MQServer
Callback function - new with MQ V7 on both sides
You are missing the MQC.MQGMO_WAIT flag on MQGetMessageOptions.Options. Change it this way:
getOptions = new MQGetMessageOptions {WaitInterval = MQC.MQWI_UNLIMITED, Options = MQC.MQGMO_WAIT | MQC.MQGMO_FAIL_IF_QUIESCING}
Please note that this would make the calling thread to be blocked till a message arrives at the queue or some connection exception occurs. MQ has another client called IBM Message Service Client (aka XMS .NET) that provides a JMS specification implementation in .NET. This has a nice little Message Listener which gets automatically invoked whenever a message arrives in a queue. Unlike in the above example, the calling thread will not be blocked when Message Listener is used.
More details on XMS .NET can be found here. Samples are also shipped with MQ and for message listener sample, please refer "SampleAsyncConsumer.cs" source file.
I was getting this. I solved it by putting the Message initiator inside the loop:
_queueManager = new MQQueueManager(Queuemanager, _mqProperties);
MQQueue queue = _queueManager.AccessQueue(
Queuename,
MQC.MQOO_INPUT_AS_Q_DEF + MQC.MQOO_FAIL_IF_QUIESCING + MQC.MQOO_INQUIRE);
string xml = "";
while (queue.CurrentDepth > 0)
{
MQMessage message = new MQMessage();
queue.Get(message);
xml = message.ReadString(message.MessageLength);
MsgQueue.Enqueue(xml);
message.ClearMessage();
}
There must be something in the Message internally that errors when reusing it for another get.
I am developing an application using twitter api and that involves writing a method to check if a user exists. Here is my code:
public static bool checkUserExists(string user)
{
//string URL = "https://twitter.com/" + user.Trim();
//string URL = "http://api.twitter.com/1/users/show.xml?screen_name=" + user.Trim();
//string URL = "http://google.com/#hl=en&sclient=psy-ab&q=" + user.Trim();
string URL = "http://api.twitter.com/1/users/show.json?screen_name=" + user.Trim();
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(URL);
try
{
var webResponse = (HttpWebResponse)webRequest.GetResponse();
return true;
}
//this part onwards does not matter
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError && ex.Response != null)
{
var resp = (HttpWebResponse)ex.Response;
if (resp.StatusCode == HttpStatusCode.NotFound)
{
return false;
}
else
{
throw new Exception("Unknown level 1 Exception", ex);
}
}
else
{
throw new Exception("Unknown level 2 Exception", ex);
}
}
}
The problem is, calling the method does not work(it doesn't get a response) more than 2 or 3 times, using any of the urls that have been commented, including the google search query(I thought it might be due to twitter API limit). On debug, it shows that it's stuck at:
var webResponse = (HttpWebResponse)webRequest.GetResponse();
Here's how I am calling it:
Console.WriteLine(TwitterFollowers.checkUserExists("handle1"));
Console.WriteLine(TwitterFollowers.checkUserExists("handle2"));
Console.WriteLine(TwitterFollowers.checkUserExists("handle3"));
Console.WriteLine(TwitterFollowers.checkUserExists("handle4"));
Console.WriteLine(TwitterFollowers.checkUserExists("handle5"));
Console.WriteLine(TwitterFollowers.checkUserExists("handle6"));
At most I get 2-3 lines of output. Could someone please point out what's wrong?
Update 1:
I sent 1 request every 15 seconds (well within limit) and it still causes an error. on the other hand, sending a request, closing the app and running it again works very well (on average accounts to 1 request every 5 seconds). The rate limit is 150 calls per hour Twitter FAQ.
Also, I did wait for a while, and got this exception at level 2:
http://pastie.org/3897499
Update 2:
Might sound surprising but if I run fiddler, it works perfectly. Regardless of whether I target this process or not!
The effect you're seeing is almost certainly due to rate-limit type policies on the Twitter API (multiple requests in quick succession). They keep a tight watch on how you're using their API: the first step is to check their terms of use and policies on rate limiting, and make sure you're in compliance.
Two things jump out at me:
You're hitting the API with multiple requests in rapid succession. Most REST APIs, including Google search, are not going to allow you to do that. These APIs are very visible targets, and it makes sense that they'd be pro-active about preventing denial-of-service attacks.
You don't have a User Agent specified in your request. Most APIs require you to send them a meaningful UA, as a way of helping them identify you.
Note that you're dealing with unmanaged resources underneath your HttpWebResponse. So calling Dispose() in a timely fashion or
wrapping the object in a using statement is not only wise, but important to avoid blocking.
Also, var is great for dealing with anonymous types, Linq query
results, and such but it should not become a crutch. Why use var
when you're well aware of the type? (i.e. you're already performing
a cast to HttpWebResponse.)
Finally, services like this often limit the rate of connections per second and/or the number of simultaneous connections allowed to prevent abuse. By not disposing of your HttpWebResponse objects, you may be violating the permitted number of simultaneous connections. By querying too often you'd break the rate limit.