We Keep Coding

Azure Service Bus scheduled message arriving too late

I'm using ASB Topics. I'm connecting the service by using Microsoft .NET ServiceBus nuget (namespace Microsoft.Azure.ServiceBus.Core)
When a message arrive, my consumer either handles it and release the message or resending it to the topic with a delay.
The problem is that when the delay is less than 15 seconds, sometimes the message only arrive after 15 seconds.
e.g setting the delay to 3s or 10s usually works fine, but some of the messages arrive only after 15s (in both 3s or 10s cases).
When setting the delay to 20 seconds it works just fine with no exceptions.
It's for sure not load on the consumer because in some cases it was idle during the wait time.
I tried using prefetchCount but it had no effect.
I wanted to track the scheduled message in Azure UI but it seems that this option available for queues (not topics) only.
Any idea why is that happening and what can I do? thanks!

I'm using premium tier, my receiver runs in azure k8s. i'm quite sure
this number 15s is defined somewhere to delay messages in certain
cases. was wondering if someone knows about this.
You can queue or subject messages for later processing; for example, you can plan a job to be ready for processing by a system at a specific time. This functionality allows for the creation of a dependable distributed time-based scheduler.
Scheduled messages do not appear in the queue until the enqueue time has passed. Scheduled messages can be cancelled before that time. The communication is deleted when you cancel it.
You can use any of our clients to schedule messages in one of two ways:
Use the standard send API, but before sending, set the ScheduledEnqueueTimeUtc property on the message.
Pass both the standard message and the planned time to the schedule message API. This will return the SequenceNumber of the planned message, which you can use to cancel it later if necessary.
For more information please refer the below links:
MICROSOFT DOCUMENTATION:- Scheduled messages & Best Practices for performance improvements using Service Bus Messaging

How to wait 24hrs before completing an async task?

I'm making an MVC app with the .NET Framework and in one of my controllers I call an async task that sends an e-mail to the signed in user.
This task is called upon when the user clicks a specific checkbox and the e-mail is meant to work as sort of reminder.
The entire task works as intended (the user gets an e-mail when the checkbox is checked), but I need it to wait 24 hours before actually sending the e-mail, as it is a reminder.
Currently the e-mail is sent right away, how can I delay the completion of my "e-mail task", while the code continues?

Use a library like Hangfire which lets you schedule background jobs and backs them with persistent storage.
You can then easily schedule a job like:
BackgroundJob.Schedule(
() => SendEmail("user#domain"),
TimeSpan.FromDays(1));

This is a classic X Y Problem. While it may be possible to make your system wait 24 hours you are creating a very fragile system that can be affected by app pool resets and server reboots.
Putting aside the possibility of an unexpected reboot, what happens when your maintenance cycle comes around and a scheduled reboot is going to happen? How many queued email reminders will you have that you can't do anything with?
The best approach for systems that don't immediately use their data is to buffer it through some form of storage scheme. It could be as simple as writing queued emails to files on the system, or something more robust like a database with a dedicated email sending service.
I have used a LOT of email sending systems over the years, and even for immediate sends we have used a database intermediary, with one dedicated email sending Windows service to produce and send the actual email. By centralizing the email production you not only get one place to maintain your email sending code, but you can also increase the durability of the whole system.
Bonus points if your database is part of a high availability cluster, as in this kind of system the database becomes the critical point. If it is then you're protected from any form of downtime other than a total network outage.

Let the Task wait for 24 hours before sending the mail.
await Task.delay(TimeSpan.FromHours(24));
Add this line in your async function bfore sending the email

Determining response time for IIS requests

We have a web service that is called by a third party. Third party's rules are that the service must return a response within 10 seconds.
We log all of the processing time, from when we receive the request, to when the web method exits.
Our problem: our third party is telling us that we are taking over 10 seconds, but according to our logs, we are finished processing well within time limit.
I suspect that the third party is having a connectivity problem, and that the time is lost after we complete processing, but while the response is coming down the wire. Our in-application logging can't captuer that timing (that I know of) because our web method has already returned.
Is there any IIS logging feature that we could use capture the time spent returning the response?

The time-taken entry in your logs accounts for the time it takes for the client to send the acknowledgment back to the server or reset the connection.
http://blogs.msdn.com/b/mike/archive/2008/11/13/time-vs-time-taken-fields-in-iis-logging.aspx
You'll want to also make use of the win32-status to determine when there is an issue.

What's the best way to send large amounts of email via C# Web page?

I have part of a project whereby I need to send 1 to up to 2000 emails from a webpage. One open source bug tracker used a Thread to send emails but it was riddled with bugs.
So just want to know if anyone has any ideas of how I can create a procedure that send up to 2000 emails a go?
I was told by one of my colleagues to leave it as one process (normal routine sending one email at a time) as it won't take long to process all the emails anyway. Doesn't seem right and I want to use threading... maybe I like complex routines?
[Extra]
I have something like:
foreach(string email in emailAddresses)
{
MailMessage mailMessage = new MailMessage();
mailMessage.To.Add(...)
...
SmtpClient client = new SmtpClient();
client.Send(mailMessage);
}

Use System.Net.Mail to send the messages; however you should use .NET 4 to avoid any connection issues, as there was a bug filed on the Connect website that will cause your messages to not get sent.
Don't use threading for three reasons:
Reason 1: A MTA is made to handle message retries and can handle failures. Your code may not be robust enough to handle this. System.Net.Mail is not able to do this out of the box.
Reason 2: If you do use threading, you will overwhelm the target SMTP server and it will block you. Most Windows SMTP relays have a default block of more than 15 (or 25?) concurrent connections.
If you're dealing with Exchange 2010, or 2007, then there is a throttling feature that gets activated if you send more than x messages per minute. This is a per MTA setting that will need to be adjusted to permit your situation.
Reason 3: The preferred way to do this is to have a dedicated IIS SMTP server (or Exchange...) that allows concurrent connections at high volume. Just use Sys.Net.Mail to hand the delivery task to the mail infrastructure. Not much is needed for a mail infrastructure. Just have a MTA that allows you to relay and it will "smart host" on your behalf out to the internet.
More questions on how to set up the MTA can be answered # serverfault.
However
You may want to use threading if your sending an email from an ASP.NET webpage... or will otherwise block the UI. Other than that, I don't see a need to run concurrent threads for the email generation task.
Lastly, if you're sending the same message to many recipients, you can either use a distribution list or append many target recipients to the same message.

You probably don't want to send 2000 emails on a thread that is servicing http requests. The user that submitted the request will be waiting for the server to respond until the emails send, and that's one less thread available for processing requests from other users. If you have many such requests it could drag down server performance.
Instead, I would post the request to a message queue, and have a seperate service process items from the queue and send emails.
Creating background threads in the asp.net app is another possibility, but then you end up in a situation where either:
You have to have your own task queue , separate from the one used by the normal thread pool.
Your email tasks are competing with, and potentially starving, the tasks for serving http requests.
That being said, there are deployment scenarios (a shared server, customer deployments) where the introduction of a second process is not desirable. If you don't have those constraints, however, I would go with the "separate process" because it's easier to scale your web site if it's focused on serving UI requests and leaves "fulfillment" tasks to a different service.

I think 2000 will take a long time (if it's a web page and the user is waiting for the page). It depends on your implementation, but you're doing this frequently you might want to have a "email thread" that's already been created. If there are no emails to send, then the thread can be suspended and not be consuming any resources. Then when the emails need to be sent you fill up an email queue with those emails and wake up the thread to start sending (if it is not already doing so).

If you're sending more than about 50 at a time, you need to contract out to a service that does this for you. Otherwise your messages will end up in people's spam folders.

Selective Reading From a Queue--Custom MSMQ Service, ESB, or Something Else?

Looking for some ideas/pattern to solve a design problem for a system I will be starting work on soon. There is no question that I will need to use some sort of messaging (probably MSMQ) to communicate between certain areas of the system. I don't want to reinvent the wheel, but at the same time I want to make sure I am using the right tool for the job. I have been tinkering with and reading up on NServiceBus, and I'm very impressed with what it does--however I'm not sure it's intended for what I'm trying to achieve.
Here is a (hopefully) very simple and conceptual description of what the system needs to do:
I have a service that clients can send messages to. The service is "Fire and Forget"--the most the client would get back is something that may say success or failure (success being that the message was received).
The handling/processing of each message is non-trivial, and may take up significant system resources. For this reason only X messages can be handled concurrently, where X is a configurable value (based on system specs, etc.). Incoming messages will be stored in queue until it's "their turn" to be handled.
For each client, messages must be handled in order (FIFO). However, some clients may send many messages in succession (thousands or more), for example if they lost connectivity for a period of time. For this reason, messages must be handled in a round-robin fashion across clients--no client is allowed to gorge and no client is allowed to starve. So the system will either have to be able to query the queue for a specific client, or create separate queues per client (automatically, since the clients won't be known at compile time) and pull from them in rotation.
My current thinking is that I really just need to use vanilla MSMQ, create a service to accept messages and write them to one or more queues, then create a process to read messages from the queue(s) and handle/process them. However, the reliability, auditing, scaleability, and ease of configuration you get with something like NServicebus looks very appealing.
Is an ESB the wrong tool for the job? Is there some other technology or pattern I should be looking at?
Update
A few clarifications.
Regarding processing messages "in order"--in the context of a single client, the messages absolutely need to be processed in the order they are received. It's complicated to explain the exact reasons why, but this is a firm requirement. I neglected to mention that only one message per client would ever be processed concurrently. So even if there were 10 worker threads and only one client had messages waiting to be processed, only one of those messages would be processed at a time--there would be no worry of a race condition.
I believe this is generally possible with vanilla MSMQ--that you can have a list of messages in a queue and always take the oldest one first.
I also wanted to clarify a use case for the round robin ordering. In this example, I have two clients (A and B) who send messages, and only one worker thread. All queues are empty. Client A has lost connectivity overnight, so at 8am sends 1000 messages to the service. These messages get queued up and the worker thread takes the oldest one and starts processing it. As this first message is being processed, client B sends a message into the service, which gets queued up (as noted, probably in a separate queue). When Client A's first message completes processing, the logic should check whether client B has a message (it's client B's "turn"), and since it finds one, process it next.
If client B hadn't sent a message during that time, the worker would continue processing client A's messages one at a time, always checking after processing to see if other client queues contained waiting messages to ensure that no client was being starved.
Where I still feel there may be a mismatch between an ESB and this problem is that an ESB is designed to facilitate communication between services; what I am trying to achieve is a combination of messaging/communication and a selective queuing system.

So the system will either have to be
able to query the queue for a specific client,
Searching through an MSMQ queue for a message from a particular client using cursors can be inefficient and doesn't scale.
or create separate queues per client (automatically, since the
clients won't be known at compile time) and pull from them in rotation.
MSMQ cannot create queues automatically. All messages have to be sent to a known queue first. Your own custom dispatcher service, though, could then create new queues on demand and put copies of the messages in them.
[[I avoid saying "move" messages as you can't do that with application code; you can only read a message and create a new message using the original data. This distinction is important when you are using Source Journaling, for example.]]
Cheers
John Breakwell

Using an ESB like NServiceBus seems like a good solution to your problem. But based on your conceptual description, there's some things to consider. Let's go through your requirements step-by-step, using NServiceBus as a possible ESB solution:
I have a service that clients can send messages to. The service is "Fire and Forget"--the most the client would get back is something that may say success or failure (success being that the message was received).
This is easily done with NServiceBus. You can Bus.Send(Message) from the client. If your client requires an answer, you can use Bus.Return(ErrorCode). You mention that "success being that the message was received". If you use an ESB like NServiceBus, it's up to the messaging platform the deliver the message. So, if your Bus.Send doesn't throw an exception, you can be sure that the message has been sent properly. Because of this you don't probably have to send success / failure messages back to the client.
The handling/processing of each message is non-trivial, and may take up significant system resources. For this reason only X messages can be handled concurrently, where X is a configurable value (based on system specs, etc.). Incoming messages will be stored in queue until it's "their turn" to be handled.
When using NServiceBus, you can configure the the number of worker threads by setting the "NumberOfWorkerThreads" option. If your server has multiple cores / cpus, you can use this setting to balance the work load.
For each client, messages must be handled in order (FIFO).
This is something that may cause problems depending on your requirements. ESBs in general don't promise to process the messages in-order, if they have many threads working on the messages. In a case of NServiceBus, you can send an array of messages from the client into the bus and these will be processed in-order. Also, you can solve some of the in-order messaging problems by using Sagas.
However, some clients may send many messages in succession (thousands or more), for example if they lost connectivity for a period of time
When using an ESB solution, your server doesn't have to be up for the client to work. Clients can still send messages and the server will start processing them as soon as it's back online. Here's a small introduction on this.
For this reason, messages must be handled in a round-robin fashion across clients--no client is allowed to gorge and no client is allowed to starve.
This isn't a problem because you've decided to use messages :)
So the system will either have to be able to query the queue for a specific client, or create separate queues per client (automatically, since the clients won't be known at compile time) and pull from them in rotation.
Could you expand on this? I'm not sure of your design on this one.