We Keep Coding

RabbitMQ work queues using Dependency Injection in c#

I am using rabbitmq in a "Work Queues" scenario.
I need eg. a pool of 5 consumers, (each with its own channel), so one consumer doing I/O operations, won't block other consumer of the same queue.
Eg.
If I have on my queue:
Message 1, Message 2, Message 3, Message 4. Each instance of (FistConsumerHandler) will take 1 message from the queue using Round Robin (default rabbitmq behavior)
The problem I am facing is I need to do this using Dependency Injection.
Here is what i have so far:
On Windows service start (my consumers are hosted in a windows service):
protected override void OnStart(string[] args)
{
BuildConnections();
// Register the consumers. For simplicity only showing FirstConsumerHandler.
AddConsumerHandlers<FistConsumerHandler>(ConstantesProcesos.Exchange, ConstantesProcesos.QueueForFirstHandler);
BuildStartup();
var logger = GetLogger<ServicioProcesos>();
logger.LogInformation("Windows Service Started");
Console.WriteLine("Press [enter] to exit.");
}
protected virtual void BuildConnections(
string notificationHubPath = "notificationhub_path",
string rabbitMQHostname = "rabbitmq_hostname",
string rabbitMQPort = "rabbitmq_port",
string rabbitMQUserName = "rabbitmq_username",
string rabbitMQPassword = "rabbitmq_password")
{
ContextHelpers.Setup(ConfigurationManager.ConnectionStrings[appContextConnectionString].ConnectionString);
if (_connection == null)
{
var factory = new ConnectionFactory
{
HostName = ConfigurationManager.AppSettings[rabbitMQHostname],
Port = int.Parse(ConfigurationManager.AppSettings[rabbitMQPort]),
UserName = ConfigurationManager.AppSettings[rabbitMQUserName],
Password = ConfigurationManager.AppSettings[rabbitMQPassword],
DispatchConsumersAsync = true,
};
// Create a connection
do
{
try
{
_connection = factory.CreateConnection();
}
catch (RabbitMQ.Client.Exceptions.BrokerUnreachableException e)
{
Thread.Sleep(5000);
}
} while (_connection == null);
}
_startupBuilder = new StartupBuilder(_connection);
}
protected void AddConsumerHandlers<THandler>(string exchange, string queue)
{
var consumerHandlerItem = new ConsumerHandlerItem
{
ConsumerType = typeof(THandler),
Exchange = exchange,
Queue = queue
};
_startupBuilder._consumerHandlerItems.Add(consumerHandlerItem);
}
protected void BuildStartup()
{
ServiceProvider = _startupBuilder.Build();
}
Startup Builder:
using Microsoft.Extensions.DependencyInjection;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
using System;
using System.Collections.Generic;
public class StartupBuilder
{
private static IConnection _connection;
private IModel _channel;
public List<ConsumerHandlerItem> _consumerHandlerItems;
public IServiceCollection Services { get; private set; }
public StartupBuilder(IConnection connection)
{
_connection = connection;
_consumerHandlerItems = new List<ConsumerHandlerItem>();
Services = new ServiceCollection();
}
public IServiceProvider Build()
{
_channel = _connection.CreateModel();
Services.InitSerilog();
// Add channel as singleton (this is not correct as I need 1 channel per ConsumerHandler)
Services.AddSingleton(_channel);
// Register the ConsumerHandler to DI
foreach (var item in _consumerHandlerItems)
{
// Add FirstHandler to DI
Type consumerType = item.ConsumerType;
Services.AddSingleton(consumerType);
}
// Finish DI Setup
var serviceProvider = Services.BuildServiceProvider();
// Bind the consumer handler to the channel and queue
foreach (var item in _consumerHandlerItems)
{
var consumerHandler = (AsyncEventingBasicConsumer)serviceProvider.GetRequiredService(item.ConsumerType);
_channel.AssignNewProcessor(item, consumerHandler);
}
return serviceProvider;
}
}
Helpers:
public static class QueuesHelpers
{
public static void AssignNewProcessor(this IModel channel, ConsumerHandlerItem item, AsyncEventingBasicConsumer consumerHandler)
{
channel.ExchangeDeclare(item.Exchange, ExchangeType.Topic, durable: true);
channel.QueueDeclare(item.Queue, true, false, false, null);
channel.QueueBind(item.Queue, item.Exchange, item.Queue, null);
channel.BasicConsume(item.Queue, false, consumerHandler);
}
}
Consumer handler:
public class FistConsumerHandler : AsyncEventingBasicConsumer
{
private readonly ILogger<FistConsumerHandler> _logger;
private Guid guid = Guid.NewGuid();
public FistConsumerHandler(
IModel channel,
ILogger<FistConsumerHandler> logger) : base(channel)
{
Received += ConsumeMessageAsync;
_logger = logger;
}
private async Task ConsumeMessageAsync(object sender, BasicDeliverEventArgs eventArgs)
{
try
{
// consumer logic to consume the message
}
catch (Exception ex)
{
}
finally
{
Model.Acknowledge(eventArgs);
}
}
}
The problem with this code is:
There is ony 1 instance of FistConsumerHandler (as is reigstered as singleton). I need, for instance 5.
I have only 1 channel, I need 1 channel per instance.
To sum up, the expected behavior using Microsoft.Extensions.DependencyInjection should be:
Create a connection (share this connection with all consumers)
When a message is received to the queue, it should be consumed by 1 consumer using its own channel
If another message is received to the queue, it should be consumed by another consumer

TL;DR; Create your own scope
I've done something similar in an app I'm working on, albeit not as cleanly as I would like (and thus why I came across this post). The key for me was using IServiceScopeFactory to get injected services and use them in a consumer method. In a typical HTTP request the API will automatically create/close scope for you as the request comes in / response goes out, respectively. But since this isn't an HTTP request, we need to create / close the scope for using injected services.
This is a simplified example for getting an injected DB context (but could be anything), assuming I've already set up the RabbitMQ consumer, deserialized the message as an object (FooEntity in this example):
public class RabbitMQConsumer
{
private readonly IServiceProvider _provider;
public RabbitMQConsumer(IServiceProvider serviceProvider)
{
this._serviceProvider = serviceProvider;
}
public async Task ConsumeMessageAsync()
{
// Using statement ensures we close scope when finished, helping avoid memory leaks
using (var scope = this._serviceProvider.CreateScope())
{
// Get your service(s) within the scope
var context = scope.ServiceProvider.GetRequiredService<MyDBContext>();
// Do things with dbContext
}
}
}
Be sure to register RabbitMQConsumer as a singleton and not a transient in Startup.cs also.
References:
Similar SO post
MS Docs

Observer pattern using gRPC - C#

Sorry, if this is a stupid question but I don't find any useful information in the internet.
Has anyone ever tried to implement the observer pattern in C# using gRPC as communication?
If yes, please show me the link.
Many thanks in advance and best regards.

I have implemented a client convenience class wrapper to turn server streaming calls into regular events for a project I am working. Not sure if this is what you are after. Here is a simple gRPC server that just publishes the time as a string once every second.
syntax = "proto3";
package SimpleTime;
service SimpleTimeService
{
rpc MonitorTime(EmptyRequest) returns (stream TimeResponse);
}
message EmptyRequest{}
message TimeResponse
{
string time = 1;
}
The server implementation, which just loops once a second returning the string representation of the current time until canceled, is as follows
public override async Task MonitorTime(EmptyRequest request, IServerStreamWriter<TimeResponse> responseStream, ServerCallContext context)
{
try
{
while (!context.CancellationToken.IsCancellationRequested)
{
var response = new TimeResponse
{
Time = DateTime.Now.ToString()
};
await responseStream.WriteAsync(response);
await Task.Delay(1000);
}
}
catch (Exception)
{
Console.WriteLine("Exception on Server");
}
}
For the client, I created a class that contains the gRPC client and exposes the results of the server streaming MonitorTime call as a plain ole .net event.
public class SimpleTimeEventClient
{
private SimpleTime.SimpleTimeService.SimpleTimeServiceClient mClient = null;
private CancellationTokenSource mCancellationTokenSource = null;
private Task mMonitorTask = null;
public event EventHandler<string> OnTimeReceived;
public SimpleTimeEventClient()
{
Channel channel = new Channel("127.0.0.1:50051", ChannelCredentials.Insecure);
mClient = new SimpleTime.SimpleTimeService.SimpleTimeServiceClient(channel);
}
public void Startup()
{
mCancellationTokenSource = new CancellationTokenSource();
mMonitorTask = Task.Run(() => MonitorTimeServer(mCancellationTokenSource.Token));
}
public void Shutdown()
{
mCancellationTokenSource.Cancel();
mMonitorTask.Wait(10000);
}
private async Task MonitorTimeServer(CancellationToken token)
{
try
{
using (var call = mClient.MonitorTime(new SimpleTime.EmptyRequest()))
{
while(await call.ResponseStream.MoveNext(token))
{
var timeResult = call.ResponseStream.Current;
OnTimeReceived?.Invoke(this, timeResult.Time);
}
}
}
catch(Exception e)
{
Console.WriteLine($"Exception encountered in MonitorTimeServer:{e.Message}");
}
}
}
Now create the client and subscribe to the event.
static void Main(string[] args)
{
SimpleTimeEventClient client = new SimpleTimeEventClient();
client.OnTimeReceived += OnTimeReceivedEventHandler;
client.Startup();
Console.WriteLine("Press any key to exit");
Console.ReadKey();
client.Shutdown();
}
private static void OnTimeReceivedEventHandler(object sender, string e)
{
Console.WriteLine($"Time: {e}");
}
Which when run produces
I have left out a lot of error checking and such to make the example smaller. One thing I have done is for gRPC interfaces with many server streaming calls that may or may not be of interest to call clients, is to implement the event accessor (add,remove) to only call the server side streaming method if there is a client that has subscribed to the wrapped event. Hope this is helpful

Avoid multiple web service access. How to achieve that?

I am trying to implement a simple Windows 8 C# XAML application where there are two calls made to access single web service one from project to load and display data and other to display notification.
Since there are two calls made for same web service I want that if one call is already made to the service the other call should wait and use the same response from the first call.
How can i achieve this kind of functionality? I have not added any code since there is no code that i have written for this. I am just trying to think first and then will I code.
Please let me know I can get some help for this kind of project structure?

You can do this by caching the Task that's currently downloading and not starting the download again if there is a cached Task:
private volatile Task<string> m_cachedWebServiceTask;
async Task<string> AccessWebServiceAsync()
{
var task = m_cachedWebServiceTask;
if (task == null)
task = m_cachedWebServiceTask = DownloadFromWebServiceAsync();
try
{
return await task;
}
finally
{
m_cachedWebServiceTask = null;
}
}
Note that this code has a race condition: if you call AccessWebServiceAsync() twice at the same time, there is a small chance DownloadFromWebServiceAsync() will be called twice too. But since this in only an optimization, I think that shouldn't be a problem. If it is a problem for you, you would need to guard the access to the field by a lock.

As I had the feeling that this problem needs further attention and it's solution could still be optimized, I decided to post another approach. The OP is mostly a problem about leveraging the following 3 scopes of requirements: user experience within the application, the application's internal requirements and the web service's loading with multiple requests.
The application needs to make an initial request to load the data.
When he asks for it, the user expects to get the results with the latest updates.
On the other side, it makes no initiate a large series of calls to the web service within a very short moment of time.
So, managing what happens in this very short moment of time it's actually the solution to the problem.
On the client side, the Service1Client class:
public partial class Service1Client
{
// default time buffer value
int _timeBuffer = 100;
// a time buffer used for returning the same response
public int TimeBuffer
{
get { return _timeBuffer; }
set { _timeBuffer = value; }
}
// the start and end time of the web service request
static DateTime _start, _end;
// a volatile static variable to store the response in the buffering time
volatile static string _response;
// used for blocking other threads until the current thread finishes it's job
object _locker = new object();
public async Task<string> GetResponseData()
{
return await Task.Factory.StartNew<string>(() =>
{
lock (_locker)
{
if (DateTime.Now >= _end.AddMilliseconds(TimeBuffer))
{
_start = DateTime.Now;
var async = GetDataAsync();
_response = async.Result;
_end = DateTime.Now;
}
}
return _response;
});
}
}
The console application used for testing:
class Program
{
static void Main(string[] args)
{
while (true)
{
var client = new ServiceReference1.Service1Client();
client.TimeBuffer = 150;
Console.WriteLine(client.GetResponseData().Result);
if (Console.ReadKey().Key == ConsoleKey.Enter)
break;
}
}
}
As a remark, note that, for the reason of a clear sample, I decided to change the returned type of the GetDate WCF service's method from DateTime to string.
[ServiceContract]
public interface IService1
{
[OperationContract]
string GetData();
}
public class Service1 : IService1
{
public string GetData()
{
System.Threading.Thread.Sleep(5000);
return DateTime.Now.ToString();
}
}

For your scenario, a feasible idea would be to extend the service class.
The IService1 interface definition:
[ServiceContract]
public interface IService1
{
[OperationContract]
DateTime GetData();
}
The Service1 class definition:
public class Service1 : IService1
{
public DateTime GetData()
{
System.Threading.Thread.Sleep(5000);
return DateTime.Now;
}
}
On the client side, extend the Service1Client class definition and add a new method:
public partial class Service1Client
{
static bool _isOpen;
static DateTime? _cachedResponse;
object _locker = new object();
public DateTime GetResponseData()
{
if (!_isOpen)
{
if (!_cachedResponse.HasValue)
{
lock (_locker)
{
_isOpen = true;
_cachedResponse = GetData();
_isOpen = false;
}
return _cachedResponse.Value;
}
else
{
Task.Factory.StartNew<DateTime>(() =>
{
lock (_locker)
{
_isOpen = true;
_cachedResponse = GetData();
_isOpen = false;
}
return _cachedResponse.Value;
});
}
}
return _cachedResponse.Value;
}
}
Test it:
class Program
{
static void Main(string[] args)
{
while (true)
{
var client = new ServiceReference1.Service1Client();
Console.WriteLine(client.GetResponseData());
if (Console.ReadKey().Key == ConsoleKey.Enter)
break;
}
}
}

Asynchronous insert in Azure Table

How to asynchronously save an entity to Windows Azure Table Service?
The code below works synchronously but raises an exception when trying to save asynchronously.
This statement:
context.BeginSaveChangesWithRetries(SaveChangesOptions.Batch,
(asyncResult => context.EndSaveChanges(asyncResult)), null);
Results in System.ArgumentException: "The current object did not originate the async result. Parameter name: asyncResult".
Additionally, what's the correct pattern for creating the service context when saving asynchronously? Should I create a separate context for each write operation? Is it too expensive (e.g. requiring a call over the network)?
TableStorageWriter.cs:
using System;
using System.Data.Services.Client;
using System.Diagnostics;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.StorageClient;
namespace WorkerRole1
{
public class TableStorageWriter
{
private const string _tableName = "StorageTest";
private readonly CloudStorageAccount _storageAccount;
private CloudTableClient _tableClient;
public TableStorageWriter()
{
_storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
_tableClient = _storageAccount.CreateCloudTableClient();
_tableClient.CreateTableIfNotExist(_tableName);
}
public void Write(string message)
{
try
{
DateTime now = DateTime.UtcNow;
var entity = new StorageTestEntity
{
Message = message,
PartitionKey = string.Format("{0:yyyy-MM-dd}", now),
RowKey = string.Format("{0:HH:mm:ss.fff}-{1}", now, Guid.NewGuid())
};
// Should I get this context before each write? It is efficient?
TableServiceContext context = _tableClient.GetDataServiceContext();
context.AddObject(_tableName, entity);
// This statement works but it's synchronous
context.SaveChangesWithRetries();
// This attempt at saving asynchronously results in System.ArgumentException:
// The current object did not originate the async result. Parameter name: asyncResult
// context.BeginSaveChangesWithRetries(SaveChangesOptions.Batch,
// (asyncResult => context.EndSaveChanges(asyncResult)), null);
}
catch (StorageClientException e)
{
Debug.WriteLine("Error: {0}", e.Message);
Debug.WriteLine("Extended error info: {0} : {1}",
e.ExtendedErrorInformation.ErrorCode,
e.ExtendedErrorInformation.ErrorMessage);
}
}
}
internal class StorageTestEntity : TableServiceEntity
{
public string Message { get; set; }
}
}
Called from WorkerRole.cs:
using System.Net;
using System.Threading;
using Microsoft.WindowsAzure.ServiceRuntime;
using log4net;
namespace WorkerRole1
{
public class WorkerRole : RoleEntryPoint
{
public override void Run()
{
var storageWriter = new TableStorageWriter();
while (true)
{
Thread.Sleep(10000);
storageWriter.Write("Working...");
}
}
public override bool OnStart()
{
ServicePointManager.DefaultConnectionLimit = 12;
return base.OnStart();
}
}
}
Examples using Windows Azure SDK for .NET 1.8.

You should call EndSaveChangesWithRetries instead of EndSaveChanges, as otherwise the IAsyncResult object returned by BeginSaveChangesWithRetries cannot be used by EndSaveChanges. So, could you please try changing your End method call as below?
context.BeginSaveChangesWithRetries(SaveChangesOptions.Batch,
(asyncResult => context.EndSaveChangesWithRetries(asyncResult)),
null);
And for your other question, I would recommend creating a new TableServiceContext for each call, as DataServiceContext is not stateless (MSDN) and the way you implemented TableStorageWriter.Write with the asynchronous call might allow concurrent operations. Actually, in Storage Client Library 2.0, we explicitly prevented concurrent operations that uses a single TableServiceContext object. Moreover, creating a TableServiceContext does not result in a request to Azure Storage.

Unknown command error when using multithread to set redis

I am using the ServiceStack.Redis C# client to talk to Redis.
With few request everything is ok, but when I get LoadRunner to request it or use multi-threading to make requests, I get some errors that say I am using the wrong command.
I check the errors, and it seems that it cut off the command, or it mess up.
Here is my code, very simple. Has anyone come across this problem? The errors happen when I call the Push method using multi-threading.
public class ImpresstionQueueService : IQueueService<InsertImpressionRequest>
{
private string _queueName;
private string _host;
private static IRedisClient redisClient = new RedisClient(ConfigHost);
private static string ConfigHost
{
get
{
return ConfigurationManager.AppSettings.Get("redis_host");
}
}
private string Host
{
get
{
if (!string.IsNullOrEmpty(_host))
return _host;
else
{
return ConfigurationManager.AppSettings.Get("redis_host");
}
}
}
public ImpresstionQueueService(string queue_name)
{
this._queueName = queue_name;
}
public ImpresstionQueueService(string host, string queu_name)
{
this._queueName = queu_name;
this._host = host;
}
#region IQueueService<InsertImpressionRequest> Members
class testData
{
}
public int Push(InsertImpressionRequest value)
{
try
{
//using (var redisClient = new RedisClient(this.Host))
{
//ser
string ser_value = TypeSerializer.SerializeToString<InsertImpressionRequest>(value);
//push
redisClient.AddItemToList(this._queueName, ser_value);//here will be error
}
}
catch (Exception ex)
{
HLogger.GetLogger("RedisLogger").Error(ex.Message + ex.StackTrace);
}
//throw new NotImplementedException();
return 1;
}
public InsertImpressionRequest Pop()
{
InsertImpressionRequest request = null;
//using (var redisClient = new RedisClient(this.Host))
{
string pop_string_value = redisClient.PopItemFromList(this._queueName);
//deseri
if (pop_string_value != null)
{
request = TypeSerializer.DeserializeFromString<InsertImpressionRequest>(pop_string_value);
}
}
return request;
}
#endregion
}

You are probably using the same Redis connection simultaneously from multiple threads. Both threads could possibly send commands or wait for replies at the same time. When this happens, one thread receives data intended for the other thread. This causes your error.
If you use one Redis client per thread (instead of one client per ImpresstionQueueService), each thread can send commands at the same time without interfering with each other.
Alternatively, you can create a client just for the single request (which you commented out just above the error location). The disadvantage of this alternative is the overhead of a new connection every time (which might be large or small or unnoticeable).