As I understand message brokers like RabbitMQ facilitates different applications written in different language/platform to communicate with each other. So since celery can use RabbitMQ as message broker, I believe we can queue task from any application to Celery, even though the producer isn't written in Python.
Now I am trying to figure out how I can queue a task to Celery from an application written in C# via RabbitMQ. But I could not find any such example yet.
The only information close to this I found is this SO question
Where the accepted answer suggests to use the Celery message format protocol to queue messages to RabbitMQ from Java. However, the link given in the answer does not have any example, only the message format.
Also, the message format says task id (UUID) is required to communicate in this protocol. How is my C# application supposed to know the task id of the celery task? As I understand it can only know about the task name, but not the task id.
I don't know whether the question is still relevant, but hopefully the answer will help others.
Here is how I succeeded in queening a task to Celery example worker.
You'll need to establish connection between your producer(client) to RabbitMQ as described here.
ConnectionFactory factory = new ConnectionFactory();
factory.UserName = username;
factory.Password = password;
factory.VirtualHost = virtualhost;
factory.HostName = hostname;
factory.Port = port;
IConnection connection = factory.CreateConnection();
IModel channel = connection.CreateModel();
In default RabbitMQ configuration there is only Guest user which can only be used for local connections (from 127.0.0.1). An answer to this question explains how to define users in RabbitMQ.
Next - creating a callback to get results. This example is using Direct reply-to, so an answer listener will look like:
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var ansBody = ea.Body;
var ansMessage = Encoding.UTF8.GetString(ansBody);
Console.WriteLine(" [x] Received {0}", ansMessage);
Console.WriteLine(" [x] Done");
};
channel.BasicConsume(queue: "amq.rabbitmq.reply-to", noAck: true, consumer: consumer);
Creating a task message that Celery will consume:
IDictionary<string, object> headers = new Dictionary<string, object>();
headers.Add("task", "tasks.add");
Guid id = Guid.NewGuid();
headers.Add("id", id.ToString());
IBasicProperties props = channel.CreateBasicProperties();
props.Headers = headers;
props.CorrelationId = (string)headers["id"];
props.ContentEncoding = "utf-8";
props.ContentType = "application/json";
props.ReplyTo = "amq.rabbitmq.reply-to";
object[] taskArgs = new object[] { 1, 200 };
object[] arguments = new object[] { taskArgs, new object(), new object()};
MemoryStream stream = new MemoryStream();
DataContractJsonSerializer ser = new DataContractJsonSerializer(typeof(object[]));
ser.WriteObject(stream, arguments);
stream.Position = 0;
StreamReader sr = new StreamReader(stream);
string message = sr.ReadToEnd();
var body = Encoding.UTF8.GetBytes(message);
And finally, publishing the message to RabbitMQ:
channel.BasicPublish(exchange: "",
routingKey: "celery",
basicProperties: props,
body: body);
Celery comes with Flower. Flower provides a REST API to managing tasks. https://flower.readthedocs.io/en/latest/api.html#post--api-task-async-apply-(.+)
In most cases this will be much simpler and robust to use than creating tasks manually and inserting them on the MQ.
According to this article, celery .Net client uses default TaskScheduler that comes with .Net Framework. This knows how to generate ID for your task. This article also points to some example here.
Related
The following is the exception i receive .Net core and rabbitMQ :
The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=405, text='RESOURCE_LOCKED - cannot obtain exclusive access to locked queue 'demo-queue' in vhost '/'. It could be originally declared on another connection or the exclusive property value does not match that of the original declaration.', classId=50, methodId=10
Producer :
var factory = new ConnectionFactory
{
Uri = new Uri("amqp://guest:guest#localhost:5672")
};
using var connection= factory.CreateConnection();
using var channel= connection.CreateModel();
channel.QueueDeclare("demo-queue",durable:true,exclusive:true,autoDelete:false, arguments:null);
var message = new
{
Name = "Producer",
Message = "Hello!"
};
var body = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(message));
channel.BasicPublish("", "demo-queue",null,body);
Consumer:
var factory = new ConnectionFactory
{
Uri = new Uri("amqp://guest:guest#localhost:5672")
};
using var connection = factory.CreateConnection();
using var channel = connection.CreateModel();
channel.QueueDeclare("demo-queue", durable: true,
exclusive: true, autoDelete: false,
arguments: null);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (sender, e) =>
{
var body = e.Body.ToArray();
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(message);
};
channel.BasicConsume("demo-queue",true,consumer);
Is it the right practice or right way IF I Change the name of the queue that I have given in producer or consumer? which is the right way to handle this?
In the QueueDeclare section, you said properties that are different from the queue you created.
Edit:
All your mistakes are in this line of code. Because you created a queue. Now you want to open it with the wrong properties. I think exclusive: false will solve the problem
//channel.QueueDeclare("demo-queue",durable:true,exclusive:true,autoDelete:false, arguments:null);
channel.QueueDeclare(queue: "demo-queue",
durable: true,
exclusive: false,
autoDelete: false,
arguments: null);
Exchange and queu need to be connected. If you do not do this, the message cannot know the queue address it will go to. After QueueDeclare, it must be bind with the exchange address. I have implemented it this way. If you want, you can look at the repo I created here.
https://github.com/oguzhanKomcu/RabbitMQ_Sample
I think this is the missing function.
public void BindQueu(string exName, string queuName, string routingKey)
{
channel.QueueBind(queuName,exName, routingKey);
}
So to explain my problem, I have to give you the context.
I got a Bot built with microsoft bot framework deployed on slack. Now it can happen these "events" on my backend that the bot communicates with. When a event occurs, I want to notify my bot of it and then let it send a message to all of it's conversations that something has happend. So basicly:
Backend>Microserivce>Bot>users
To do this I have to store all conversations in my backend, which I do in a database there. When a event happends, the backend will post an activity to the bot with all the conversations(basicly their id's) and the event it should show them.
So in essence my backend need to post a message to my bot.
For doing this I found the microsoft directline api which acts as a middleman here, a more abstract way to talk with the bot. The problem is that I don't know how to do it. I followed microsofts own tutorial but it doesn't seem to work for me:
This is the endpoint that my backend uses to notify the bot. "content" contains conversations and events as a json formated string.
[HttpPost]
[Route("conversationsEvents")]
public HttpResponseMessage PostConversationsEvents([FromBody]string content)
{
NotifyBot.Notify(content);
return Request.CreateResponse(HttpStatusCode.NoContent );
}
NotifyBot.Notify(content) looks like this:
private static async Task StartBotConversation( string contents)
{
string directLineSecret = "secret";
string fromUser = "microserviceNotifyEndpoint";
Activity activity = new Activity
{
From = new ChannelAccount(fromUser),
Text = contents,
Type = ActivityTypes.Event
};
DirectLineClient client = new DirectLineClient(directLineSecret);
var conversation = await client.Conversations.StartConversationAsync();
await client.Conversations.PostActivityAsync(conversation.ConversationId, activity);
}
Basicly the execution get's stuck at var conversation = await client.Conversations.StartConversationAsync(); , it just waits forever.
I tried changing it to var conversation = await client.Conversations.StartConversationAsync().ConfigureAwait(continueOnCapturedContext: false);´the execution goes on but the activity doesn't seem to get posted.
I'm not sure why the call to .StartConversationAsync() would freeze in your case. Maybe you haven't enabled the Direct Line channel on dev.botframework.com/bots? Nonetheless, as pointed out by Sergey, the Direct Line is a Channel and not a means for communicating with your bot on other channels.
Check out the Connector Client: bot-builder-dotnet-connector
Here is a static example of using it to proactively send a message to a user from a bot: MicrosoftDX/botFramework-proactiveMessages - sample: ConversationStarter.cs
pertinent code from sample:
public static async Task Resume(string conversationId,string channelId)
{
var userAccount = new ChannelAccount(toId,toName);
var botAccount = new ChannelAccount(fromId, fromName);
var connector = new ConnectorClient(new Uri(serviceUrl));
IMessageActivity message = Activity.CreateMessageActivity();
if (!string.IsNullOrEmpty(conversationId) && !string.IsNullOrEmpty(channelId))
{
message.ChannelId = channelId;
}
else
{
conversationId = (await connector.Conversations.CreateDirectConversationAsync( botAccount, userAccount)).Id;
}
message.From = botAccount;
message.Recipient = userAccount;
message.Conversation = new ConversationAccount(id: conversationId);
message.Text = "Hello, this is a notification";
message.Locale = "en-Us";
await connector.Conversations.SendToConversationAsync((Activity)message);
}
The serviceUrl, the channelId, conversationId, toId, fromId, etc are cached from previous communication by the user to the bot (these are statically stored in this example, so only work for one user). This example shows how it is possible to proactively send a message to a user from a bot. The Direct Line api is not required.
You don't need to use DirectLine, it is designed for creating alternative bot UIs.
To implementing what your want, you may try the following:
First, you need to store users addresses to whom you want to send the messages. It my be done by storing the ResumptionCookie of a user last message in your backend database.
var state = new ResumptionCookie(message).GZipSerialize();
When your PostConversationsEvents is called, you may resume the conversation at the latest point with each users.
var resumptionCookie = ResumptionCookie.GZipDeserialize(state);
var message = resumptionCookie.GetMessage();
message.Text = content;
await Conversation.ResumeAsync(resumptionCookie, message);
It is not the only solution. As I said, in this case you just resumed the conversation with the user at the latest point. Another solution is to save the user address (user the same ResumptionCookie class) but start the conversation when you need to:
var resumptionCookie = ResumptionCookie.GZipDeserialize(state);
var message = cookie.GetMessage();
ConnectorClient client = new ConnectorClient(new Uri(message.ServiceUrl));
var conversation = await
client.Conversations.CreateDirectConversationAsync(message.Recipient, message.From);
message.Conversation.Id = conversation.Id;
var newMessage = message.CreateReply();
newMessage.Text = content;
await client.Conversations.SendToConversationAsync(newMessage);
See more details on BotFramework documentation.
We're currently switching out NServiceBus for MassTransit and I'm having a little difficulty with the request/response pattern.
In NServiceBus, I'm able to do reply in the Handler and it goes back to the client that sent it.
In MassTransit, it appears as though the response is being sent back to the queue that it was consumed from, thus creating a loop...
Weird thing, if I'm creating the Bus using InMemory, and both client and consumer on the same machine, I do not have the issue.
I am expecting my client to catch the response, but instead my Consumer picks it up, which is also odd, since it's not setup to receive that message type...
Am I missing something in the client's Request setup?
Client:
....
IRequestClient<IWorklistRequest, IWorklistResponse> client = CreateRequestClient(busControl, WorklistEndpointUri);
Console.Write("Sending Request");
Task.Run(async () =>
{
IWorklistRequest request = new WorklistRequest
{
CurrentDateFrom = new DateTime(2016, 11, 07)
};
var response = await client.Request(request);
Console.WriteLine("Worklist Items retrieved: {0}", response.ExamItemList.Length);
}).Wait();
....
static IRequestClient<IWorklistRequest, IWorklistResponse> CreateRequestClient(IBusControl busControl, string endpointAddress)
{
Console.WriteLine("Creating Request client...");
var serviceAddress = new Uri(endpointAddress);
IRequestClient<IWorklistRequest, IWorklistResponse> client =
busControl.CreateRequestClient<IWorklistRequest, IWorklistResponse>(serviceAddress, TimeSpan.FromSeconds(10));
return client;
}
Consumer:
public Task Consume(ConsumeContext<IWorklistRequest> context)
{
_log.InfoFormat("Received Worklist Request with Id: {0}", context.RequestId);
try
{
var result = _provider.GetAllWorklistsByStartDate(context.Message.CurrentDateFrom);
IWorklistResponse response = new WorklistResponse
{
ExamItemList = result.ToArray()
};
// the below is sending the response right back to the original queue and is getting picked up again by this same consumer
context.Respond(response);
}
catch (Exception ex)
{
_log.Info(ex.Message);
}
return Task.FromResult(0);
}
If you are using RabbitMQ, and you are using the request client, you should not see this behavior.
There is a sample that demonstrates how to use the request client on the MassTransit GitHub repository: https://github.com/MassTransit/Sample-RequestResponse
The code above appears to be correct, and the Respond() call should use the response address from the request message, which is a direct endpoint send to the temporary bus address.
There is pretty extensive unit test coverage around this area, and the sample above was updated and verified with the latest version of MassTransit. You might consider deleting/recreating your RabbitMQ virtual host and running your application from scratch (start the response service first, so that the endpoints are setup).
I am using rabbit-Mq in my web app(Asp.net-MVC 4.0). My requirement is to send a message to a particular user. Suppose if user1 is online and he sends a message to user2 by rabbit-Mq. It should be received by "user2" only. The code I have used is a template which stores the message in the queue and whenever the user clicks on receive he will get that message but there is no restriction of particular user in my case. Anyone can get that message which is wrong and I have to handle that. Please help me regarding this.
Do we have something in rabbit-Mq that can distinguish the correct message to correct user/Consumer? Can we set a key with message and check the key while receiving?
Is this possible?
Below I am writing the code I am using to send and receive the messages
public ActionResult SendMessage(MessagingModel ObjModel)
{ var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
Message = ObjModel.Message;
channel.QueueDeclare("MessageQueue", true, false, false, null);
var body = Encoding.UTF8.GetBytes(ObjModel.Message);
channel.BasicPublish("", "MessageQueue", null, body);
}
}
}
public JsonResult RecieveMessage()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("MessageQueue", true, false, false, null);
bool noAck = true;
BasicGetResult result = channel.BasicGet("MessageQueue", noAck);
if (result == null)
{
Message = "No Messages Found.";
}
else
{
IBasicProperties props = result.BasicProperties;
byte[] Body = result.Body;
Message = Encoding.Default.GetString(Body);
}
}
}
First, you must remember the following things:
All messages in RabbitMQ published through exchanges.
Queues binded to exchanges.
Event if you publish message directly into queue, actually it still passes through the default exchange - (AMPQ default).
There are different kinds of exchanges. You can read a bit about exchanges here: https://www.rabbitmq.com/tutorials/tutorial-three-dotnet.html
In you case you might consider use a topic or headers exchanges, but in this case you should for each user
have a queue, and if the number of users in the system is large, then it will be very resource intensive.
Also you can add specific header to you message:
var props = model.CreateBasicProperties();
props.Headers.Add("UserId", userId);
and then in RecieveMessage() method after read message from queue see this header and if message intended for current user - receive it and acknowledge this message, otherwise
not acknowledge this message.
But this is bad solution. I would just kept messages from the queue to the database, and then read them out filtering by user.
I am trying to implement a 2-way chat using UCWA(Lync web app) , I am able to send the data to a client , but how do i receive it back from that client ? to implement a proper 2-way chat ?
I have attached the code to send some ping to a specific user , how do i receive data back from the user to implement the chat ?
private void StarConversation(string startConversationLink, string hostName, string accessToken, string tokenType)
{
var myObject = new JObject();
myObject.Add("operationId", Guid.NewGuid().ToString().Replace("-", ""));
myObject.Add("to", "sip:xyz#abcd.com");
myObject.Add("subject", "TestConv");
List<string> outgoing = new List<string>();
List<string> incoming = new List<string>();
List<string> timeout = new List<string>();
JObject href = new JObject();
string sending = "Hey there ! how are you doing ?";
href.Add("href", "data:text/plain;base64," + Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(sending)));
JObject message = new JObject();
message.Add("message", href);
myObject.Add("_links", message);
HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Add("Authorization", tokenType + " " + accessToken);
client.PostAsync(new Uri(hostName + startConversationLink), new StringContent(JsonConvert.SerializeObject(myObject), Encoding.UTF8, "application/json")).Result;
if (response.IsSuccessStatusCode)
{
Uri temp = new Uri(hostName + m_eventsLink);
HttpResponseMessage response2 = Get(client, temp.ToString());
string nextLink = string.Empty;
string timeo ;
string ss = string.Empty;
int i =0;
if (response2.IsSuccessStatusCode)
{
string retData2 = response2.Content.ReadAsStringAsync().Result;
var o = JObject.Parse(retData2);
nextLink = o["_links"]["next"]["href"].Value<string>();
response = client.PostAsync(new Uri(hostName + startConversationLink), new StringContent(JsonConvert.SerializeObject(myObject), Encoding.UTF8, "application/json")).Result;
temp = new Uri(hostName + m_eventsLink);
response2 = Get(client, temp.ToString());
retData2 = response2.Content.ReadAsStringAsync().Result;
o = JObject.Parse(retData2);
}
}
}
What you are doing at this point is starting a Conversation with the user and providing an initial message to go with the messagingInvitation. What I do not know from this code slice is:
Are you listening to Events? (This is really important as this is how you will know that the 2nd party accepted/declined the messagingInvitation among other things like who is typing and messages). I would suggest looking into GettingStarted-Events and setting up a mechanism to request events.
Are you making frequent (~3-5min) POST request on reportMyActivity to ensure the application is not recycled. Check out Resources-reportMyActivity if not.
If the code is listening to events and reporting activity. The next step will be listening for the messagingInvitation and check conversation/messaging events. Conversation will transition from Connecting (state) (messageInvitation is being processed) to Connected (messagingInvitation was accepted) or Disconnected (messagingInvitation was declined, conversation ended, other errors). Assuming the other user accepts the next event to check is messaging state for Connected (this means the messaging modality is active in the conversation and useful methods like sendMessage, addParticipant, and setIsTyping will be available) or Disconnected (this means messaging modality is not active and addMessaging can be used to enable this modality.
If you can reach this point message events will appear when any participant sends a message to the conversation. The direction property can be used to determine when the current UCWA user has sent a message (outgoing) and for all other users there should be a participant property that can identify the user.