We Keep Coding

Unreliable count of messages pushed to Service Bus using Durable Functions

I'm getting this weird issue when using Durable Azure functions to submit messages to azure service bus.
My code is a simple Fan-Out implementation
REST trigger get the number of messages to be submitted and hands that an orchestrator.
Orchestrator stores the calls activity which will create and submit the message to Service bus.
The issue is when I send the REST parameter asking to add 3000 messages, more than 3000 get added.
Worse, it's not the same number either - 3104, 3100, 3286 anything...
See code below:
[FunctionName("Function1_HttpStart")]
//public static async Task<HttpResponseMessage> HttpStart(
public static async Task<IActionResult> HttpStart(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequest req,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log)
{
String type = req.Query["type"];
if(!long.TryParse(req.Query["count"], out var count))
{
return new ObjectResult($"Parse failed for parameter 'count' ({req.Query["count"]}) to Int.") { StatusCode = 400};
}
var restInputs = new RestInputs()
{ Type = type, Count = count };
// Function input comes from the request content.
string instanceId = await starter.StartNewAsync
("EmailQueueSubmitter_OrchestratorSingleton"
, restInputs);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
return starter.CreateCheckStatusResponse(req, instanceId);
}
[FunctionName("EmailQueueSubmitter_OrchestratorSingleton")]
public static async Task<List<string>> EmailQueueSubmitter_OrchestratorSingleton(
[OrchestrationTrigger] IDurableOrchestrationContext context, ILogger log)
{
var outputs = new List<string>();
try
{
var restInputs = context.GetInput<RestInputs>();
var parallelTasks = new List<Task>();
long runBatchLen;
long i_batch, i_iter, batchCount = 0;
for (i_batch = 0; i_batch < restInputs.Count; i_batch++)
{
parallelTasks.Add(context.CallActivityAsync("EmailQueueSubmitter_ActivitySendMessageBatchSingleton", i_batch.ToString()));
log.LogWarning($"Message {i_batch} Added");
}
log.LogWarning($"Awaiting {parallelTasks.Count} tasks");
await Task.WhenAll(parallelTasks);
var doneTaskCount = parallelTasks.Where(t => t.IsCompleted).ToList().Count;
var successTaskCount = parallelTasks.Where(t => t.IsCompletedSuccessfully).ToList().Count;
var faultedTaskCount = parallelTasks.Where(t => t.IsFaulted).ToList().Count;
var exceptionTaskCount = parallelTasks.Where(t => t.Exception != null).ToList().Count;
log.LogWarning($"Done:{doneTaskCount}, Success: {successTaskCount}, Fault:{faultedTaskCount}, Exception:{exceptionTaskCount}");
log.LogWarning($"Achieved completion.");
}
catch (Exception ex)
{
log.LogError(ex.Message);
throw new InvalidOperationException(ex.Message);
}
return outputs;
}
[FunctionName("EmailQueueSubmitter_ActivitySendMessageBatchSingleton")]
public static async Task EmailQueueSubmitter_ActivitySendMessageBatchSingleton([ActivityTrigger] IDurableActivityContext activityContext, ILogger log)
{
log.LogWarning($"Starting Activity.");
var payload = activityContext.GetInput<String>();
await ServiceBus_Sender.SendMessageBatch(payload);
log.LogWarning($"Finished Activity.");
}
public static ServiceBusMessage CreateMessage(String Payload)
{
try
{
var sbMsg = new ServiceBusMessage(Payload)
{
MessageId = Guid.NewGuid().ToString(),
ContentType = "text/plain"
};
//sbMsg.ApplicationProperties.Add("RequestType", "Publish");
return sbMsg;
}
catch (Exception ex)
{
throw new InvalidOperationException(ex.Message, ex);
}
}

Thanks #Camilo Terevinto for the information, I am converting this to an answer so that may help other community members:
As suggested in the comments, to run a duplicate check you could generate a Guid and send it together with the data, and then check that the Guid wasn't handled already before. Hopefully this resolved your issue.
OP Edit: Duplicates check was enabled by changing the service bus queue to be session enabled and have de-duplication turned on. The submitted messages' MessageId was set to be unique in each session. This is the only way I can think of to deal with the at-least-once guarantees...

How to use websockets in asp.net core

im trying to develop a game where i store a scoreboard in a text file which is stored on server (currently on localhost). I am using http get and post calls in order to communicate with the server and get and send the data that i want. Now i want to implement websockets in order to send a notification from the server to the c# client. The notification will just display on the console a message for the user, for example in mu case i want to display a message to the user each time a user is added to the scoreboard, each time the UpdateScoreBoard method is called. Based on tutorials i found online i have managed to build the following code, can anyone make it more clear for me how the i will build the websocket for the server and how i will initialize the websocket on the client? Thank you
Startup.cs (Server)
public void Configure(IApplicationBuilder app, IHostEnvironment env)
{
//deleted code
var webSocketOptions = new WebSocketOptions()
{
KeepAliveInterval = TimeSpan.FromSeconds(120),
ReceiveBufferSize = 4 * 1024
};
app.UseWebSockets(webSocketOptions);
app.Use(async (context, next) =>
{
if (context.Request.Path == "/ws")
{
if (context.WebSockets.IsWebSocketRequest)
{
WebSocket webSocket = await context.WebSockets.AcceptWebSocketAsync();
await Echo(context, webSocket);
}
else
{
context.Response.StatusCode = 400;
}
}
else
{
await next();
}
});
}
private async Task Echo(HttpContext context, WebSocket webSocket)
{
var buffer = new byte[1024 * 4];
WebSocketReceiveResult result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
while (!result.CloseStatus.HasValue)
{
await webSocket.SendAsync(new ArraySegment<byte>(buffer, 0, result.Count), result.MessageType, result.EndOfMessage, CancellationToken.None);
result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
}
await webSocket.CloseAsync(result.CloseStatus.Value, result.CloseStatusDescription, CancellationToken.None);
}
HttpClass.cs (Client) - where i call the http post request
public async override Task<List<Scoreboard>> UpdateScoreBoards(string username, int attempts, int seconds, DateTime date)
{
HttpResponseMessage response = null;
//Creating a new instance of object Scoreboard
//deleted code
var url = "http://localhost:5000/api/Scoreboard";
var socket_url = new Uri("ws://localhost:5000");
var exitEvent = new ManualResetEvent(false);
using (var client = new WebsocketClient(socket_url))
{
client.ReconnectTimeout = TimeSpan.FromSeconds(30);
client.ReconnectionHappened.Subscribe(info =>
Log.Information($"Reconnection happened, type: {info.Type}"));
client.MessageReceived.Subscribe(msg => Log.Information($"Message received: {msg}"));
await client.Start();
await Task.Run(() => client.Send("test"));
exitEvent.WaitOne();
}
// deleted code
}

can anyone make it more clear for me how the i will build the websocket for the server and how i will initialize the websocket on the client?
As the example that you referenced demonstrated, making use of WebSocket in ASP.NET Core, we can add the WebSockets middleware in the Configure method, then add/configure request delegate to check and handle incoming WebSocket requests.
And after transitioned a request to a WebSocket connection with AcceptWebSocketAsync() method, we can use the returned WebSocket object to send and receive messages.
In Echo method, we can also perform custom code logic to generate and send reply message/notification based on received message(s).
//received message
var mes = Encoding.UTF8.GetString(buffer, 0, result.Count);
//code logic here
//...
//create reply message
var reply_mes = $"You sent {mes}.";
byte[] reply_mes_buffer = Encoding.UTF8.GetBytes(reply_mes);
await webSocket.SendAsync(new ArraySegment<byte>(reply_mes_buffer, 0, reply_mes.Length), result.MessageType, result.EndOfMessage, CancellationToken.None);
Besides, ASP.NET Core SignalR is an open-source library that simplifies implementing real-time communication functionality. And it does support WebSockets transport and we can easily achieving push messages/notifications to all connected clients or specified subsets of connected clients.
For more information about ASP.NET Core SignalR, you can check this doc: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/websockets?view=aspnetcore-3.1

The only thing you need in your Startup is to add the UseWebsockets middleware.
Then you can define your own middleware and filter connections if they are websocket type like below:
Startup
public void Configure(IApplicationBuilder app, IWebHostEnvironment env) {
app.UseWebSockets();
app.UseMiddleware<SocketWare>();
}
Middleware
public class SocketWare {
private RequestDelegate next;
public SocketWare(RequestDelegate _next) {
this.next = _next;
}
public async Task Invoke(HttpContext context) {
if (!context.WebSockets.IsWebSocketRequest) {
return;
}
var socket=await context.WebSockets.AcceptWebSocketAsync();
await RunAsync(socket);
}
private async Task RunAsync(WebSocket socket) {
try {
var client = new ChatClient(socket);
await client.RunAsync();
} catch (Exception ex) {
throw;
}
}
}
In my middleware i prefer to keep my business logic in a separate class that gets the Websocket injected in it like below:
Client
public class ChatClient
{
private Task writeTask;
private Task readTask;
private WebSocket socket;
private CancellationTokenSource cts=new CancellationTokenSource();
ChatClient(WebSocket socket)
{
this.socket=socket;
}
public async Task RunAsync()
{
this.readTask=Task.Run(async ()=>await ReadLoopAsync(cts.Token),cts.Token);
this.writeTask=Task.Run(async()=>await WriteLoopAsync(cts.Token),cts.Token);
await Task.WhenAny(this.readTask,this.writeTask);
}
public async Task WriteLoopAsync()
{
Memory<byte> buffer=ArrayPool<byte>.Shared.Rent(1024);
try {
while (true) {
var result= await this.socket.ReceiveAsync(buffer,....);
var usefulBuffer=buffer.Slice(0,result.Count).ToArray();
var raw=Encoding.Utf8.GetString(usefulBuffer);
//deserialize it to whatever you need
//handle message as you please (store it somwhere whatever)
}
} catch (Exception ex) {
//socket error handling
//break loop or continue with go to
}
}
public async Task ReadLoopAsync()
{
try {
while (true) {
var data = await this.[someMessageProvider].GetMessageAsync() //read below !!!
var bytes = Encoding.UTF8.GetBytes(data);
//send the message on the websocket
await this.socket.SendAsync(data, WebSocketMessageType.Text, true, CancellationToken.None);
}
} catch (Exception ex) {
//do incorrect message/socket disconnect logic
}
}
}
Now regarding producing messages and consuming them. In your case you could define your producers as some Controller routes like below .You would hit a route , produce a message and publish it to some message broker. I would use a Message Queue (RabbitMQ) or even a Redis Pub/Sub as a message bus.
You would publish messages from your route(s) and then consume them in your ReadLoopAsync method from WebSocketClient (look above).
Producing messages
public UpdateController:Controller
{
private IConnection
[HttpPost]
[someroute]
public void UpdateScoreboard(string someMessage)
{
this.connection.Publish("someChannel",someMessage);
}
[HttpPost]
[someotherroute]
public void DeletePlayer(string someOtherMessage)
{
this.connection.Publish("someChannel",someMessage);
}
}
Redis pub/sub Check redis pub/sub here Also check my repository
on github here in which i am
using exactly what you need (websockets, redis,pub sub)
RabbitMq Another option as a message bus is to use RabbitMQ , for more info regarding C# API
here
In Memory
You could also avoid using a third party and use some in memory data
structure like a BlockingCollection.You could inject it as a
Singleton service both in your Controller(s) and your socket
Middleware(s)

SignalR client side receives calls from dashboard client but not ecommerce client

I have 2 Vue.js clients that share the same backend server (ASP.NET Core). One of the clients is an e-commerce site, and the other client is the admin dashboard that manages the orders.
I have implemented SignalR to allow live order updates on the order dashboard table. The updates currently work in real time without the need to refresh the browser when I update an order's status on the admin dashboard, e.g. when updating from "Out for Delivery" to "Delivered" on browser 1, browser 2 displays the changes immediately without having to refresh the page.
Order updated via dashboard, SignalR event received
However, when I create a new order from the e-commerce site, even though I have checked that my code enters the Hub method to broadcast the event and the new order details to all clients, the order dashboard does not update/refresh, and it does not receive any events as well when I checked the dev tools. I would appreciate some help on this problem.
Hub methods in mounted()
// Establish hub connection
this.connection = await OrderHub.connectToOrderHub();
// Establish hub methods
this.connection.on("OneOrder", order => {
console.log("OneOrder called");
console.log(order);
this.getAllOrders();
});
this.connection.on("MultipleOrders", orders => {
console.log("MultipleOrders called");
console.log(orders);
this.getAllOrders();
});
// start the connection
this.connection
.start()
.then(() => {
console.log("Connection to hub started");
})
.catch(err => console.log(err));
orderHub.js
const signalR = require("#aspnet/signalr");
class OrderHub {
async connectToOrderHub() {
return new signalR.HubConnectionBuilder()
.withUrl("https://localhost:44393/order-hub")
.configureLogging(signalR.LogLevel.Error)
.build();
}
}
export default new OrderHub();
OrderHub.cs in server side
public interface IOrderHub
{
Task NotifyOneChange(Order newOrder);
Task NotifyMultipleChanges(List<Order> newOrders);
}
public class OrderHub : Hub, IOrderHub
{
private readonly IHubContext<OrderHub> _hubContext;
public OrderHub(IHubContext<OrderHub> hubContext)
{
_hubContext = hubContext;
}
public async Task NotifyOneChange(Order newOrder)
{
await _hubContext.Clients.All.SendAsync("OneOrder", newOrder);
}
public async Task NotifyMultipleChanges(List<Order> newOrders)
{
await _hubContext.Clients.All.SendAsync("MultipleOrders", newOrders);
}
}
Create Order method
public async Task<Order> Create(Order order)
{
Order newOrder;
try
{
List<string> imgKeys = new List<string>();
foreach (OrderItem item in order.OrderItems)
{
imgKeys.Add(item.OrderImageKey);
}
// make images on s3 permanent
List<string> imgUrls = await _s3Service.CopyImagesAsync(imgKeys);
// put the new images url into object
for (int i = 0; i < order.OrderItems.Count; i++)
{
order.OrderItems.ElementAt(i).OrderImageUrl = imgUrls.ElementAt(i);
}
// create new order object to be added
newOrder = new Order()
{
CreatedAt = DateTime.Now,
UpdatedAt = DateTime.Now,
OrderSubtotal = decimal.Parse(order.OrderSubtotal.ToString()),
OrderTotal = decimal.Parse(order.OrderTotal.ToString()),
ReferenceNo = order.ReferenceNo,
Request = order.Request,
Email = EncryptString(order.EmailString, encryptionKey),
UpdatedById = order.UpdatedById,
DeliveryTypeId = order.DeliveryTypeId,
Address = order.Address,
StatusId = 1,
OrderItems = order.OrderItems
};
// add to database
await _context.Orders.AddAsync(newOrder);
await _context.SaveChangesAsync();
}
catch (Exception ex)
{
throw new AppException("Unable to create product record.", new { message = ex.Message });
}
await _orderHub.NotifyOneChange(newOrder); // it steps inside & executes the method, but client side does not receive any events
// returns product once done
return newOrder;
}

I managed to figure out the cause of this issue. SignalR seems to break when the message is over a certain size, and this issue is referenced here. To fix this, I decided to pass in the order's ID instead of the whole order object, and do another GET call to retrieve the details in a separate API call to keep the SignalR message small.

How to get the Service URL of a deployed bot application in Microsoft Bot framework?

I am using DirectLine API to send message to the bot, I need the service URL of the Published Bot to perform a post request for the load test as mentioned in the steps here https://blog.botframework.com/2017/06/19/Load-Testing-A-Bot/
This is the code, can anyone point where I am going wrong
private static async Task<Chat> TalkToTheBot(string Message)
{
Chat objChat = null;
// Connect to the DirectLine service
try
{
DirectLineClient client = new DirectLineClient(directLineSecret);
Conversation conversation = await client.Conversations.StartConversationAsync();
string watermark = null;
Activity reply = new Activity
{
From = new ChannelAccount("User1", "User Name"),
Text = "Hello",
Type = ActivityTypes.Message,
};
//await client.Conversations.PostActivityAsync(conversation.ConversationId, reply.CreateReply(text: Message, locale: "en-US"), CancellationToken.None);
await client.Conversations.PostActivityAsync(conversation.ConversationId,reply , CancellationToken.None);
// Get the response as a Chat object
objChat = await ReadBotMessagesAsync(client, conversation.ConversationId, watermark);
}
catch (Exception e)
{
throw;
}
// Return the response as a Chat object
return objChat;
}
private static async Task<Chat> ReadBotMessagesAsync(DirectLineClient client, string conversationId, string watermark)
{
// Create an Instance of the Chat object
Chat objChat = new Chat();
// We want to keep waiting until a message is received
bool messageReceived = false;
while (!messageReceived)
{
// Get any messages related to the conversation since the last watermark
ActivitySet messages = await client.Conversations.GetActivitiesAsync(conversationId, watermark, CancellationToken.None);
// Set the watermark to the message received
watermark = messages?.Watermark;
// Get all the messages
var messagesFromBotText = from message in messages.Activities
where message.From.Id == botId
select message;
// Loop through each message
foreach (var message in messagesFromBotText)
{
// We have Text
if (message.Text != null)
{
// Set the text response
// to the message text
objChat.ChatResponse
+= " "
+ message.Text.Replace("\n\n", "<br />");
}
}
// Mark messageReceived so we can break
// out of the loop
messageReceived = true;
}
// Set watermark on the Chat object that will be
// returned
objChat.watermark = watermark;
// Return a response as a Chat object
return objChat;
}

Per the article,
The serviceUrl property here is critical to note, and needs to be set
to the endpoint of your message sink/client.
and:
In order to test your bot, you’ll need to create a custom UI/message
sink to send and receive messages to your bot. This message sink will
effectively act like a channel and accept HTTP POST messages with
JSON-serialized bot framework activities.
Which basically means that you will have to build a "message client" and the url of that client is the one that you will have to provide in the serviceUrl of your request.

GetAsync azure call no result

Using VS 2017 Community. Azure.
I have Azure setup, I have a blank webapp created just for test purpose.
My actual site is an Angular2 MVC5 site, currently run locally.
The following is the code that should... Contact azure providing secret key(the site is registered in azure Active directory).
From this i get a token i then can use to contact azure api and get list of sites.
WARNING: code is all Sausage code/prototype.
Controller
public ActionResult Index()
{
try
{
MainAsync().ConfigureAwait(false);
}
catch (Exception e)
{
Console.WriteLine(e.GetBaseException().Message);
}
return View();
}
static async System.Threading.Tasks.Task MainAsync()
{
string tenantId = ConfigurationManager.AppSettings["AzureTenantId"];
string clientId = ConfigurationManager.AppSettings["AzureClientId"];
string clientSecret = ConfigurationManager.AppSettings["AzureClientSecret"];
string token = await AuthenticationHelpers.AcquireTokenBySPN(tenantId, clientId, clientSecret).ConfigureAwait(false);
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Authorization", "Bearer " + token);
client.BaseAddress = new Uri("https://management.azure.com/");
await MakeARMRequests(client);
}
}
static async System.Threading.Tasks.Task MakeARMRequests(HttpClient client)
{
const string ResourceGroup = "ProtoTSresGrp1";
// Create the resource group
// List the Web Apps and their host names
using (var response = await client.GetAsync(
$"/subscriptions/{Subscription}/resourceGroups/{ResourceGroup}/providers/Microsoft.Web/sites?api-version=2015-08-01"))
{
response.EnsureSuccessStatusCode();
var json = await response.Content.ReadAsAsync<dynamic>().ConfigureAwait(false);
foreach (var app in json.value)
{
Console.WriteLine(app.name);
foreach (var hostname in app.properties.enabledHostNames)
{
Console.WriteLine(" " + hostname);
}
}
}
}
Controller class uses a static helper class that gets the token from Azure...
public static class AuthenticationHelpers
{
const string ARMResource = "https://management.core.windows.net/";
const string TokenEndpoint = "https://login.windows.net/{0}/oauth2/token";
const string SPNPayload = "resource={0}&client_id={1}&grant_type=client_credentials&client_secret={2}";
public static async Task<string> AcquireTokenBySPN(string tenantId, string clientId, string clientSecret)
{
var payload = String.Format(SPNPayload,
WebUtility.UrlEncode(ARMResource),
WebUtility.UrlEncode(clientId),
WebUtility.UrlEncode(clientSecret));
var body = await HttpPost(tenantId, payload).ConfigureAwait(false);
return body.access_token;
}
static async Task<dynamic> HttpPost(string tenantId, string payload)
{
using (var client = new HttpClient())
{
var address = String.Format(TokenEndpoint, tenantId);
var content = new StringContent(payload, Encoding.UTF8, "application/x-www-form-urlencoded");
using (var response = await client.PostAsync(address, content).ConfigureAwait(false))
{
if (!response.IsSuccessStatusCode)
{
Console.WriteLine("Status: {0}", response.StatusCode);
Console.WriteLine("Content: {0}", await response.Content.ReadAsStringAsync());
}
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsAsync<dynamic>().ConfigureAwait(false);
}
}
}
}
ISSUE:
Ok so the issue I was faced with was Async Deadlocks in my code. So i looked at this stack post stack post here
I fixed the issues by putting in .ConfigureAwait(false) on most of the await declarations.
Code runs and gets all the way back to the controller with a token etc and runs through the MakeARMRequests(HttpClient client) method, however the json only returns 1 result "{[]}" when i debug and as such ignores the loops.
My question is, is my code the culprit here? or would this point to a configuration setting in azure?

Not sure if this is the issue you are facing now BUT you never wait for a result from your async action in the first method Index in your code. MainAsync().ConfigureAwait(false); will immediately return and continue to the next block while the task MainAsync() will start in the background. The catch handler also does nothing because you dont wait f or a result.
Option 1 (recommended)
public async Task<ActionResult> Index()
{
try
{
await MainAsync().ConfigureAwait(false);
}
catch (Exception e)
{
Console.WriteLine(e.GetBaseException().Message);
}
return View();
}
Option 2 if you can't use async/await for some reason
public ActionResult Index()
{
try
{
MainAsync().GetAwaiter().GetResult();
}
catch (Exception e)
{
Console.WriteLine(e.GetBaseException().Message);
}
return View();
}

The Code looks OK and runs fine, Anyone who could help verify would be good, but one can assume this is OK.
The issue for this was configuration in azure, When you register an app you must set a certain number of Access controls via the subscription.
In this case I set some more specific things for the web api , for now set the app as owner and made reference to service management api.
Probably don't need half the "IAM" added in the subscription to the registered app, I simply went through adding the relevant ones and debugging each time until finally i got the results expected.