We have been using bidirectional grpc streams in our architecture extensively, but have been experiencing issues when the CPU load on one end does not allow it to read data as fast as the other end is sending it. I have been struggling to understand how grpc is supposed to handle this situation. I've made a test program that mimics in a simple fashion this situation (using C# and the async APIs which we use exclusively). The server side is unable to keep up with the rate of data coming from the client. The result is the client side's memory usage continually climbs until ultimately OOMing the machine. My understanding is grpc is supposed to have some sort of protection against this.
I found the channel argument named GRPC_ARG_HTT2_WRITE_BUFFER_SIZE thinking it might limit the client side memory usage (and either cause blocking or an exception on the client end once that buffer is filled). This test program is sending 10000 byte messages and the write buffer size is set to 11000, yet it appears setting this argument has no effect.
My questions are: is grpc behaving correctly in this example and if so what am I doing incorrectly? Why does GRPC_ARG_HTT2_WRITE_BUFFER_SIZE seem to have no effect (and perhaps what is the intended effect)?
The sample is written using the following message/service:
syntax = "proto3";
package test;
option csharp_namespace = "Test";
service TestService {
rpc Publish(stream TestMsg) returns (stream TestMsg);
}
message TestMsg {
string value = 1;
bytes dummy = 2;
}
and the C# program is here:
using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Google.Protobuf;
using Grpc.Core;
namespace Test
{
class Program
{
static void Main( string[] args )
{
if ( args[0] == "client" )
{
using ( new Client() )
Console.ReadKey();
}
else if ( args[0] == "server" )
{
using ( new Server() )
Console.ReadKey();
}
}
}
public class Server : TestService.TestServiceBase, IDisposable
{
private readonly Grpc.Core.Server _server;
public Server()
{
_server = new Grpc.Core.Server
{
Services = { TestService.BindService( this ) },
Ports = { new ServerPort( "localhost", 1234, ServerCredentials.Insecure ) }
};
_server.Start();
Console.WriteLine( "Server started" );
}
public void Dispose()
{
_server.ShutdownAsync();
}
public override async Task Publish(
IAsyncStreamReader<TestMsg> requestStream,
IServerStreamWriter<TestMsg> responseStream,
ServerCallContext context )
{
try
{
Console.WriteLine( "Client connected" );
for (; ; )
{
var requestTask = requestStream.MoveNext();
await requestTask;
if ( requestTask.Status == TaskStatus.Canceled )
break;
if ( !requestTask.Result )
break;
var request = requestStream.Current;
if ( request != null )
{
try
{
Console.Write( request.Value + ".." );
// We're really working hard.
Thread.Sleep( 1000 );
}
catch ( Exception ex )
{
await responseStream.WriteAsync( new TestMsg { Value = ex.Message } );
Console.WriteLine( ex );
}
}
}
}
finally
{
Console.WriteLine( "Client disconnected" );
}
}
}
public class Client : IDisposable
{
private bool _isSending;
private readonly TestService.TestServiceClient _client;
private readonly Channel _channel;
private readonly IClientStreamWriter<TestMsg> _requestStream;
private readonly Timer _timer;
private int _i;
public Client()
{
Console.WriteLine( "Client started" );
var options = new List<ChannelOption>();
options.Add( new ChannelOption( "GRPC_ARG_HTTP2_WRITE_BUFFER_SIZE", 11000 ) );
_channel = new Channel( "localhost:1234", ChannelCredentials.Insecure, options );
_client = new TestService.TestServiceClient( _channel );
var call = _client.Publish();
_requestStream = call.RequestStream;
_timer = new Timer( OnTimerTick, null, 0, 1000 / 25 );
}
public void Dispose()
{
_channel.ShutdownAsync();
_timer.Dispose();
}
private async void OnTimerTick( object state )
{
// Skip timer ticks if the previous isn't done yet.
if ( _isSending )
return;
try
{
_isSending = true;
var bytes = new byte[10000];
var msg = new TestMsg { Value = _i.ToString(), Dummy = ByteString.CopyFrom( bytes ) };
Console.Write( _i + ".." );
await _requestStream.WriteAsync( msg );
++_i;
}
catch ( Exception e )
{
Console.WriteLine( e );
}
finally
{
_isSending = false;
}
}
}
}
Note: I'm using grpc.core.1.18.0 nuget package for my testing.
My service is register with azzure notification hub. And using my .net server API it push notification to particular device within particular time frame.
Everything goes right except when I try to send multiple push in same code it stuck with "BadRequest" except first one.
Below is the code
public static async void SendAzzurePushNotification()
{
for (int i = 0; i < 10; i++)
{
HttpStatusCode pushNotificationStatus = await CreateAndPushAsync("user_37");
Console.WriteLine(pushNotificationStatus);
}
}
static async Task<HttpStatusCode> CreateAndPushAsync(string tag)
{
HttpStatusCode pushNotificationStatus = HttpStatusCode.NotImplemented;
try
{
HttpResponseMessage response = null;
string uri = "<HUBURI>";
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("SharedAccessSignature", <SASTOKEN>);
client.DefaultRequestHeaders.Add("ServiceBusNotification-Format", "gcm");
client.DefaultRequestHeaders.Add("ServiceBusNotification-Tags", tag);
client.DefaultRequestHeaders.Add("x-ms-version", "2015-01");
response = await client.PostAsync(uri,
new StringContent("{\"data\":{\"message\":\"Notification Hub test notification\"}}", Encoding.UTF8, "application/json"));
pushNotificationStatus = response.StatusCode;
}
catch (Exception ex)
{
throw;
}
return pushNotificationStatus;
}
Above code give me Created status for first time and then BadRequest after that. If same api I call from client like postman. It work fine.
I also tried nuget package from azure notification hub, regarding which code is as below. Which solve my above issue but it won't return me any status code which I can have in my above code for success.
NotificationHubClient hub = NotificationHubClient.CreateClientFromConnectionString("<CONNECTIONSTRING>", "<HUB>");
NotificationOutcome outcome = await hub.SendGcmNativeNotificationAsync("{\"data\":{\"message\":\"Notification Hub test notification\"}}", "user_37");
Call send method with your tags and your notification-data
private static readonly string Endpoint = #"Your End Point";
private static readonly string HubName = #"You Hub Name";
private static NotificationHubClient Hub { get { return NotificationHubClient.CreateClientFromConnectionString(Endpoint, HubName); } }
public static async Task Send(string[] tags, object data)
{
try
{
string payload = string.Empty;
string json_gcm = string.Empty;
if (data.GetType() != typeof(string))
{
//If your notification data is of type
payload = JsonConvert.SerializeObject(data);
json_gcm = "{ \"data\" : " + payload + "}";
}
else
{
//If your notification data is simply is in string
payload = Convert.ToString(data);
json_gcm = "{ \"data\" : {\"message\":\"" + payload + "\"}}";
}
// Android
NotificationOutcome gcmOutcome = null;
gcmOutcome = await Hub.SendGcmNativeNotificationAsync(json_gcm, tags);
if (gcmOutcome != null)
{
if (!((gcmOutcome.State == NotificationOutcomeState.Abandoned) || (gcmOutcome.State == NotificationOutcomeState.Unknown)))
{
//Do code when notification successfully send to Android
}
}
}
catch (Exception ex)
{
//Do code when any exception occurred while sending notification
}
}
NotificationOutcomeState: Gives you status code in the form of enum that represent your notification has been successfully sent or not.
You may ignore if-else block as your need.
Try once may it help you
How do you delete the dead letters in an Azure Service Bus queue?
I can process the messages in the queue ...
var queueClient = QueueClient.CreateFromConnectionString(sbConnectionString, queueName);
while (queueClient.Peek() != null)
{
var brokeredMessage = queueClient.Receive();
brokeredMessage.Complete();
}
but can't see anyway to handle the dead letter messages
The trick is to get the deadletter path for the queue which you can get by using QueueClient.FormatDeadLetterPath(queueName).
Please try the following:
var queueClient = QueueClient.CreateFromConnectionString(sbConnectionString, QueueClient.FormatDeadLetterPath(queueName));
while (queueClient.Peek() != null)
{
var brokeredMessage = queueClient.Receive();
brokeredMessage.Complete();
}
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Microsoft.Azure.ServiceBus;
using Microsoft.Azure.ServiceBus.Core;
using System.Text;
using System.Collections.Generic;
using System.Threading.Tasks;
namespace ClearDeadLetterQ
{
[TestClass]
public class UnitTest1
{
const string ServiceBusConnectionString = "Endpoint=sb://my-domain.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=youraccesskeyhereyouraccesskeyhere";
[TestMethod]
public async Task TestMethod1()
{
await this.ClearDeadLetters("my.topic.name", "MySubscriptionName/$DeadLetterQueue");
}
public async Task ClearDeadLetters(string topicName, string subscriptionName)
{
var messageReceiver = new MessageReceiver(ServiceBusConnectionString, EntityNameHelper.FormatSubscriptionPath(topicName, subscriptionName), ReceiveMode.PeekLock);
var message = await messageReceiver.ReceiveAsync();
while ((message = await messageReceiver.ReceiveAsync()) != null)
{
await messageReceiver.CompleteAsync(message.SystemProperties.LockToken);
}
await messageReceiver.CloseAsync();
}
}
}
There are some great samples available in our GitHub sample repo (https://github.com/Azure-Samples/azure-servicebus-messaging-samples). The DeadletterQueue project should show you an example of how to do this in your code:
var dead-letterReceiver = await receiverFactory.CreateMessageReceiverAsync(
QueueClient.FormatDeadLetterPath(queueName), ReceiveMode.PeekLock);
while (true)
{
var msg = await dead-letterReceiver.ReceiveAsync(TimeSpan.Zero);
if (msg != null)
{
foreach (var prop in msg.Properties)
{
Console.WriteLine("{0}={1}", prop.Key, prop.Value);
}
await msg.CompleteAsync();
}
else
{
break;
}
}
}
Hope that helps!
Open Azure into your target Bus Queue Service.
(Set this value in place of <BUS-QUEUE-NAME> on the code)
Go into the queue you want to delete.
(Set this value in place of <QUEUE-NAME> on the code)
Create a Shared Access Policy and name it: RemoveDeadLetterQueue with checkbox Manage been selected.
Copy this Primary Key onto <QUEUE-POLICY-PRIMARYKEY> in this code.
And the code is ready to run.
using Microsoft.Azure.ServiceBus.Core;
using Microsoft.Azure.ServiceBus;
string serviceBusQueue = "<BUS-QUEUE-NAME>";
string serviceBusQueueName = "<QUEUE-NAME>";
string policyName = "RemoveDeadLetterQueue";
string policyPrimaryKey = "<QUEUE-POLICY-PRIMARYKEY>";
var receiver = new MessageReceiver(
connectionString: $"Endpoint=sb://{serviceBusQueue}.servicebus.windows.net/;SharedAccessKeyName={policyName};SharedAccessKey={policyPrimaryKey}",
entityPath: $"{serviceBusQueueName}/$DeadLetterQueue",
receiveMode: ReceiveMode.ReceiveAndDelete
);
var messages = await receiver.ReceiveAsync(maxMessageCount: 1000);
while(messages != null)
{
foreach (var message in messages)
{
Console.WriteLine($"[Delete Message] ID: {message.MessageId}");
}
messages = await receiver.ReceiveAsync(maxMessageCount: 1000);
}
await receiver.CloseAsync();
I am new to WebSockets (this AM) and have set up a WCF WebSocket app that works when doing a trivial example I found online (http://www.codeproject.com/Articles/619343/Using-WebSocket-in-NET-Part).
I added Entity Framework and as soon as I add code to try to access data the process (just sending a message back and forth) no longer works.
Could there be some fundamental concept I could be missing?
Does anyone have any good ideas for troubleshooting?
namespace PBWebSocket
{
public class PBWebSocket : IBWebSocket
{
private SPEntities db = new SPEntities();
public async Task SendMessageToServer(Message msg)
{
var callback = OperationContext.Current.GetCallbackChannel<IPBCallback>();
if (msg.IsEmpty || ((IChannel)callback).State != CommunicationState.Opened)
{
return;
}
byte[] body = msg.GetBody<byte[]>();
string msgTextFromClient = Encoding.UTF8.GetString(body);
var reqId = Int32.Parse(msgTextFromClient);
// *** The below line breaks it ***
var req = db.Requests.Where(r => r.Id == 164).FirstOrDefault();
reqId = reqId + 2;
Message newMsg = ByteStreamMessage.CreateMessage(
new ArraySegment<byte>(Encoding.UTF8.GetBytes(reqId.ToString())));
newMsg.Properties["WebSocketMessageProperty"] =
new WebSocketMessageProperty
{ MessageType = WebSocketMessageType.Text };
await callback.SendMessageToClient(newMsg);
}
}
}
I have a problem with receiving messages from a queue i have set up in azure.
I have done this successfully using the same code before but now i just get null when i try to fetch messages.
When i view the queue in azure management console i clearly see that the queue contains 5 messages.
Here is the code:
ServiceBus SB = new ServiceBus();
Microsoft.ServiceBus.Messaging.BrokeredMessage message;
while (true)
{
message = SB.ReceiveMessage("orders");
if (message == null)
{
break;
}
Procurement.Order order = message.GetBody<Procurement.Order>();
order.id = Guid.NewGuid().ToString();
order.remindercount = 0;
using (DbManager db = new DbManager())
{
if (db.SetSpCommand("CreateOrderHead",
db.Parameter("#companyId", order.companyId),
db.Parameter("#orderId", order.orderId),
db.Parameter("#suppliercode", order.suppliercode),
db.Parameter("#supplierorderId", order.supplierorderId),
db.Parameter("#orderdate", order.orderdate),
db.Parameter("#desireddate", order.desireddate),
db.Parameter("#ordertext", order.ordertext),
db.Parameter("#name", order.name),
db.Parameter("#street", order.street),
db.Parameter("#zip", order.zip),
db.Parameter("#city", order.city),
db.Parameter("#country", order.country),
db.Parameter("#countrycode", order.countrycode),
db.Parameter("#deliveryterms", order.deliveryterms),
db.Parameter("#reference", order.reference),
db.Parameter("#deliveryinstruction", order.deliveryinstruction),
db.Parameter("#id", order.id),
db.Parameter("#partycode", order.partyCode)
).ExecuteNonQuery() == 1)
{
message.Complete();
message = null;
}
db.SetSpCommand("DeleteOrderRows",
db.Parameter("#orderid", order.orderId),
db.Parameter("#companyId", order.companyId)
).ExecuteNonQuery();
foreach (Procurement.Orderrow r in order.Orderrows)
{
db.SetSpCommand("CreateOrderRow",
db.Parameter("#companyId", r.companyId),
db.Parameter("#orderId", r.orderId),
db.Parameter("#orderrowId", r.orderrowId),
db.Parameter("#itemId", r.itemId),
db.Parameter("#itemdesc", r.itemdesc),
db.Parameter("#orderqty", r.orderqty),
db.Parameter("#desireddate", r.desireddate),
db.Parameter("#rowtext", r.rowtext),
db.Parameter("#supplieritemId", r.supplieritemId),
db.Parameter("#unit", r.unit),
db.Parameter("#id", order.id),
db.Parameter("#unitprice", r.unitprice),
db.Parameter("#rowprice", r.rowprice)
).ExecuteNonQuery();
}
}
}
Thread.Sleep(new TimeSpan(0, 1, 0));
And this is the ServiceBus-class:
public class ServiceBus
{
TokenProvider TokenProvider;
MessagingFactory Factory;
public ServiceBus()
{
TokenProvider = TokenProvider.CreateSharedSecretTokenProvider(GetIssuerName(), GetSecret());
Factory = MessagingFactory.Create(
GetURINameSpace(),
TokenProvider
);
}
public void SendMessage(string queue, BrokeredMessage message)
{
var client = Factory.CreateQueueClient(queue);
client.Send(message);
}
public BrokeredMessage ReceiveMessage(string queue)
{
var client = Factory.CreateQueueClient(queue, ReceiveMode.ReceiveAndDelete);
BrokeredMessage message = client.Receive();
return message;
}
private static Uri GetURINameSpace()
{
return ServiceBusEnvironment.CreateServiceUri("sb", GetNamespace(), string.Empty);
}
private static string GetNamespace()
{
return "Namespace i have verified its the right one";
}
private static string GetIssuerName()
{
return "Issuer i have verified its the right one";
}
private static string GetSecret()
{
return "Key i have verified its the right one";
}
}
I think this should be pretty straight forward but i cant find out what im doing wrong.
Its probably something small that im missing...
Anyways, thanks in advance!
Those BrokeredMessages you see in your SubcriptionDescription.MessageCount are not just regular messages but also the count of the messages in the $DeadLetterQueue-sub queue!!!
Use this code snippet to retrieve all messages from that sub-queue and print out their details. Rename [topic] and [subscription] to your actual ones:
MessagingFactory msgFactory = MessagingFactory.Create(_uri, _tokenProvider);
MessageReceiver msgReceiver = msgFactory.CreateMessageReceiver("[topic]/subscriptions/[subscription]/$DeadLetterQueue", ReceiveMode.PeekLock);
while (true)
{
BrokeredMessage msg = msgReceiver.Receive();
if (msg != null)
{
Console.WriteLine("Deadlettered message.");
Console.WriteLine("MessageId: {0}", msg.MessageId);
Console.WriteLine("DeliveryCount: {0}", msg.DeliveryCount);
Console.WriteLine("EnqueuedTimeUtc: {0}", msg.EnqueuedTimeUtc);
Console.WriteLine("Size: {0} bytes", msg.Size);
Console.WriteLine("DeadLetterReason: {0}",
msg.Properties["DeadLetterReason"]);
Console.WriteLine("DeadLetterErrorDescription: {0}",
msg.Properties["DeadLetterErrorDescription"]);
Console.WriteLine();
msg.Complete();
}
}
The solution to this problem was either a bug in azure management-portal making it show the wrong number of messages on the queue or the messages somehow got flagged so that they would not be read.
In other words it worked all along, i just had to add some new messages to the queue.