Azure event hub End of stream reached while consuming message C# - c#

I'm using Azure eventhub with schema registry. I'm sending a message with a huge string.
The message has property JsonConent. I am using Avro schema for nullable string with a default value.
I am trying to convert the message to GenericRecord, when I consume it, but when the JsonContent is too large I have an error: Avro.AvroException - End of stream reached.
public async Task ProcessEventHandler(ProcessEventArgs arg)
{
var a = avroSerializer.Deserialize(arg.Data.EventBody.ToStream(), typeof(GenericRecord);
}
I am using Azure.Data.SchemaRegistry.ApacheAvro 1.0.0-beta.1,
Azure.Messaging.EventHubs 5.3.0-beta.4,
Azure.Messaging.EventHubs.Processor 5.3.0-beta.4

I found this issue, logged into the apache Jira board. Maybe Microsoft Azure is using it.

Related

Setting up an HTTP endpoint for Azure EventGrid Webhook in my MVS Webapp?

I'm trying to create an Azure Eventgrid and Webhook response to send me inbound text messages from Azure communications services to my Azure hosted WebApp. Right now I'm having a really hard time finding documentation that shows how to create an endpoint within a Webapp controller to get the Azure webhook response to handshake and validate. I've created an endpoint within my controller that I believe should be catching the the data and processing it in a POST method, but it fails because of the arguments I'm trying to mimic. Any insight on this topic is appreciated.
I tried integrating a lot of what I found in these Docs into my app controller to try and get it to work, but I think I might be doing this all the wrong way since it says this code is for an Azure function??? I'm not entirely sure how those are used, but I tried integrating the same C# code into my controller. See Docs below:
https://learn.microsoft.com/en-us/azure/event-grid/receive-events
And here is the controller I have that is trying to imitate what I read in the docs I linked
[HttpPost("incoming")]
public async Task<IActionResult> GetFlightInfo([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequest incoming,
ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
string response = string.Empty;
BinaryData events = await BinaryData.FromStreamAsync(incoming.Body);
log.LogInformation($"Received events: {events}");
EventGridEvent[] egEvents = EventGridEvent.ParseMany(events);
foreach (EventGridEvent eventGridEvent in egEvents)
{
// Handle system events
if (eventGridEvent.TryGetSystemEventData(out object eventData))
{
// Handle the subscription validation event
if (eventData is SubscriptionValidationEventData subscriptionValidationEventData)
{
log.LogInformation($"Got SubscriptionValidation event data, validation code: {subscriptionValidationEventData.ValidationCode}, topic: {eventGridEvent.Topic}");
// Do any additional validation (as required) and then return back the below response
var responseData = new SubscriptionValidationResponse()
{
ValidationResponse = subscriptionValidationEventData.ValidationCode
};
return new OkObjectResult(responseData);
}
}
}
return new OkObjectResult(response);
}
I'd suggest to start by deploying and exploring the Azure Event Grid Viewer sample application according to the Handle SMS events for Delivery Reports and Inbound Messages tutorial.
This app is designed to consume any events generated by the Event Grid, including the SMS ones. The app utilizes SignalR, just as #roman-kiss suggests in his answer, to push the events in near real time to the user.
Once you get a good grasp of the whole flow, you can start adjusting the code to match your use case. A good first step would be adjusting the deserialization logic to take advantage of more specific models. You can get the sample JSON models for SMS events here and convert them to C#.

How to set up HTTP outputs in Azure Functions

I've got a simple Azure function for which I set up a DocumentDB output (as an example):
I then added the outputDocument parameter to the function and assigned a value to it in the code (by the way I was surprised that when I set up the output that the runtime didn't automatically modify the function signature):
using System;
public static void Run(string input, out object outputDocument, TraceWriter log)
{
log.Info($"C# manually triggered function called with input: {input}");
outputDocument = new {
text = $"I'm running in a C# function! {input}"
};
}
When I run the function the Azure Functions runtime does it's binding magic and the DocumentDB document gets created.
I then set up an HTTP output:
and defined the res output parameter.
But now what? What's the process of assigning to res? I've of course got to define the destination, request type, parms, etc.
Howiecamp,
The HTTP output binding works in conjunction with the HTTP Trigger to act as the response handler for an HTTP request.
Currently, there isn't an output binding that would send the output payload over HTTP for you, so you'd need to make that HTTP request from your function code (e.g. using the HttpClient and issuing the request). You can see an example in one of our templates here: https://github.com/Azure/azure-webjobs-sdk-templates/blob/10650dbf9c4bad75b0c89b9c355edc53fe913cde/Templates/GitHubCommenter-CSharp/run.csx#L40-L49
I hope this helps!

How to remove #strin3http//schemas.microsoft.com/2003/10/Serialization/� received from service bus queue received in python script?

I have set the output of Azure stream analytics job to service bus queue which sends the data in JSON serialized format. When I receive the queue message in python script, along with the data in curly braces, I get #strin3http//schemas.microsoft.com/2003/10/Serialization/� appended in front. I am not able to trim it as the received message is not being recognized as either a string or a message. Because of this I cannot de-serialize the data.
This TechNet article suggests the following code:
// Get indices of actual message
var start = jsonString.IndexOf("{");
var end = jsonString.LastIndexOf("}") + 1;
var length = end - start;
// Get actual message
string cleandJsonString = jsonString.Substring(start, length);
Pretty primitive but whatever works I suppose...
The issue was similiar with the SO thread Interoperability Azure Service Bus Message Queue Messages.
Per my experience, the data from Azure Stream Analytics to Service Bus was sent via AMQP protocol, but the protocol of receiving the data in Python is HTTP. The excess content was generated by AMQP during transmission.
Assumption that receiving the message via the code below, please see https://azure.microsoft.com/en-us/documentation/articles/service-bus-python-how-to-use-queues/#receive-messages-from-a-queue. The function receive_queue_message with the False value of the argument peek_lock wrapped the REST API Receive and Delete Message (Destructive Read).
msg = bus_service.receive_queue_message('taskqueue', peek_lock=False)
print(msg.body)
According to the source code of Azure Service Bus SDK for Python include the functions receive_queue_message, read_delete_queue_message and _create_message, I think you can directly remove the excess content from the msg.body using the string common function lstrip or strip.
I ran into this issue as well. The previous answers are only workarounds and do not fix the root cause of this issue. The problem you are encountering is likely due to your Stream Analytics compatibility level. Compatibility level 1.0 uses an XML serializer producing the XML tag you are seeing. Compatibility level 1.1 "fixes" this issue.
See my previous answer here: https://stackoverflow.com/a/49307178/263139.
I had the same issue but in a .net solution. I was writing a service which sends data to a queue, and on the other hand, I was writing a service which gets that data from the queue. I've tried to send a JSON, like this:
var documentMessage = new DocumentMessage();
var json = JsonConvert.SerializeObject(documentMessage);
BrokeredMessage message = new BrokeredMessage(json);
await _client.SendAsync(message);
In this second service I was getting the JSON but with this prefix:
#strin3http//schemas.microsoft.com/2003/10/Serialization/�
I solved this problem by add DataContractJsonSerializer like that:
var documentMessage = new DocumentMessage();
var serializer = new DataContractJsonSerializer(typeof(DocumentMessage));
BrokeredMessage message = new BrokeredMessage(documentMessage , serializer);
await _client.SendAsync(message);
If you want to solve the problem in that way, you will have to add Data Attributes from System.Runtime.Serialization to the model:
[DataContract]
public class DocumentMessage
{
[DataMember]
public string Property1 { get; private set; }
[DataMember]
public string Property2 { get; private set; }
}
When using Microsoft.ServiceBus nuget package, replace
message.GetBody<Stream>();
with
message.GetBody<string>();

Azure Service Bus Queue sending a message in NodeJs to .NET client

I am sending a message from a C# worker to the Queue, then I am getting it on another C# worker and call
string body = message.GetBody<string>();
This works and I later de-serialize the string/JSON message.
Now I am trying to send the same message from NodeJS in form of a JSON message. When I try to receive it and
string body = message.GetBody<string>();
call this I get an exception saying the input is in incorrect format.
My message object on NodeJS looks like this
{
body: JSON.stringify(message)
}
Any ideas?
Got it fixed!
By default the .NET Azure Queue library uses a DataContractSerializer and a binary XmlDictionaryWriter to serialize the string message when using
new BrokeredMessage("my message");
So instead you need to use this
new BrokeredMessage(new MemoryStream(Encoding.UTF8.GetBytes("my message")), true);
and to read the message in C# you need to use
string body = new StreamReader(message.GetBody<Stream>(), Encoding.UTF8).ReadToEnd();
I also stopped wrapping my JSON.stringify message in an object and pass it directly to the sendQueueMessage. The send message code looks like this now:
serviceBusService.sendQueueMessage('my_queue', JSON.stringify("my message"), function(error){});
JSON.stringify outputs a UTF8 string so it is fully compatible with the C# code.

FormatExcpetion from Azure Webjobs Host

Understand this is pre-release :)
When trying to use QueueInput in Azure WebJobs and sticking the hex string of a hash in the message.
public System.Guid GetOwner(CloudQueueMessage msg)
Looking at ilspy seems like it is trying to parse out $AzureJobsParentId and the JSON parser is throwing the exception I can get around it by encoding my hash in a JSON snippet but I'd prefer not to. Is this a known bug?
[QueueInput] will normally use JSON.Net to deserialize the queue message payload to the parameter type. So if the queue message is not JSON, you'll get a exception (which should then be wrapped in something more friendly).
You can also work around it by using a string parameter with [QueueInput], like:
public static void Function([QueueInput] string testqueue)
{
}
For string parameter, the SDK will give you the QueueMessage.AsString directly, without any JSON serialization.
FYI, $AzureJobsParentId is a special field placed on json payloads that identifies which function instance enqueued a message. This gets used when you enqueue a message with [QueueOutput]. You can then view that relationship in the SDK dashboard (http://blogs.msdn.com/b/jmstall/archive/2014/01/27/getting-a-dashboard-for-local-development-with-the-webjobs-sdk.aspx)

Categories