Kafka Avro Value deserialization issue - c#

I send data using the AvroSpecific or AvroGeneric method and it works fine. The problem occurs when I try to consume the data sent by the above methods.
I use version libraries:
Confluent.Kafka v1.4.0
Confluent.SchemaRegistry v1.4.0
Confluent.SchemaRegistry.Serdes v1.3.0
Here is my Consumer code:
var schemaRegistryConfig = new SchemaRegistryConfig
{
Url = schemaRegistryUrl
};
using (var schemaRegistry = new CachedSchemaRegistryClient(schemaRegistryConfig))
using (var consumer =
new ConsumerBuilder<Ignore, GenericRecord>(consumerConfig)
.SetValueDeserializer(new AvroDeserializer<GenericRecord>(schemaRegistry).AsSyncOverAsync())
.Build())
{
consumer.Subscribe(topicName);
try
{
while (true)
{
try
{
var consumeResult = consumer.Consume(token);
//do something
}
catch (ConsumeException ex)
{
logger.LogError($"Error occured: {ex.Error.Reason}", ex);
}
}
}
catch (OperationCanceledException)
{
consumer.Close();
}
}

Related

UdpClient wont connect to IpAdress.Any

I'm trying to listen for UDP packets from unknown source. But can't bind on the "unspecified adress" (0.0.0.0 or ::)
I've already tried listening on ::1. But from what i tested that only works for local connections that don't pass the network interface.
public async void AwaitDiscoveryReply()
{
try
{
using (var client = new UdpClient(AddressFamily.InterNetworkV6))
{
client.Connect(IPAddress.IPv6Any,4568);
var result = await client.ReceiveAsync();
Debug.WriteLine("Received DR");
var stateProtocol = StateProtocol.FromBytes(result.Buffer);
var robeatsDevice = new RobeatsDevice
{
Id = stateProtocol.DeviceId,
Name = stateProtocol.DeviceName,
EndPoint = client.Client.RemoteEndPoint,
StateProtocol = stateProtocol
};
OnDiscoveryReply(new DeviceDiscoveryEventArgs {RobeatsDevice = robeatsDevice});
}
}
catch (Exception ex)
{
Debug.WriteLine(ex);
}
}
This keeps throwing the exception: The requested address is not valid in its context [::]:4568
UDP sockets are connectionless. "Connect" methods on UDP socket implementations, by convention (don't ask me why) establish default endpoints / filter traffic. If you want to receive traffic from any address, you don't need to "connect" at all. Use the constructor with signature UdpClient(Int32, AddressFamily) and delete the Connect() invocation:
public async void AwaitDiscoveryReply()
{
try
{
using (var client = new UdpClient(4568,AddressFamily.InterNetworkV6))
{
var result = await client.ReceiveAsync();
Debug.WriteLine("Received DR");
var stateProtocol = StateProtocol.FromBytes(result.Buffer);
var robeatsDevice = new RobeatsDevice
{
Id = stateProtocol.DeviceId,
Name = stateProtocol.DeviceName,
EndPoint = client.Client.RemoteEndPoint,
StateProtocol = stateProtocol
};
OnDiscoveryReply(new DeviceDiscoveryEventArgs {RobeatsDevice = robeatsDevice});
}
}
catch (Exception ex)
{
Debug.WriteLine(ex);
}
}

How can computation on an IotEdge module be triggered from within a .net core app?

I need to trigger some computation on an IotEdge module from an Administration-Backend Application.
On https://learn.microsoft.com/en-us/azure/iot-edge/module-development it says
Currently, a module cannot receive cloud-to-device messages
So it seems that calling direct methods seems to be the way to go. How can I implement a direct method and trigger it from within a .NET Core App?
In Main or Init Method of your IotEdge module you have to create a ModuleClient and connect it to a MethodHandler:
AmqpTransportSettings amqpSetting = new AmqpTransportSettings(TransportType.Amqp_Tcp_Only);
ITransportSettings[] settings = { amqpSetting };
ModuleClient ioTHubModuleClient = await ModuleClient.CreateFromEnvironmentAsync(settings);
await ioTHubModuleClient.OpenAsync();
await ioTHubModuleClient.SetMethodHandlerAsync("MyDirectMethodName", MyDirectMethodHandler, null);
Then you have to add the DirectMethodHandler to your IotEge module:
static async Task<MethodResponse> MyDirectMethodHandler(MethodRequest methodRequest, object userContext)
{
Console.WriteLine($"My direct method has been called!");
var payload = methodRequest.DataAsJson;
Console.WriteLine($"Payload: {payload}");
try
{
// perform your computation using the payload
}
catch (Exception e)
{
Console.WriteLine($"Computation failed! Error: {e.Message}");
return new MethodResponse(Encoding.UTF8.GetBytes("{\"errormessage\": \"" + e.Message + "\"}"), 500);
}
Console.WriteLine($"Computation successfull.");
return new MethodResponse(Encoding.UTF8.GetBytes("{\"status\": \"ok\"}"), 200);
}
From within your .Net core Application you can then trigger the direct method like this:
var iotHubConnectionString = "MyIotHubConnectionString";
var deviceId = "MyDeviceId";
var moduleId = "MyModuleId";
var methodName = "MyDirectMethodName";
var payload = "MyJsonPayloadString";
var cloudToDeviceMethod = new CloudToDeviceMethod(methodName, TimeSpan.FromSeconds(10));
cloudToDeviceMethod.SetPayloadJson(payload);
ServiceClient serviceClient = ServiceClient.CreateFromConnectionString(iotHubConnectionString);
try
{
var methodResult = await serviceClient.InvokeDeviceMethodAsync(deviceId, moduleId, cloudToDeviceMethod);
if(methodResult.Status == 200)
{
// Handle Success
}
else if (methodResult.Status == 500)
{
// Handle Failure
}
}
catch (Exception e)
{
// Device does not exist or is offline
Console.WriteLine(e.Message);
}

async processing in c#

I have a service that implements functionality to return data from sentiment analysis APIs. The client can request results from one or all engines and I want to collate all the data together. I want to process these async and wait for them all to complete before returning the result set. I'm new to async programming and I really cant figure out how to arrange the code and how to implement it syntactically. Here's an EXAMPLE of what I'm TRYING to achieve (I know this doesn't work, but you get the idea; hopefully :-) ):
private ISentimentResponse ProcessRequest(ISentimentRequest request, SentimentEngineServices selectedEngines)
{
SentimentResponse response = new SentimentResponse();
List<Task> taskList = new List<Task>();
foreach (SentimentEngineServices engineService in (SentimentEngineServices[])Enum.GetValues(typeof(SentimentEngineServices)))
{
if (((int)engineService & (int)selectedEngines) > 0)
{
ISentimentEngine engine = _engineFactory.GetSentimentEngine(engineService, null);
Task<ISentimentEngineResult> task = new Task<ISentimentEngineResult>(engine.AnalyseSentimentASync(request));
taskList.Add(task);
}
}
if (taskList.Count > 0)
{
ISentimentEngineResult[] results = Task<ISentimentEngineResult>.WaitAll(taskList);
foreach (result in results)
response.Add(results);
}
return response;
}
The engine has the following code implementation of engine.AnalyseSentimentASync:
public ISentimentEngineResult AnalyseSentiment(ISentimentRequest request)
{
try
{
MultiLanguageBatchInput sentimentList = SentimentRequestToMicrosoftBatchInput(request, Properties.Settings.Default.DefaultLanguage);
SentimentBatchResult sentiment = _client.Sentiment(sentimentList);
KeyPhraseBatchResult keyPhrases = _client.KeyPhrases(sentimentList);
return MicrosoftBatchResultsToSentimentEngineResult(sentiment, keyPhrases);
}
catch (Exception ex)
{
_logger.LogMessage(ex,$"{EngineName} threw an unknown exception: ", LoggingLevel.Error);
throw;
}
}
public async Task<ISentimentEngineResult> AnalyseSentimentAsync(ISentimentRequest request)
{
return AnalyseSentiment(request);
}
What do I need to do and is there any better way to achieve this?
I've looked everywhere for an example but I cant find one that addresses my implementation requirements, or the whole approach is completely wrong!
Thanks all,
Stu.
This is how you can convert it to async:
public async Task<ISentimentEngineResult> AnalyseSentimentAsync(ISentimentRequest request)
{
try
{
MultiLanguageBatchInput sentimentList = SentimentRequestToMicrosoftBatchInput(request, Properties.Settings.Default.DefaultLanguage);
SentimentBatchResult sentiment = await _client.SentimentAsync(sentimentList);
KeyPhraseBatchResult keyPhrases = await _client.KeyPhrasesAsync(sentimentList);
return MicrosoftBatchResultsToSentimentEngineResult(sentiment, keyPhrases);
}
catch (Exception ex)
{
_logger.LogMessage(ex,$"{EngineName} threw an unknown exception: ", LoggingLevel.Error);
throw;
}
}
private async Task<ISentimentResponse> ProcessRequestAsync(ISentimentRequest request, SentimentEngineServices selectedEngines)
{
SentimentResponse response = new SentimentResponse();
List<Task<ISentimentEngineResult>> taskList = new List<Task<ISentimentEngineResult>>();
foreach (SentimentEngineServices engineService in (SentimentEngineServices[])Enum.GetValues(typeof(SentimentEngineServices)))
{
if (((int)engineService & (int)selectedEngines) > 0)
{
ISentimentEngine engine = _engineFactory.GetSentimentEngine(engineService, null);
Task<ISentimentEngineResult> task = engine.AnalyseSentimentASync(request);
taskList.Add(task);
}
}
if (taskList.Count > 0)
{
ISentimentEngineResult[] results = await Task.WhenAll(taskList);
foreach (result in results)
response.Add(results);
}
return response;
}
Remember that you have to call it from some kind of event handler. I don't know what framework you are using (wpf, asp.net, windows service, webapi).
Ok, so here it is:
public ISentimentResponse AnalyseSentiment(ISentimentRequest request, SentimentEngineServices selectedEngines)
{
if (selectedEngines == SentimentEngineServices.None) throw new ArgumentException(nameof(selectedEngines));
ValidateRequest(request);
return ProcessRequestAsync(request, selectedEngines).Result;
}
private async Task<ISentimentResponse> ProcessRequestAsync(ISentimentRequest request, SentimentEngineServices selectedEngines)
{
SentimentResponse response = new SentimentResponse();
List<Task<ISentimentEngineResult>> taskList = new List<Task<ISentimentEngineResult>>();
foreach (SentimentEngineServices engineService in (SentimentEngineServices[])Enum.GetValues(typeof(SentimentEngineServices)))
{
if (((int)engineService & (int)selectedEngines) > 0)
{
ISentimentEngine engine = _engineFactory.GetSentimentEngine(engineService, null);
Task<ISentimentEngineResult> task = engine.AnalyseSentimentASync(request);
taskList.Add(task);
}
}
if (taskList.Count > 0)
{
ISentimentEngineResult[] results = await Task.WhenAll(taskList);
foreach (var result in results)
response.Add(result);
}
return response;
}
And the sentiment interface implementation:
public async Task<ISentimentEngineResult> AnalyseSentiment(ISentimentRequest request)
{
try
{
MultiLanguageBatchInput sentimentList = SentimentRequestToMicrosoftBatchInput(request, Properties.Settings.Default.DefaultLanguage);
SentimentBatchResult sentiment = await _client.SentimentAsync(sentimentList);
KeyPhraseBatchResult keyPhrases = await _client.KeyPhrasesAsync(sentimentList);
return MicrosoftBatchResultsToSentimentEngineResult(sentiment, keyPhrases);
}
catch (Exception ex)
{
_logger.LogMessage(ex,$"{EngineName} threw an unknown exception: ", LoggingLevel.Error);
throw;
}
}
Thanks FCin.

Get a black/no image with asp.net web api

I'am posting an image from a console application to a asp.net web api. I'am getting a file in the folder but the image is black (no image). Do I have something wrong in my code?
public class UploadController : ApiController
{
[System.Web.Mvc.HttpPost]
public string Upload()
{
var request = HttpContext.Current.Request;
var filePath = Path.Combine(HttpContext.Current.Server.MapPath("~/Uploads/"), request.Headers["filename"]);
try
{
using (var fs = new System.IO.FileStream(filePath, System.IO.FileMode.Create))
{
request.InputStream.CopyTo(fs);
}
}
catch (Exception e)
{
return e.Message;
}
return "uploaded";
}
}
Edit
My console app
http://pastebin.com/VsnDMYpb
try this. This works for me. I used this for multiple file upload
var httpRequest = HttpContext.Current.Request;
foreach (string file in httpRequest.Files)
{
var postedFile = httpRequest.Files[file];
var filePath = HttpContext.Current.Server.MapPath("~/Uploads/" + postedFile.FileName);
postedFile.SaveAs(filePath);
}
Use Request.Content.ReadAsMultipartAsync
public Task<IQueryable<HDFile>> Post()
{
try
{
var uploadFolderPath = HostingEnvironment.MapPath("~/App_Data/" + UploadFolder);
log.Debug(uploadFolderPath);
if (Request.Content.IsMimeMultipartContent())
{
var streamProvider = new WithExtensionMultipartFormDataStreamProvider(uploadFolderPath);
var task = Request.Content.ReadAsMultipartAsync(streamProvider).ContinueWith<IQueryable<HDFile>>(t =>
{
if (t.IsFaulted || t.IsCanceled)
{
throw new HttpResponseException(HttpStatusCode.InternalServerError);
}
var fileInfo = streamProvider.FileData.Select(i =>
{
var info = new FileInfo(i.LocalFileName);
return new HDFile(info.Name, Request.RequestUri.AbsoluteUri + "?filename=" + info.Name, (info.Length / 1024).ToString());
});
return fileInfo.AsQueryable();
});
return task;
}
else
{
throw new HttpResponseException(Request.CreateResponse(HttpStatusCode.NotAcceptable, "This request is not properly formatted"));
}
}
catch (Exception ex)
{
log.Error(ex);
throw new HttpResponseException(Request.CreateResponse(HttpStatusCode.BadRequest, ex.Message));
}
}
The code I get from this post

EventStore duplicate commit exception

UPDATE:
We are getting getting System.Data.SqlClient.SqlException. The message is:
Violation of PRIMARY KEY constraint 'PK_Commits'. Cannot insert duplicate key in object 'dbo.Commits'.\r\nThe statement has been terminated
It seems like EventStore is using streamid and commitid as unique id.
We use event store to append events as below.
public bool TryAppend(object[] content)
{
if (content == null)
throw new ArgumentNullException("content");
try
{
using (var stream = m_storage.OpenStream(m_streamID, 0, int.MaxValue))
{
var versionInStore = stream.StreamRevision;
content.ToList().ForEach(m =>
{
var version = ++versionInStore;
var key = string.Format("{0}-{1:00000000}", m.GetType().Name, version);
var savedMessage = new SavedRecord(key, version, m);
stream.Add(new EventMessage { Body = savedMessage });
});
stream.CommitChanges(Guid.NewGuid());
}
return true;
}
catch (Exception e)
{
m_logger.LogError(e);
return false;
}
}
The configuration of EventStore is as below. We are using Sql Serer 2008 as persistance store.
return Wireup.Init()
.LogToOutputWindow()
.UsingSqlPersistence(m_connectionName)
.WithDialect(new MsSqlDialect())
.EnlistInAmbientTransaction() // two-phase commit
.InitializeStorageEngine()
.UsingJsonSerialization()
.Compress()
.UsingSynchronousDispatchScheduler()
.DispatchTo(new DelegateMessageDispatcher(DispatchCommit))
.Build();
Any ideas why are gettin the dupplicate commit exception?
Thanks
Have got the same issue; in my case it was probably because of different threads was adding different events to the stream with same id at the same time.
Have writtent the following code to be able to retry adding events:
private void TryAddEvent(IStoreEvents storeEvents, IUserEvent anEvent, Guid streamId)
{
var isCommitSuccessful = false;
for (var i = 0; i < 10 && !isCommitSuccessful; i++)
{
try
{
using (var stream = storeEvents.OpenStream(streamId, 0, int.MaxValue))
{
stream.Add(new EventMessage {Body = anEvent});
if (stream.UncommittedEvents.All(e => e.Body != anEvent))
{
stream.Add(new EventMessage {Body = anEvent});
}
stream.CommitChanges(Guid.NewGuid());
}
isCommitSuccessful = true;
}
catch (Exception ex)
{
if (!(ex is SqlException) && !(ex is ConcurrencyException))
{
throw;
}
using (var stream = storeEvents.OpenStream(streamId, 0, int.MaxValue))
{
if (stream.CommittedEvents.Any(e => e.Body == anEvent))
{
isCommitSuccessful = true;
}
}
}
}
if (!isCommitSuccessful)
{
throw new ConcurrencyException(String.Format("Cannot add {0} to event store", anEvent.GetType()));
}
}
Hope it would help.

Categories