We Keep Coding

How to serialize/deserialize from ksql avro format to c# using confluent platform

I am using KsqlDb a table with the following form:
KSQL-DB Query
create table currency (id integer,name varchar) with (kafka_topic='currency',partitions=1,value_format='avro');
C# model
public class Currency
{
public int Id{get;set;}
public string Name{get;set;}
}
Now i want to know how should i write/read data from this topic in C# using the Confluent library:
Writing
IProducer<int, Currency> producer=....
Currency cur=new Currency();
Message<int,Currency> message = new Message<int, Currency>
{
Key = msg.Id,
Timestamp = new Timestamp(DateTime.UtcNow, TimestampType.CreateTime),
Value = msg
};
DeliveryResult<int,Currency> delivery = await this.producer.ProduceAsync(topic,message);
Reading
IConsumer<int,Currency> iconsumer = new ConsumerBuilder<int, Currency>(config)
.SetKeyDeserializer(Deserializers.Int32) //i assume i need to use the id from my dto
.SetValueDeserializer(...) //what deserializer
.Build();
ConsumeResult<int,Currency> result = consumer.Consume();
Currency message = // what deserializer JsonSerializer.Deserialize<Currency>(result.Message.Value);
I am not sure how to go about this so i tried looking for serializer. I found this library AvroSerializer , but i do not get where the author fetches the schema.
Any help on how to read/write to a specific topic that would match with my ksqldb models ?
Update
After some research and some answers here i have started using the schemaRegistry
var config = new ConsumerConfig
{
GroupId = kafkaConfig.ConsumerGroup,
BootstrapServers = kafkaConfig.ServerUrl,
AutoOffsetReset = AutoOffsetReset.Earliest
};
var schemaRegistryConfig = new SchemaRegistryConfig
{
Url = kafkaConfig.SchemaRegistryUrl
};
var schemaRegistry = new CachedSchemaRegistryClient(schemaRegistryConfig);
IConsumer<int,Currency> consumer = new ConsumerBuilder<int, Currency>(config)
.SetKeyDeserializer(new AvroDeserializer<int>(schemaRegistry).AsSyncOverAsync())
.SetValueDeserializer(new AvroDeserializer<Currency>(schemaRegistry).AsSyncOverAsync())
.Build();
ConsumeResult<int, Currency> result = consumer.Consume();
Now i am getting another error:
Expecting data framing of length 5 bytes or more but total data size
is 4 bytes
As someone kindly pointed out it seems i retrieving only the id from the schema registry.
How can i just : insert into currency (id,name) values (1,3) and retrieve it in C# as a POCO (listed above) ?
Update 2
After i have found this source program it seems i am not able to publish messages to tables for some reason.
There is no error when sending the message but it is not published to Kafka.

I found this library AvroSerializer , but i do not get where the author fetches the schema.
Unclear why you need to use a library other than the Confluent one, but they get it from the Schema Registry. You can use CachedSchemaRegistryClient to get the schema string easily, however you shouldn't need this in the code as the deserializer will download from the registry on its own.
If you refer to the examples/ in the confluent-kafka-dotnet repo for Specific Avro consumption, you can see they generate the User class from User.avsc file, which seems to be exactly what you want to do here for Currency rather than write it yourself

I have solved the problem by defining my custom serializer , thus implementing the ISerializer<T> and IDeserializer<T> interfaces which in their belly are just wrappers over System.Text.Json.JsonSerializer or NewtonsoftJson.
Serializer
public class MySerializer:ISerializer<T>
{
byte[] Serialize(T data, SerializationContext context)
{
var str=System.Text.Json.JsonSerializer.Serialize(data); //you can also use Newtonsoft here
var bytes=Encoding.UTF8.GetBytes(str);
return bytes;
}
}
Usage
var config = new ConsumerConfig
{
GroupId = kafkaConfig.ConsumerGroup,
BootstrapServers = kafkaConfig.ServerUrl,
AutoOffsetReset = AutoOffsetReset.Earliest
};
IConsumer<int,Currency> consumer = new ConsumerBuilder<int, Currency>(config)
.SetValueDeserializer(new MySerializer<Currency>())
.Build();
ConsumeResult<int, Currency> result = consumer.Consume();
P.S
I am not even using the schema registry here afteri implemented the interface

Retrieve all contents of Zoho module via REST API c#

I am trying to get the full contents of my modules From Zoho to our local Server. The deluge code does work as it returns to me the data which is being sent via the API. However, once it reaches the API, it is null. Any idea?
Below is the deluge code:
// Create a map that holds the values of the new contact that needs to be created
evaluation_info = Map();
evaluation_info.put("BulkData",zoho.crm.getRecords("Publishers"));
data = Map();
data.put(evaluation_info);
response = invokeurl
[
url :"https://zohoapi.xxxxx.com/publisher/publish"
type :POST
parameters:data
connection:"zohowebapi"
];
info data; (data returns all the data from publishers)
Here is my ASP.NET core restful API. It does ping it and create the file but the content of the file is null.
Route("[controller]")]
[ApiController]
public class PublisherController : ControllerBase
{
[HttpGet("[action]"), HttpPost("[action]")]
public void Publish(string data)
{
(it's already null when it comes here. why?)
string JSONresult = JsonConvert.SerializeObject(data);
string path = #"C:\storage\journalytics_evaluationsv2.json";
using (var file = new StreamWriter(path, true))
{
file.WriteLine(JSONresult.ToString());
file.Close();
}
}
}
}
What am I missing? Thank you

After contacting Zoho support, the solution he offered was to loop through the data in order to get all the contents from a module (if they are more than 200 records. With the solution provided, one doesn't really need the deluge code anymore as long as you have the ZOHO api set to your account in code. This was my final solution. This solution is not scalable at all. It's best to work with the BULK CSV.
// Our own ZohoAPI which lets us connect and authenticate etc. Yours may look slightly different
ZohoApi zohoApi = new ZohoApi();
zohoApi.Initialize();
ZCRMRestClient restClient = ZCRMRestClient.GetInstance();
var allMedicalJournals = new List<ZCRMRecord>();
for (int i = 1; i <= 30; i++)
{
List<ZCRMRecord> accountAccessRecords2 =
restClient.GetModuleInstance("Journals").SearchByCriteria("Tag:equals:MedicalSet", i, 200).BulkData.ToList();
foreach (var newData in accountAccessRecords2)
allMedicalJournals.Add(newData);
}

How to call google.apis.dialogflow.v2 in C#

I am new to Google APIs. I want to know how to call Google Dialogflow API in C# to get intent form the input text. But I can't find any example to call Dialogflow using C#.
Please provide some example to call Dialogflow from C#.

If I understand your question correctly you want to call the DialogFlow API from within a C# application (rather than writing fulfillment endpoint(s) that are called from DialogFlow. If that's the case here's a sample for making that call:
using Google.Cloud.Dialogflow.V2;
...
...
var query = new QueryInput
{
Text = new TextInput
{
Text = "Something you want to ask a DF agent",
LanguageCode = "en-us"
}
};
var sessionId = "SomeUniqueId";
var agent = "MyAgentName";
var creds = GoogleCredential.FromJson("{ json google credentials file)");
var channel = new Grpc.Core.Channel(SessionsClient.DefaultEndpoint.Host,
creds.ToChannelCredentials());
var client = SessionsClient.Create(channel);
var dialogFlow = client.DetectIntent(
new SessionName(agent, sessionId),
query
);
channel.ShutdownAsync();
In an earlier version of the DialogFlowAPI I was running into file locking issues when trying to re-deploy a web api project which the channel.ShutDownAsync() seemed to solve. I think this has been fixed in a recent release.
This is the simplest version of a DF request I've used. There is a more complicated version that passes in an input context in this post:
Making DialogFlow v2 DetectIntent Calls w/ C# (including input context)

(Nitpicking: I assume you know DialogFlow will call your code as specified/registered in the action at DialogFlow? So your code can only respond to DialogFlow, and not call it.)
Short answer/redirect:
Don't use Google.Apis.Dialogflow.v2 (with GoogleCloudDialogflowV2WebhookRequest and GoogleCloudDialogflowV2WebhookResponse) but use Google.Cloud.Dialogflow.v2 (with WebhookRequest and WebhookResponse) - see this eTag-error. I will also mention some other alternatives underneath.
Google.Cloud.Dialogflow.v2
Using Google.Cloud.Dialogflow.v2 NuGet (Edit: FWIW: this code was written for the beta-preview):
[HttpPost]
public dynamic PostWithCloudResponse([FromBody] WebhookRequest dialogflowRequest)
{
var intentName = dialogflowRequest.QueryResult.Intent.DisplayName;
var actualQuestion = dialogflowRequest.QueryResult.QueryText;
var testAnswer = $"Dialogflow Request for intent '{intentName}' and question '{actualQuestion}'";
var dialogflowResponse = new WebhookResponse
{
FulfillmentText = testAnswer,
FulfillmentMessages =
{ new Intent.Types.Message
{ SimpleResponses = new Intent.Types.Message.Types.SimpleResponses
{ SimpleResponses_ =
{ new Intent.Types.Message.Types.SimpleResponse
{
DisplayText = testAnswer,
TextToSpeech = testAnswer,
//Ssml = $"<speak>{testAnswer}</speak>"
}
}
}
}
}
};
var jsonResponse = dialogflowResponse.ToString();
return new ContentResult { Content = jsonResponse, ContentType = "application/json" }; ;
}
Edit: It turns out that the model binding may not bind all properties from the 'ProtoBuf-json' correctly (e.g. WebhookRequest.outputContexts[N].parameters),
so one should probably use the Google.Protobuf.JsonParser (e.g. see this documentation).
This parser may trip over unknown fields, so one probably also wants to ignore that. So now I use this code (I may one day make the generic method more generic and thus useful, by making HttpContext.Request.InputStream a parameter):
public ActionResult PostWithCloudResponse()
{
var dialogflowRequest = ParseProtobufRequest<WebhookRequest>();
...
var jsonResponse = dialogflowResponse.ToString();
return new ContentResult { Content = jsonResponse, ContentType = "application/json" }; ;
}
private T ParseProtobufRequest<T>() where T : Google.Protobuf.IMessage, new()
{
// parse ProtoBuf (not 'normal' json) with unknown fields, else it may not bind ProtoBuf correctly
// https://github.com/googleapis/google-cloud-dotnet/issues/2425 "ask the Protobuf code to parse the result"
string requestBody;
using (var reader = new StreamReader(HttpContext.Request.InputStream))
{
requestBody = reader.ReadToEnd();
}
var parser = new Google.Protobuf.JsonParser(JsonParser.Settings.Default.WithIgnoreUnknownFields(true));
var typedRequest = parser.Parse<T>(requestBody);
return typedRequest;
}
BTW: This 'ProtoBuf-json' is also the reason to use WebhookResponse.ToString() which in turn uses Google.Protobuf.JsonFormatter.ToDiagnosticString.
Microsoft's BotBuilder
Microsoft's BotBuilder packages and Visual Studio template.
I havent't used it yet, but expect approximately the same code?
Hand written proprietary code
A simple example of incoming request code (called an NLU-Response by Google) is provided by Madoka Chiyoda (Chomado) at Github. The incoming call is simply parsed to her DialogFlowResponseModel:
public static async Task<HttpResponseMessage> Run([...]HttpRequestMessage req, [...]CloudBlockBlob mp3Out, TraceWriter log)
...
var data = await req.Content.ReadAsAsync<Models.DialogFlowResponseModel>();
Gactions
If you plan to work without DialogFlow later on, please note that the interface for Gactions differs significantly from the interface with DialogFlow.
The json-parameters and return-values have some overlap, but nothing gaining you any programming time (probably loosing some time by starting 'over').
However, starting with DialogFlow may gain you some quick dialog-experience (e.g. question & answer design/prototyping).
And the DialogFlow-API does have a NuGet package, where the Gactions-interface does not have a NuGet-package just yet.

C# Engine Object replacement for web service call

I am currently working on an MVC site that will hopefully consume an old .asmx Web Service. The documentation for the Web Service provides the following example:
// Construct a request object
TrimRequest request = new TrimRequest();
// Construct a RecordStringSearchClause, with type
// TitleWord, and argument "reef"
RecordStringSearchClause clause = new RecordStringSearchClause();
clause.Type = RecordStringSearchClauseType.TitleWord;
clause.Arg = "reef";
// Construct a record search, and put our search clause in it
WorkerPortalTest.TRIMWS.RecordSearch search = new WorkerPortalTest.TRIMWS.RecordSearch();
search.Items = new RecordClause[] { clause };
// If we had more than one clause, it would look like:
// search.Items = new RecordClause[] { clause1, clause2, clause3 }
// Put our search operation into our TrimRequest
request.Items = new Operation[] { search };
// Send it off. Whatever comes back will be in response
Engine engine = new Engine();
engine.Credentials = newSystem.Net.NetworkCredential(username, password);
TrimResponse response = engine.Execute(request);
As a fairly new C# programmer I understand all of it apart from the last three lines. I have never seen or used the Engine object and Visual Studio does not know of it either.. I looked through MSDN and found this page but it said it was deprecated.
I am just looking for some pointers in the right direction to call the Web Service and receive back the desired result.
Thanks.

Engine is an example of the webservice class. It calls a method in Engine class known as "Execute"..

Amazon Product Advertising API for Asp.net & C#

I want to fetch books using Amazon Product Advertising API with asp.net and C#. All the guides and codes are so confusing as to they don't give you a single method to search the books.
Is there any single stub that can be used to call the service and fetch the books based on the ISBN.
thanks

There's a good sample solution you can download.
http://aws.amazon.com/code/2480?_encoding=UTF8&queryArg=searchQuery&x=0&fromSearch=1&y=0&searchPath=code&searchQuery=Advertising
They give you a class called SignedRequestHelper, then you make a call like this:
public static void Main()
{
SignedRequestHelper helper = new SignedRequestHelper(MY_AWS_ACCESS_KEY_ID, MY_AWS_SECRET_KEY, DESTINATION);
/*
* The helper supports two forms of requests - dictionary form and query string form.
*/
String requestUrl;
String title;
/*
* Here is an ItemLookup example where the request is stored as a dictionary.
*/
IDictionary<string, string> r1 = new Dictionary<string, String>();
r1["Service"] = "AWSECommerceService";
r1["Version"] = "2009-03-31";
r1["Operation"] = "ItemLookup";
r1["ItemId"] = ITEM_ID;
r1["ResponseGroup"] = "Small";
/* Random params for testing */
r1["AnUrl"] = "http://www.amazon.com/books";
r1["AnEmailAddress"] = "foobar#nowhere.com";
r1["AUnicodeString"] = "αβγδεٵٶٷٸٹٺチャーハン叉焼";
r1["Latin1Chars"] = "ĀāĂ㥹ĆćĈĉĊċČčĎďĐđĒēĔĕĖėĘęĚěĜĝĞğĠġĢģĤĥĦħĨĩĪīĬĭĮįİıIJij";
requestUrl = helper.Sign(r1);
title = FetchTitle(requestUrl);
System.Console.WriteLine("Method 1: ItemLookup Dictionary form.");
System.Console.WriteLine("Title is \"" + title + "\"");
System.Console.WriteLine();
}
You need to use the ItemLookup (like the example) but set the IdType to ISBN. Then set the ItemId to the actual ISBN. Here are the details on ItemLookup:
docs.amazonwebservices.com/AWSECommerceService/latest/DG/index.html?ItemLookup.html

I get this when I use that sample. looks like there has been a change in the API recently.
System.InvalidOperationException: There is an error in the XML document. ---> Sy
stem.InvalidOperationException: <ItemLookupResponse xmlns='http://webservices.am
azon.com/AWSECommerceService/2011-08-01'> was not expected.

To fetch books install this library (Install-Package Nager.AmazonProductAdvertising)
https://www.nuget.org/packages/Nager.AmazonProductAdvertising/
Example:
var authentication = new AmazonAuthentication("accesskey", "secretkey");
var client = new AmazonProductAdvertisingClient(authentication, AmazonEndpoint.UK);
var result = await client.GetItemsAsync("978-0261102385");