We Keep Coding

Null reference exception on c# grpc request

As the title described, I'm having a problem with this proto message:
(Proto file)
message StreamingRecognizeRequest{
oneof streaming_request{
StreamingRecognitionConfig streaming_config = 1;
bytes audio_content = 2;
}
}
Here is how I called it:
(C# main function)
await voice.RequestStream.WriteAsync(new StreamingRecognizeRequest
{
StreamingConfig =
{
SingleUtterance = true,
InterimResults = true,
Config =
{
Encoding = RecognitionConfig.Types.AudioEncoding.Linear16,
MaxAlternatives = 1,
SampleRateHertz = 8000,
Enhanced = true,
SpeechContexts =
{
}
}
}
});
But the debugger keep marking my Stream Recognize Reuqest as null reference exception, the same pattern works fine in NodeJS but I only have this error in C#. Note that the first message send to the gRPC server doesn't require "Audio Content". So what is the root cause of this error and how to solve it? Or this is gRPC bug on c#?
Thank you!

It's pretty funny how I fixed it. Just not putting them inside each other but seperately create new object:
RecognitionConfig regcfg = new RecognitionConfig
{
Encoding = RecognitionConfig.Types.AudioEncoding.Linear16,
MaxAlternatives = 1,
SampleRateHertz = 8000,
Enhanced = true,
};
StreamingRecognitionConfig stc = new StreamingRecognitionConfig
{
Config = regcfg,
SingleUtterance = true,
InterimResults = true,
};
//Create new gRPC stream listener
var voice = vaisclient.StreamingRecognize(meta);
await voice.RequestStream.WriteAsync(new StreamingRecognizeRequest
{
StreamingConfig = stc
}) ;

ZeroMQ Broadcast with Proxy losing messages

I am doing some performance tests on ZeroMQ in order to compare it with others like RabbitMQ and ActiveMQ.
In my broadcast tests and to avoid "The Dynamic Discovery Problem" as referred by ZeroMQ documentation I have used a proxy. In my scenario, I am using 50 concurrent publishers each one sending 500 messages with 1ms delay between sends. Each message is then read by 50 subscribers. And as I said I am losing messages, each of the subscribers should receive a total of 25000 messages and they are each receiving between 5000 and 10000 messages only.
I am using Windows and C# .Net client clrzmq4 (4.1.0.31).
I have already tried some solutions that I found on other posts:
I have set linger to TimeSpan.MaxValue
I have set ReceiveHighWatermark to 0 (as it is presented as infinite, but I have tried also Int32.MaxValue)
I have set checked for slow start receivers, I made receivers start some seconds before publishers
I had to make sure that no garbage collection is made to the socket instances (linger should do it but to make sure)
I have a similar scenario (with similar logic) using NetMQ and it works fine. The other scenario does not use security though and this one does (and that's also the reason why I use clrzmq in this one because I need client authentication with certificates that is not yet possible on NetMQ).
EDIT:
public class MCVEPublisher
{
public void publish(int numberOfMessages)
{
string topic = "TopicA";
ZContext ZContext = ZContext.Create();
ZSocket publisher = new ZSocket(ZContext, ZSocketType.PUB);
//Security
// Create or load certificates
ZCert serverCert = Main.GetOrCreateCert("publisher");
var actor = new ZActor(ZContext, ZAuth.Action, null);
actor.Start();
// send CURVE settings to ZAuth
actor.Frontend.Send(new ZFrame("VERBOSE"));
actor.Frontend.Send(new ZMessage(new List<ZFrame>()
{ new ZFrame("ALLOW"), new ZFrame("127.0.0.1") }));
actor.Frontend.Send(new ZMessage(new List<ZFrame>()
{ new ZFrame("CURVE"), new ZFrame(".curve") }));
publisher.CurvePublicKey = serverCert.PublicKey;
publisher.CurveSecretKey = serverCert.SecretKey;
publisher.CurveServer = true;
publisher.Linger = TimeSpan.MaxValue;
publisher.ReceiveHighWatermark = Int32.MaxValue;
publisher.Connect("tcp://127.0.0.1:5678");
Thread.Sleep(3500);
for (int i = 0; i < numberOfMessages; i++)
{
Thread.Sleep(1);
var update = $"{topic} {"message"}";
using (var updateFrame = new ZFrame(update))
{
publisher.Send(updateFrame);
}
}
//just to make sure it does not end instantly
Thread.Sleep(60000);
//just to make sure publisher is not garbage collected
ulong Affinity = publisher.Affinity;
}
}
public class MCVESubscriber
{
private ZSocket subscriber;
private List<string> prints = new List<string>();
public void read()
{
string topic = "TopicA";
var context = new ZContext();
subscriber = new ZSocket(context, ZSocketType.SUB);
//Security
ZCert serverCert = Main.GetOrCreateCert("xpub");
ZCert clientCert = Main.GetOrCreateCert("subscriber");
subscriber.CurvePublicKey = clientCert.PublicKey;
subscriber.CurveSecretKey = clientCert.SecretKey;
subscriber.CurveServer = true;
subscriber.CurveServerKey = serverCert.PublicKey;
subscriber.Linger = TimeSpan.MaxValue;
subscriber.ReceiveHighWatermark = Int32.MaxValue;
// Connect
subscriber.Connect("tcp://127.0.0.1:1234");
subscriber.Subscribe(topic);
while (true)
{
using (var replyFrame = subscriber.ReceiveFrame())
{
string messageReceived = replyFrame.ReadString();
messageReceived = Convert.ToString(messageReceived.Split(' ')[1]);
prints.Add(messageReceived);
}
}
}
public void PrintMessages()
{
Console.WriteLine("printing " + prints.Count);
}
}
public class Main
{
static void Main(string[] args)
{
broadcast(500, 50, 50, 30000);
}
public static void broadcast(int numberOfMessages, int numberOfPublishers, int numberOfSubscribers, int timeOfRun)
{
new Thread(() =>
{
using (var context = new ZContext())
using (var xsubSocket = new ZSocket(context, ZSocketType.XSUB))
using (var xpubSocket = new ZSocket(context, ZSocketType.XPUB))
{
//Security
ZCert serverCert = GetOrCreateCert("publisher");
ZCert clientCert = GetOrCreateCert("xsub");
xsubSocket.CurvePublicKey = clientCert.PublicKey;
xsubSocket.CurveSecretKey = clientCert.SecretKey;
xsubSocket.CurveServer = true;
xsubSocket.CurveServerKey = serverCert.PublicKey;
xsubSocket.Linger = TimeSpan.MaxValue;
xsubSocket.ReceiveHighWatermark = Int32.MaxValue;
xsubSocket.Bind("tcp://*:5678");
//Security
serverCert = GetOrCreateCert("xpub");
var actor = new ZActor(ZAuth.Action0, null);
actor.Start();
// send CURVE settings to ZAuth
actor.Frontend.Send(new ZFrame("VERBOSE"));
actor.Frontend.Send(new ZMessage(new List<ZFrame>()
{ new ZFrame("ALLOW"), new ZFrame("127.0.0.1") }));
actor.Frontend.Send(new ZMessage(new List<ZFrame>()
{ new ZFrame("CURVE"), new ZFrame(".curve") }));
xpubSocket.CurvePublicKey = serverCert.PublicKey;
xpubSocket.CurveSecretKey = serverCert.SecretKey;
xpubSocket.CurveServer = true;
xpubSocket.Linger = TimeSpan.MaxValue;
xpubSocket.ReceiveHighWatermark = Int32.MaxValue;
xpubSocket.Bind("tcp://*:1234");
using (var subscription = ZFrame.Create(1))
{
subscription.Write(new byte[] { 0x1 }, 0, 1);
xpubSocket.Send(subscription);
}
Console.WriteLine("Intermediary started, and waiting for messages");
// proxy messages between frontend / backend
ZContext.Proxy(xsubSocket, xpubSocket);
Console.WriteLine("end of proxy");
//just to make sure it does not end instantly
Thread.Sleep(60000);
//just to make sure xpubSocket and xsubSocket are not garbage collected
ulong Affinity = xpubSocket.Affinity;
int ReceiveHighWatermark = xsubSocket.ReceiveHighWatermark;
}
}).Start();
Thread.Sleep(5000); //to make sure proxy started
List<MCVESubscriber> Subscribers = new List<MCVESubscriber>();
for (int i = 0; i < numberOfSubscribers; i++)
{
MCVESubscriber ZeroMqSubscriber = new MCVESubscriber();
new Thread(() =>
{
ZeroMqSubscriber.read();
}).Start();
Subscribers.Add(ZeroMqSubscriber);
}
Thread.Sleep(10000);//to make sure all subscribers started
for (int i = 0; i < numberOfPublishers; i++)
{
MCVEPublisher ZeroMqPublisherBroadcast = new MCVEPublisher();
new Thread(() =>
{
ZeroMqPublisherBroadcast.publish(numberOfMessages);
}).Start();
}
Thread.Sleep(timeOfRun);
foreach (MCVESubscriber Subscriber in Subscribers)
{
Subscriber.PrintMessages();
}
}
public static ZCert GetOrCreateCert(string name, string curvpath = ".curve")
{
ZCert cert;
string keyfile = Path.Combine(curvpath, name + ".pub");
if (!File.Exists(keyfile))
{
cert = new ZCert();
Directory.CreateDirectory(curvpath);
cert.SetMeta("name", name);
cert.Save(keyfile);
}
else
{
cert = ZCert.Load(keyfile);
}
return cert;
}
}
This code also produces the expected number of messages when security is disabled, but when turned on it doesn't.
Does someone know another thing to check? Or has it happened to anyone before?
Thanks

OPC UA Foundation SDK: Server does not have an instance certificate assigned

I am using the Foundation SDK with C#, trying to launch a simple server in a minimal fashion.
Here is my attempt sofar.
public void StartServer()
{
var config = new ApplicationConfiguration
{
ApplicationName = "TestServer",
ApplicationType = ApplicationType.Client,
SecurityConfiguration = new SecurityConfiguration
{
ApplicationCertificate = new CertificateIdentifier
{
StoreType = #"Windows",
StorePath = #"CurrentUser\My",
SubjectName = Utils.Format(#"CN={0}, DC={1}", "TestServer", Dns.GetHostName())
},
TrustedPeerCertificates = new CertificateTrustList
{
StoreType = #"Windows",
StorePath = #"CurrentUser\TrustedPeople",
},
NonceLength = 32,
AutoAcceptUntrustedCertificates = true,
ConfigureFirewall = false
},
TransportConfigurations = new TransportConfigurationCollection(),
TransportQuotas = new TransportQuotas { OperationTimeout = 15000 },
ServerConfiguration = new ServerConfiguration
{
SecurityPolicies = new ServerSecurityPolicyCollection
{
new ServerSecurityPolicy
{
SecurityLevel = 0,
SecurityMode = MessageSecurityMode.None,
SecurityPolicyUri = "http://opcfoundation.org/UA/SecurityPolicy#None"
}
},
UserTokenPolicies = new UserTokenPolicyCollection
{
new UserTokenPolicy { TokenType = UserTokenType.Anonymous }
},
DiagnosticsEnabled = true,
MaxSessionCount = 100,
MinSessionTimeout = 5000,
MaxSessionTimeout = 10000,
MaxBrowseContinuationPoints = 10,
MaxQueryContinuationPoints = 10,
MaxHistoryContinuationPoints = 100,
MaxRequestAge = 600000,
MinPublishingInterval = 100,
MaxPublishingInterval = 3600000,
PublishingResolution = 50,
MaxSubscriptionLifetime = 3600000,
MaxMessageQueueSize = 10,
MaxNotificationQueueSize = 100,
MaxNotificationsPerPublish = 1000,
MinMetadataSamplingInterval = 1000
}
};
config.Validate(ApplicationType.Server);
var server = new MyCustomServer();
server.Start(config);
}
When I try to call the method, I get the following exception:
Opc.Ua.ServiceResultException: Server does not have an instance certificate assigned.
à Opc.Ua.ServerBase.OnServerStarting(ApplicationConfiguration configuration) dans ...\OPC Foundation\Stack\Core\Stack\Server\ServerBase.cs:ligne 1607
à Opc.Ua.Server.StandardServer.OnServerStarting(ApplicationConfiguration configuration) dans ...\OPC Foundation\SampleApplications\SDK\Server\Server\StandardServer.cs:ligne 2628
à Opc.Ua.ServerBase.Start(ApplicationConfiguration configuration) dans ...\OPC Foundation\Stack\Core\Stack\Server\ServerBase.cs:ligne 232
à SlimFixtures.ServerDriver.StartServer() dans ...\ServerDriver.cs:ligne 71
What am I doing wrong?

So you found that servers based on foundation code always need a certificate.
Creating a self-signed certificate is easy, and does not need Admin login if you are using the Current User/My Windows store.
You can call this extension method after you validate:
config.Validate(ApplicationType.Server);
config.EnsureApplicationCertificate();
elsewhere
public static class ServiceExtensions
{
/// <summary>
/// Ensures the application certificate is present and valid.
/// </summary>
public static void EnsureApplicationCertificate(this ApplicationConfiguration configuration)
{
const ushort keySize = 1024;
const ushort lifetimeInMonths = 300;
if (configuration == null)
{
throw new ArgumentNullException("configuration");
}
bool errorFlag = false;
string hostName = Dns.GetHostName();
var serverDomainNames = configuration.GetServerDomainNames();
var applicationCertificate = configuration.SecurityConfiguration.ApplicationCertificate;
var certificate = applicationCertificate.Find(true);
if (certificate != null)
{
// if cert found then check domains
var domainsFromCertficate = Utils.GetDomainsFromCertficate(certificate);
foreach (string serverDomainName in serverDomainNames)
{
if (Utils.FindStringIgnoreCase(domainsFromCertficate, serverDomainName))
{
continue;
}
if (String.Equals(serverDomainName, "localhost", StringComparison.OrdinalIgnoreCase))
{
if (Utils.FindStringIgnoreCase(domainsFromCertficate, hostName))
{
continue;
}
var hostEntry = Dns.GetHostEntry(hostName);
if (hostEntry.Aliases.Any(alias => Utils.FindStringIgnoreCase(domainsFromCertficate, alias)))
{
continue;
}
if (hostEntry.AddressList.Any(ipAddress => Utils.FindStringIgnoreCase(domainsFromCertficate, ipAddress.ToString())))
{
continue;
}
}
Trace.TraceInformation("The application is configured to use domain '{0}' which does not appear in the certificate.", serverDomainName);
errorFlag = true;
} // end for
// if no errors and keySizes match
if (!errorFlag && (keySize == certificate.PublicKey.Key.KeySize))
{
return; // cert okay
}
}
// if we get here then we'll create a new cert
if (certificate == null)
{
certificate = applicationCertificate.Find(false);
if (certificate != null)
{
Trace.TraceInformation("Matching certificate with SubjectName '{0}' found but without a private key.", applicationCertificate.SubjectName);
}
}
// lets check if there is any to delete
if (certificate != null)
{
using (var store2 = applicationCertificate.OpenStore())
{
store2.Delete(certificate.Thumbprint);
}
}
if (serverDomainNames.Count == 0)
{
serverDomainNames.Add(hostName);
}
CertificateFactory.CreateCertificate(applicationCertificate.StoreType, applicationCertificate.StorePath, configuration.ApplicationUri, configuration.ApplicationName, null, serverDomainNames, keySize, lifetimeInMonths);
Trace.TraceInformation("Created new certificate with SubjectName '{0}', in certificate store '{1}'.", applicationCertificate.SubjectName, applicationCertificate.StorePath);
configuration.CertificateValidator.Update(configuration.SecurityConfiguration);
}
}

With a more recent version of the library, there is a built-in option to check the application instance certificate. It's available on the ApplicationInstance class.
Here's how you'd use it:
var applicationConfiguration = new ApplicationConfiguration
{
ApplicationName = "Aggregation server",
...
};
await applicationConfiguration.Validate(ApplicationType.ClientAndServer);
var applicationInstance = new ApplicationInstance(applicationConfiguration);
// This call will check that the application instance certificate exists, and will create it if not
var result =
await applicationInstance.CheckApplicationInstanceCertificate(false, CertificateFactory.DefaultKeySize);
var server = new AggregationServer();
await applicationInstance.Start(server);
System.Console.ReadKey();
server.Stop();

WebConsumer.ProcessUserAuthorization returns null

I use DotNetOpenAuth.
So.. I am getting looking good response which has state Authenticated.
That is fine.
Now I want to get user profile info but always getting NULL.
Here is the code.
private ServiceProviderDescription GetServiceDescription()
{
string ValidateTokenEndPoint = ConfigurationManager.AppSettings["identityOAuthValidateTokenEndPointUrl"];
string ValidateAuthorizationHeaderEndPoint = ConfigurationManager.AppSettings["identityOAuthValidateAuthorizationHeaderEndPointUrl"];
string AccessTokenEndPoint = ConfigurationManager.AppSettings["identityOAuthAccessTokenURL"];
bool UseVersion10A = Convert.ToBoolean(ConfigurationManager.AppSettings["identityOAuthUseVersion10a"]);
string RequestTokenStr = ConfigurationManager.AppSettings["identityOAuthRequestTokenURL"];
string UserAuthStr = ConfigurationManager.AppSettings["identityOAuthAuthorizeUserURL"];
string AccessTokenStr = ConfigurationManager.AppSettings["identityOAuthAccessTokenURL"];
string InvalidateTokenStr = ConfigurationManager.AppSettings["identityOAuthRequestInvalidateTokenURL"];
return new ServiceProviderDescription
{
AccessTokenEndpoint = new MessageReceivingEndpoint(AccessTokenStr, HttpDeliveryMethods.PostRequest),
RequestTokenEndpoint = new MessageReceivingEndpoint(RequestTokenStr, HttpDeliveryMethods.PostRequest),
UserAuthorizationEndpoint = new MessageReceivingEndpoint(UserAuthStr, HttpDeliveryMethods.PostRequest),
TamperProtectionElements = new ITamperProtectionChannelBindingElement[] { new HmacSha1SigningBindingElement() },
ProtocolVersion = DotNetOpenAuth.OAuth.ProtocolVersion.V10a
};
}
void GetUserProfile()
{
var tokenManager = TokenManagerFactory.GetTokenManager(TokenManagerType.InMemoryTokenManager);
tokenManager.ConsumerKey = ConfigurationManager.AppSettings["identityOAuthConsumerKey"];
tokenManager.ConsumerSecret = ConfigurationManager.AppSettings["identityOAuthConsumerSecret"];
var serviceDescription = GetServiceDescription();
var consumer = new WebConsumer(serviceDescription, tokenManager);
var result = consumer.ProcessUserAuthorization(response);
if (result != null) // It is always null
{
}
Well I checked 10 times and I am pretty sure that all URLs to create ServiceProviderDescription are correct.
Any clue?

Well
finally check your web.config app keys
add key="identityOAuthConsumerKey" value="put here correct data!!!"
add key="identityOAuthConsumerSecret" value="put here correct data!!!"
and if you use hosts file you have to put correct sitename as well
127.0.0.1 site1.host1.com

How do I implement a C# Thrift service and consume it with a Silverlight client?

I'm current looking at Thrift to use as a RPC framework for our apps (mostly written in C# and Silverlight). I've come as far as implementing a service and consuming it from a C# console app (using a socket as transport).
For the C# server side code my code looked like: (basically copying the tutorials included with the source code)
MyServiceHandler handler = new MyServiceHandler();
MyService.Processor processor = new MyService.Processor(handler);
TServerTransport serverTransport = new TServerSocket(9090);
TServer server = new TSimpleServer(processor, serverTransport);
server.Serve();
For the client side code it looked like:
TTransport transport = new TSocket("localhost", 9090);
TProtocol protocol = new TBinaryProtocol(transport);
MyService.Client client = new MyService.Client(protocol);
transport.Open();
client.SomeServiceCall();
However, we will be consuming the service from a Silverlight client, and unfortunately there is no support for sockets in Silverlight for Thrift. I assume I'm forced to use HTTP communication between the client and service, using Thrift's C# THttpClient and THttpHandler classes? I could not find any examples of how to do this out there, can anyone point me in the right direction? Some example server and client side code would be appreciated.

It seems that this issue was already addressed by this guy. According to this JIRA, the fix is available in Thrift 0.9. You can either try this snapshot (note that, as it's not a final release, it might not be stable) or you can apply this patch to the 0.8 release.

I believe by now you would have understood, there is no direct way of communicating from Silverlight to the Cassandra database either using Thrift or any other clients.
I have one simple option related to this. Write a Silverlight enabled web service and consume it from the client.
For example, on the server side you can have a web service which does insert/update/read etc., like this. I just managed to pull out some code which we use for our project. Hope this helps.
using Apache.Cassandra;
using Thrift.Protocol;
using Thrift.Transport;
namespace CassandraWebLibrary
{
public class MyDb
{
String _host;
int _port;
String _keyspace;
bool _isConnected;
TTransport _transport = null;
Apache.Cassandra.Cassandra.Client _client = null;
String columnFamily = "ColumnFamilyName";
public VazhikaattiDB(String host, int port, String keyspace)
{
_host = host;
_port = port;
_keyspace = keyspace;
_isConnected = false;
}
public bool Connect()
{
try
{
_transport = new TFramedTransport(new TSocket(_host, _port));
TProtocol protocol = new TBinaryProtocol(_transport);
_client = new Apache.Cassandra.Cassandra.Client(protocol);
_transport.Open();
_client.set_keyspace(_keyspace);
_isConnected = true;
}
catch (Exception ex)
{
log.Error(ex.ToString());
}
return _isConnected;
}
public bool Close()
{
if (_transport.IsOpen)
_transport.Close();
_isConnected = false;
return true;
}
public bool InsertData(Send your data as parameters here)
{
try
{
List<Column> list = new List<Column>();
string strKey = keyvalue;
#region Inserting into Coulmn family
List<Byte> valbytes = new List<byte>(BitConverter.GetBytes(value)); //You might have to pad this with more bytes to make it length of 8 bytes
Column doublecolumn1 = new Column()
{
Name = Encoding.UTF8.GetBytes("column1"),
Timestamp = timestampvalue,
Value = valbytes.ToArray()
};
list.Add(doublecolumn1);
Column stringcolumn2 = new Column()
{
Name = Encoding.UTF8.GetBytes("column2"),
Timestamp = timestampvalue,
Value = Encoding.UTF8.GetBytes("StringValue")
};
list.Add(stringcolumn2);
Column timecolumn3 = new Column()
{
Name = Encoding.UTF8.GetBytes("column3"),
Timestamp = timestampvalue,
Value = BitConverter.GetBytes(DateTime.Now.Ticks)
};
list.Add(timecolumn3);
#endregion
ColumnParent columnParent = new ColumnParent();
columnParent.Column_family = columnFamily;
Byte[] key = Encoding.UTF8.GetBytes(strKey);
foreach (Column column in list)
{
try
{
_client.insert(key, columnParent, column, ConsistencyLevel.QUORUM);
}
catch (Exception e)
{
log.Error(e.ToString());
}
}
return true;
}
catch (Exception ex)
{
log.Error(ex.ToString());
return false;
}
}
public List<YourReturnObject> GetData(parameters)
{
try
{
ColumnParent columnParent = new ColumnParent();
columnParent.Column_family = columnFamily;
DateTime curdate = startdate;
IndexExpression indExprsecondkey = new IndexExpression();
indExprsecondkey.Column_name = Encoding.UTF8.GetBytes("column");
indExprsecondkey.Op = IndexOperator.EQ;
List<Byte> valbytes = PadLeftBytes((int)yourid, 8);
indExprsecondkey.Value = valbytes.ToArray();
indExprList.Add(indExprsecondkey);
IndexClause indClause = new IndexClause()
{
Expressions = indExprList,
Count = 1000,
Start_key = Encoding.UTF8.GetBytes("")
};
SlicePredicate slice = new SlicePredicate()
{
Slice_range = new SliceRange()
{
//Start and Finish cannot be null
Start = new byte[0],
Finish = new byte[0],
Count = 1000,
Reversed = false
}
};
List<KeySlice> keyslices = _client.get_indexed_slices(columnParent, indClause, slice, ConsistencyLevel.ONE);
foreach (KeySlice ks in keyslices)
{
String stringcolumnvalue = Encoding.UTF8.GetString(cl.Column.Value);
double doublevalue= (Double)BitConverter.ToDouble(cl.Column.Value);
long timeticks = BitConverter.ToInt64(cl.Column.Value, 0);
DateTime dtcolumntime = new DateTime(timeticks);
}
}
catch (Exception ex)
{
log.Error(ex.ToString());
}
return yourdatalist;
}
}
}
Now the above class can be used by your webservice, which in turn will be used by Silverlight. Btw, you'll have to take care of other silverlight issues like size of data to be downloaded from server/webservice etc.,
FYI, our client service of Cassandra runs on port 9160..