I am fairly new to ZeroMQ and have been comparing security of messages using the ZeroMQ NuGet package and the NetMQ & NetMQ Security NuGet packages.
So far, I have not been able to find a C# version of the Ironhouse example using Curve Security. There is a "todo" item on the ZGuides repo but so far nothing implemented. (https://github.com/metadings/zguide/issues/1)
I am also trying to determine whether the NetMQ.Security approach to security is better than the curve security approach that is built into ZeroMQ 4. It seems like most information about Curve is at least from 2014 or earlier.
Any information would be greatly appreciated!
Both publisher and subscriber need to use its own set of public\private keys. In your sample code for subscriber you set CurvePublicKey (to that of server, which is wrong but still) but do not set CurveSecretKey - that's why you get "cannot open client INITIATE vouch". Here is your sample from another question fixed:
public class Program
{
static void Main(string[] args) {
using (var context = new ZContext()) {
Console.WriteLine($"Curve Supported: {ZeroMQ.ZContext.Has("curve")}");
byte[] serverPublicKey;
byte[] serverSecretKey;
Z85.CurveKeypair(out serverPublicKey, out serverSecretKey);
var publisher = new ZSocket(context, ZSocketType.PUB);
publisher.CurvePublicKey = serverPublicKey;
publisher.CurveSecretKey = serverSecretKey;
publisher.CurveServer = true;
publisher.Bind("tcp://*:5050");
var subscriber = new ZSocket(context, ZSocketType.SUB);
byte[] subPublicKey;
byte[] subSecretKey;
Z85.CurveKeypair(out subPublicKey, out subSecretKey);
subscriber.CurvePublicKey = subPublicKey;
subscriber.CurveSecretKey = subSecretKey;
subscriber.CurveServerKey = serverPublicKey;
ZError connectError;
subscriber.Connect("tcp://mybox:5050", out connectError);
if (connectError != null) {
Console.WriteLine($"Connection error: {connectError.Name} - {connectError.Number} - {connectError.Text}");
}
subscriber.SubscribeAll();
// Publish some messages
Task.Run(() => {
for (var i = 1; i <= 5; i++) {
var msg = $"Pub msg: {Guid.NewGuid().ToString()}";
using (var frame = new ZFrame(msg)) {
publisher.Send(frame);
}
}
});
Task.Run(() => {
// Receive some messages
while (true) {
using (var frame = subscriber.ReceiveFrame()) {
var msg = frame.ReadString();
Console.WriteLine($"Received: {msg}");
}
}
});
Console.WriteLine("Press ENTER to exit");
Console.ReadLine();
ZError subError;
subscriber.Disconnect("tcp://mybox:5050", out subError);
subscriber.Dispose();
ZError pubError;
publisher.Disconnect("tcp://*:5050", out pubError);
publisher.Dispose();
}
}
}
Indeed, there are not much C# examples with NetMQ. I found this that works "CurveTests.cs":
public void CurveTest()
{
var serverPair = new NetMQCertificate();
using var server = new DealerSocket();
server.Options.CurveServer = true;
server.Options.CurveCertificate = serverPair;
server.Bind($"tcp://127.0.0.1:55367");
var clientPair = new NetMQCertificate();
using var client = new DealerSocket();
client.Options.CurveServerKey = serverPair.PublicKey;
client.Options.CurveCertificate = clientPair;
client.Connect("tcp://127.0.0.1:55367");
for (int i = 0; i < 100; i++)
{
client.SendFrame("Hello");
var hello = server.ReceiveFrameString();
Assert.Equal("Hello", hello);
server.SendFrame("World");
var world = client.ReceiveFrameString();
Assert.Equal("World", world);
}
}
Important note - if you want to share server public key between different applications, don't use string representation (serverPair.PublicKeyZ85), because encryption won't work. I assume it related to encoding. Better save byte array representation to some file and share it instead:
File.WriteAllBytes("serverPublicKey.txt", serverPair.PublicKey);
Related
I'm trying to create a topic (an Event Hub) programmatically from the Kafka interface using AdminClient.CreateTopicsAsync. This works when connecting to Kafka, but not to Event Hub. I'm running into the following error:
Default partition count (KIP-464) not supported by broker, requires
broker version <= 2.4.0
using Confluent.Kafka;
using Confluent.Kafka.Admin;
var adminClient =
new AdminClientBuilder(
new[] {
("sasl.mechanism","PLAIN"),
("security.protocol","SASL_SSL"),
("bootstrap.servers", Address),
("sasl.username", "$ConnectionString"),
("sasl.password", ConnectionString),
}.Select((kvp) => new KeyValuePair<string, string>(kvp.Item1, kvp.Item2))
)
.Build();
await adminClient.CreateTopicsAsync(new[] {
new TopicSpecification {
Name = "test-topic"
}
});
It complains that using a default number of partitions is not supported, but as far as I can tell, I can't provide one as the underlying librdkafka does not support it.
The only information I could find by googling this is that someone in 2021 did make it work.
This code works for me on both Kafka and EventHub.
using (var kafkaProducer = new ProducerBuilder<string, string>(producerConfig)
.Build())
{
using (var adminClient = new DependentAdminClientBuilder(kafkaProducer.Handle).Build())
{
var metaData = adminClient.GetMetadata(TimeSpan.FromSeconds(5));
var topicInfo = metaData.Topics.Where(tp => string.Equals(fullTopicName, tp.Topic, StringComparison.OrdinalIgnoreCase)).FirstOrDefault();
if (topicInfo == null)
{
var t = new Confluent.Kafka.Admin.TopicSpecification
{
Name = fullTopicName,
// at least 2 partitions
NumPartitions = kafkaTestConfig.CreateTopicOptions.NumPartitions, // 3, //1,
//// at least 1 replication factor
ReplicationFactor = kafkaTestConfig.CreateTopicOptions.ReplicationFactor, // 3, //(short)numberOfBrokers,
Configs = kafkaTestConfig.CreateTopicOptions.DynamicConfigs,
};
var o = new CreateTopicsOptions { OperationTimeout = TimeSpan.FromMilliseconds(_timeout), };
AssertES.True(adminClient.CreateTopicsAsync(new List<Confluent.Kafka.Admin.TopicSpecification> { t, }
, o).Wait(_timeout), "Failed to create topic in time: " + fullTopicName);
}
}
}
I am doing some performance tests on ZeroMQ in order to compare it with others like RabbitMQ and ActiveMQ.
In my broadcast tests and to avoid "The Dynamic Discovery Problem" as referred by ZeroMQ documentation I have used a proxy. In my scenario, I am using 50 concurrent publishers each one sending 500 messages with 1ms delay between sends. Each message is then read by 50 subscribers. And as I said I am losing messages, each of the subscribers should receive a total of 25000 messages and they are each receiving between 5000 and 10000 messages only.
I am using Windows and C# .Net client clrzmq4 (4.1.0.31).
I have already tried some solutions that I found on other posts:
I have set linger to TimeSpan.MaxValue
I have set ReceiveHighWatermark to 0 (as it is presented as infinite, but I have tried also Int32.MaxValue)
I have set checked for slow start receivers, I made receivers start some seconds before publishers
I had to make sure that no garbage collection is made to the socket instances (linger should do it but to make sure)
I have a similar scenario (with similar logic) using NetMQ and it works fine. The other scenario does not use security though and this one does (and that's also the reason why I use clrzmq in this one because I need client authentication with certificates that is not yet possible on NetMQ).
EDIT:
public class MCVEPublisher
{
public void publish(int numberOfMessages)
{
string topic = "TopicA";
ZContext ZContext = ZContext.Create();
ZSocket publisher = new ZSocket(ZContext, ZSocketType.PUB);
//Security
// Create or load certificates
ZCert serverCert = Main.GetOrCreateCert("publisher");
var actor = new ZActor(ZContext, ZAuth.Action, null);
actor.Start();
// send CURVE settings to ZAuth
actor.Frontend.Send(new ZFrame("VERBOSE"));
actor.Frontend.Send(new ZMessage(new List<ZFrame>()
{ new ZFrame("ALLOW"), new ZFrame("127.0.0.1") }));
actor.Frontend.Send(new ZMessage(new List<ZFrame>()
{ new ZFrame("CURVE"), new ZFrame(".curve") }));
publisher.CurvePublicKey = serverCert.PublicKey;
publisher.CurveSecretKey = serverCert.SecretKey;
publisher.CurveServer = true;
publisher.Linger = TimeSpan.MaxValue;
publisher.ReceiveHighWatermark = Int32.MaxValue;
publisher.Connect("tcp://127.0.0.1:5678");
Thread.Sleep(3500);
for (int i = 0; i < numberOfMessages; i++)
{
Thread.Sleep(1);
var update = $"{topic} {"message"}";
using (var updateFrame = new ZFrame(update))
{
publisher.Send(updateFrame);
}
}
//just to make sure it does not end instantly
Thread.Sleep(60000);
//just to make sure publisher is not garbage collected
ulong Affinity = publisher.Affinity;
}
}
public class MCVESubscriber
{
private ZSocket subscriber;
private List<string> prints = new List<string>();
public void read()
{
string topic = "TopicA";
var context = new ZContext();
subscriber = new ZSocket(context, ZSocketType.SUB);
//Security
ZCert serverCert = Main.GetOrCreateCert("xpub");
ZCert clientCert = Main.GetOrCreateCert("subscriber");
subscriber.CurvePublicKey = clientCert.PublicKey;
subscriber.CurveSecretKey = clientCert.SecretKey;
subscriber.CurveServer = true;
subscriber.CurveServerKey = serverCert.PublicKey;
subscriber.Linger = TimeSpan.MaxValue;
subscriber.ReceiveHighWatermark = Int32.MaxValue;
// Connect
subscriber.Connect("tcp://127.0.0.1:1234");
subscriber.Subscribe(topic);
while (true)
{
using (var replyFrame = subscriber.ReceiveFrame())
{
string messageReceived = replyFrame.ReadString();
messageReceived = Convert.ToString(messageReceived.Split(' ')[1]);
prints.Add(messageReceived);
}
}
}
public void PrintMessages()
{
Console.WriteLine("printing " + prints.Count);
}
}
public class Main
{
static void Main(string[] args)
{
broadcast(500, 50, 50, 30000);
}
public static void broadcast(int numberOfMessages, int numberOfPublishers, int numberOfSubscribers, int timeOfRun)
{
new Thread(() =>
{
using (var context = new ZContext())
using (var xsubSocket = new ZSocket(context, ZSocketType.XSUB))
using (var xpubSocket = new ZSocket(context, ZSocketType.XPUB))
{
//Security
ZCert serverCert = GetOrCreateCert("publisher");
ZCert clientCert = GetOrCreateCert("xsub");
xsubSocket.CurvePublicKey = clientCert.PublicKey;
xsubSocket.CurveSecretKey = clientCert.SecretKey;
xsubSocket.CurveServer = true;
xsubSocket.CurveServerKey = serverCert.PublicKey;
xsubSocket.Linger = TimeSpan.MaxValue;
xsubSocket.ReceiveHighWatermark = Int32.MaxValue;
xsubSocket.Bind("tcp://*:5678");
//Security
serverCert = GetOrCreateCert("xpub");
var actor = new ZActor(ZAuth.Action0, null);
actor.Start();
// send CURVE settings to ZAuth
actor.Frontend.Send(new ZFrame("VERBOSE"));
actor.Frontend.Send(new ZMessage(new List<ZFrame>()
{ new ZFrame("ALLOW"), new ZFrame("127.0.0.1") }));
actor.Frontend.Send(new ZMessage(new List<ZFrame>()
{ new ZFrame("CURVE"), new ZFrame(".curve") }));
xpubSocket.CurvePublicKey = serverCert.PublicKey;
xpubSocket.CurveSecretKey = serverCert.SecretKey;
xpubSocket.CurveServer = true;
xpubSocket.Linger = TimeSpan.MaxValue;
xpubSocket.ReceiveHighWatermark = Int32.MaxValue;
xpubSocket.Bind("tcp://*:1234");
using (var subscription = ZFrame.Create(1))
{
subscription.Write(new byte[] { 0x1 }, 0, 1);
xpubSocket.Send(subscription);
}
Console.WriteLine("Intermediary started, and waiting for messages");
// proxy messages between frontend / backend
ZContext.Proxy(xsubSocket, xpubSocket);
Console.WriteLine("end of proxy");
//just to make sure it does not end instantly
Thread.Sleep(60000);
//just to make sure xpubSocket and xsubSocket are not garbage collected
ulong Affinity = xpubSocket.Affinity;
int ReceiveHighWatermark = xsubSocket.ReceiveHighWatermark;
}
}).Start();
Thread.Sleep(5000); //to make sure proxy started
List<MCVESubscriber> Subscribers = new List<MCVESubscriber>();
for (int i = 0; i < numberOfSubscribers; i++)
{
MCVESubscriber ZeroMqSubscriber = new MCVESubscriber();
new Thread(() =>
{
ZeroMqSubscriber.read();
}).Start();
Subscribers.Add(ZeroMqSubscriber);
}
Thread.Sleep(10000);//to make sure all subscribers started
for (int i = 0; i < numberOfPublishers; i++)
{
MCVEPublisher ZeroMqPublisherBroadcast = new MCVEPublisher();
new Thread(() =>
{
ZeroMqPublisherBroadcast.publish(numberOfMessages);
}).Start();
}
Thread.Sleep(timeOfRun);
foreach (MCVESubscriber Subscriber in Subscribers)
{
Subscriber.PrintMessages();
}
}
public static ZCert GetOrCreateCert(string name, string curvpath = ".curve")
{
ZCert cert;
string keyfile = Path.Combine(curvpath, name + ".pub");
if (!File.Exists(keyfile))
{
cert = new ZCert();
Directory.CreateDirectory(curvpath);
cert.SetMeta("name", name);
cert.Save(keyfile);
}
else
{
cert = ZCert.Load(keyfile);
}
return cert;
}
}
This code also produces the expected number of messages when security is disabled, but when turned on it doesn't.
Does someone know another thing to check? Or has it happened to anyone before?
Thanks
I'm trying to create an API that consumes various topics.
For this, I'm trying to multi-thread things, so that the whole thing can be scalable into multiple APIs, later on, but that's very besides the point.
I'm using ASP.net Core 4.0, if that's got anything to do with it. Entity Framework as well.
My problem is based on my connection to my Mosquitto server being broken without throwing an exception or anything of the like, after a minute or so. It doesn't matter how big the messages are, or how many are exchanged. I have no idea of how I can create a callback or anything of the kind to know what's going on with my connection. Can anyone help?
I'll link the code I use to establish a connection and subscribe to a connection below. Using the Subscribe method or doing it manually also changes nothing. I'm at a loss, here.
Thanks in advance!
Main.cs:
Task.Factory.StartNew(() => DataflowController.ResumeQueuesAsync());
BuildWebHost(args).Run();
DataflowController.cs:
public static Boolean Subscribe(String topic)
{
Console.WriteLine("Hello from " + topic);
MqttClient mqttClient = new MqttClient(brokerAddress);
byte code = mqttClient.Connect(Guid.NewGuid().ToString());
// Register to message received
mqttClient.MqttMsgPublishReceived += client_recievedMessageAsync;
string clientId = Guid.NewGuid().ToString();
mqttClient.Connect(clientId);
// Subscribe to topic
mqttClient.Subscribe(new String[] { topic }, new byte[] { MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE });
System.Console.ReadLine();
return true;
}
public static async Task ResumeQueuesAsync()
{
var mongoClient = new MongoClient(connectionString);
var db = mongoClient.GetDatabase(databaseName);
var topics = db.GetCollection<BsonDocument>(topicCollection);
var filter = new BsonDocument();
List<BsonDocument> result = topics.Find(filter).ToList();
var resultSize = result.Count;
Task[] subscriptions = new Task[resultSize];
MqttClient mqttClient = new MqttClient(brokerAddress);
byte code = mqttClient.Connect(Guid.NewGuid().ToString());
// Register to message received
mqttClient.MqttMsgPublishReceived += client_recievedMessageAsync;
string clientId = Guid.NewGuid().ToString();
mqttClient.Connect(clientId);
int counter = 0;
foreach(var doc in result)
{
subscriptions[counter] = new Task(() =>
{
Console.WriteLine("Hello from " + doc["topic"].ToString());
// Subscribe to topic
mqttClient.Subscribe(new String[] { doc["topic"].ToString() }, new byte[] { MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE });
System.Console.ReadLine();
});
counter++;
}
foreach(Task task in subscriptions)
{
task.Start();
}
}
static async void client_recievedMessageAsync(object sender, MqttMsgPublishEventArgs e)
{
// Handle message received
var message = System.Text.Encoding.Default.GetString(e.Message);
var topic = e.Topic;
var id = topic.Split("/")[2];
BsonDocument doc = new BsonDocument {
{"Plug ID", id },
{"Consumption", message }
};
await Save(doc, "smartPDM_consumption");
System.Console.WriteLine("Message received from " + topic + " : " + message);
}
This line was the issue:
byte code = mqttClient.Connect(Guid.NewGuid().ToString());
Deleted it, and it just worked.
I am new to WebSockets (this AM) and have set up a WCF WebSocket app that works when doing a trivial example I found online (http://www.codeproject.com/Articles/619343/Using-WebSocket-in-NET-Part).
I added Entity Framework and as soon as I add code to try to access data the process (just sending a message back and forth) no longer works.
Could there be some fundamental concept I could be missing?
Does anyone have any good ideas for troubleshooting?
namespace PBWebSocket
{
public class PBWebSocket : IBWebSocket
{
private SPEntities db = new SPEntities();
public async Task SendMessageToServer(Message msg)
{
var callback = OperationContext.Current.GetCallbackChannel<IPBCallback>();
if (msg.IsEmpty || ((IChannel)callback).State != CommunicationState.Opened)
{
return;
}
byte[] body = msg.GetBody<byte[]>();
string msgTextFromClient = Encoding.UTF8.GetString(body);
var reqId = Int32.Parse(msgTextFromClient);
// *** The below line breaks it ***
var req = db.Requests.Where(r => r.Id == 164).FirstOrDefault();
reqId = reqId + 2;
Message newMsg = ByteStreamMessage.CreateMessage(
new ArraySegment<byte>(Encoding.UTF8.GetBytes(reqId.ToString())));
newMsg.Properties["WebSocketMessageProperty"] =
new WebSocketMessageProperty
{ MessageType = WebSocketMessageType.Text };
await callback.SendMessageToClient(newMsg);
}
}
}
I'm current looking at Thrift to use as a RPC framework for our apps (mostly written in C# and Silverlight). I've come as far as implementing a service and consuming it from a C# console app (using a socket as transport).
For the C# server side code my code looked like: (basically copying the tutorials included with the source code)
MyServiceHandler handler = new MyServiceHandler();
MyService.Processor processor = new MyService.Processor(handler);
TServerTransport serverTransport = new TServerSocket(9090);
TServer server = new TSimpleServer(processor, serverTransport);
server.Serve();
For the client side code it looked like:
TTransport transport = new TSocket("localhost", 9090);
TProtocol protocol = new TBinaryProtocol(transport);
MyService.Client client = new MyService.Client(protocol);
transport.Open();
client.SomeServiceCall();
However, we will be consuming the service from a Silverlight client, and unfortunately there is no support for sockets in Silverlight for Thrift. I assume I'm forced to use HTTP communication between the client and service, using Thrift's C# THttpClient and THttpHandler classes? I could not find any examples of how to do this out there, can anyone point me in the right direction? Some example server and client side code would be appreciated.
It seems that this issue was already addressed by this guy. According to this JIRA, the fix is available in Thrift 0.9. You can either try this snapshot (note that, as it's not a final release, it might not be stable) or you can apply this patch to the 0.8 release.
I believe by now you would have understood, there is no direct way of communicating from Silverlight to the Cassandra database either using Thrift or any other clients.
I have one simple option related to this. Write a Silverlight enabled web service and consume it from the client.
For example, on the server side you can have a web service which does insert/update/read etc., like this. I just managed to pull out some code which we use for our project. Hope this helps.
using Apache.Cassandra;
using Thrift.Protocol;
using Thrift.Transport;
namespace CassandraWebLibrary
{
public class MyDb
{
String _host;
int _port;
String _keyspace;
bool _isConnected;
TTransport _transport = null;
Apache.Cassandra.Cassandra.Client _client = null;
String columnFamily = "ColumnFamilyName";
public VazhikaattiDB(String host, int port, String keyspace)
{
_host = host;
_port = port;
_keyspace = keyspace;
_isConnected = false;
}
public bool Connect()
{
try
{
_transport = new TFramedTransport(new TSocket(_host, _port));
TProtocol protocol = new TBinaryProtocol(_transport);
_client = new Apache.Cassandra.Cassandra.Client(protocol);
_transport.Open();
_client.set_keyspace(_keyspace);
_isConnected = true;
}
catch (Exception ex)
{
log.Error(ex.ToString());
}
return _isConnected;
}
public bool Close()
{
if (_transport.IsOpen)
_transport.Close();
_isConnected = false;
return true;
}
public bool InsertData(Send your data as parameters here)
{
try
{
List<Column> list = new List<Column>();
string strKey = keyvalue;
#region Inserting into Coulmn family
List<Byte> valbytes = new List<byte>(BitConverter.GetBytes(value)); //You might have to pad this with more bytes to make it length of 8 bytes
Column doublecolumn1 = new Column()
{
Name = Encoding.UTF8.GetBytes("column1"),
Timestamp = timestampvalue,
Value = valbytes.ToArray()
};
list.Add(doublecolumn1);
Column stringcolumn2 = new Column()
{
Name = Encoding.UTF8.GetBytes("column2"),
Timestamp = timestampvalue,
Value = Encoding.UTF8.GetBytes("StringValue")
};
list.Add(stringcolumn2);
Column timecolumn3 = new Column()
{
Name = Encoding.UTF8.GetBytes("column3"),
Timestamp = timestampvalue,
Value = BitConverter.GetBytes(DateTime.Now.Ticks)
};
list.Add(timecolumn3);
#endregion
ColumnParent columnParent = new ColumnParent();
columnParent.Column_family = columnFamily;
Byte[] key = Encoding.UTF8.GetBytes(strKey);
foreach (Column column in list)
{
try
{
_client.insert(key, columnParent, column, ConsistencyLevel.QUORUM);
}
catch (Exception e)
{
log.Error(e.ToString());
}
}
return true;
}
catch (Exception ex)
{
log.Error(ex.ToString());
return false;
}
}
public List<YourReturnObject> GetData(parameters)
{
try
{
ColumnParent columnParent = new ColumnParent();
columnParent.Column_family = columnFamily;
DateTime curdate = startdate;
IndexExpression indExprsecondkey = new IndexExpression();
indExprsecondkey.Column_name = Encoding.UTF8.GetBytes("column");
indExprsecondkey.Op = IndexOperator.EQ;
List<Byte> valbytes = PadLeftBytes((int)yourid, 8);
indExprsecondkey.Value = valbytes.ToArray();
indExprList.Add(indExprsecondkey);
IndexClause indClause = new IndexClause()
{
Expressions = indExprList,
Count = 1000,
Start_key = Encoding.UTF8.GetBytes("")
};
SlicePredicate slice = new SlicePredicate()
{
Slice_range = new SliceRange()
{
//Start and Finish cannot be null
Start = new byte[0],
Finish = new byte[0],
Count = 1000,
Reversed = false
}
};
List<KeySlice> keyslices = _client.get_indexed_slices(columnParent, indClause, slice, ConsistencyLevel.ONE);
foreach (KeySlice ks in keyslices)
{
String stringcolumnvalue = Encoding.UTF8.GetString(cl.Column.Value);
double doublevalue= (Double)BitConverter.ToDouble(cl.Column.Value);
long timeticks = BitConverter.ToInt64(cl.Column.Value, 0);
DateTime dtcolumntime = new DateTime(timeticks);
}
}
catch (Exception ex)
{
log.Error(ex.ToString());
}
return yourdatalist;
}
}
}
Now the above class can be used by your webservice, which in turn will be used by Silverlight. Btw, you'll have to take care of other silverlight issues like size of data to be downloaded from server/webservice etc.,
FYI, our client service of Cassandra runs on port 9160..