Azure IoT Hub Supports AMQP, MQTT, HTTP protocols. In order to customize these protocols we have Azure IoT protocol gateway. I can find good samples on MQTT protocol customization. I need some sample codes for TCP based protocol customization using Azure IoT Protocol Gateway.
EDIT (in order to get an answer): what the OP was asking, is an example using the Azure Protocol Gateway to support a proprietary TCP-based protocol. Currently the IoT hub only supports AMQP, MQTT and HTTP. While those protocols actually rely on TCP, the hub doesn't support direct TCP connection without the extra layer of AMQP, MQTT or HTTP.
As explained here, we need a basic example of a custom TCP based protocol.
Imagine a basic device that can only send some proprietary payload through TCP on a given IP address/port: we need an example of a gateway customization that allows this device to send data to the hub.
The current code of the protocol gateway is poorly designed, as it heavily relies on MQTT.
Adding some bounty too.
The default Protocol Gateway sample are indeed somewhat confusing because of all the MQTT code.
The protocol gateway works by 'simulating' a IoTHub connection for each custom protocol device you connect to the gateway.
To do this translation from the TCP device to an IoTHub device you first need to have a connection to the IoTHub on behalf of the device. This is the gateway part.
Below is the core essentials for this IoTHubConnection.
namespace GatewayTest
{
using System;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using DotNetty.Buffers;
using Microsoft.Azure.Devices.ProtocolGateway.Identity;
using Microsoft.Azure.Devices.ProtocolGateway.IotHubClient;
using Microsoft.Azure.Devices.ProtocolGateway.Messaging;
public class IoTHubConnection : IMessagingChannel<IMessage>
{
private readonly string iotHubHostName;
private readonly Func<IDeviceIdentity, Task<IMessagingServiceClient>> deviceClientFactory;
private readonly Func<string, Task> onMessage;
private IMessagingServiceClient deviceClient;
private IDeviceIdentity deviceIdentity;
public IoTHubConnection(
string iotHubHostName,
Func<IDeviceIdentity, Task<IMessagingServiceClient>> deviceClientFactory,
Func<string, Task> onMessage)
{
this.iotHubHostName = iotHubHostName;
this.deviceClientFactory = deviceClientFactory;
this.onMessage = onMessage;
}
public event EventHandler CapabilitiesChanged;
public async Task OpenAsync(string deviceId, string deviceKey)
{
this.deviceIdentity = this.GetDeviceIdentity(deviceId, deviceKey);
if (this.deviceIdentity != UnauthenticatedDeviceIdentity.Instance)
{
this.deviceClient = await this.deviceClientFactory(this.deviceIdentity);
this.deviceClient.BindMessagingChannel(this);
}
}
public async Task CloseAsync()
{
await this.deviceClient.DisposeAsync(null);
this.deviceClient = null;
}
public void Handle(IMessage message)
{
var messageBody = message.Payload.ToString(Encoding.UTF8);
this.onMessage(messageBody);
this.deviceClient.CompleteAsync(message.Id);
}
public Task SendMessage(string message)
{
var buffer = Unpooled.WrappedBuffer(Encoding.UTF8.GetBytes(message));
var deviceMessage = this.deviceClient.CreateMessage($"devices/{this.deviceIdentity.Id}/messages/events", buffer);
return this.deviceClient.SendAsync(deviceMessage);
}
protected virtual void OnCapabilitiesChanged(EventArgs e)
{
this.CapabilitiesChanged?.Invoke(this, e);
}
private IDeviceIdentity GetDeviceIdentity(string userName, string deviceKey)
{
IotHubDeviceIdentity ideviceIdentity;
if (!IotHubDeviceIdentity.TryParse($"{this.iotHubHostName}/{userName}", out ideviceIdentity))
{
return UnauthenticatedDeviceIdentity.Instance;
}
ideviceIdentity.WithDeviceKey(deviceKey);
return ideviceIdentity;
}
}
}
The deviceClientFactory callback method should be implemented as shown below and in this line in the ProtocolGateway repo in Github.
deviceClientFactory = IotHubClient.PreparePoolFactory(
"IotHubConnectionString",
400,
TimeSpan.FromMinutes(3),
iotHubClientSettings,
PooledByteBufferAllocator.Default,
new ConfigurableMessageAddressConverter("TopicNameConversion"));
When a Tcp Device connects to the protocol, you should create an instance of this IoTHubConnection and send messages from the Device to the IoTHubConnection and vica versa.
The code below shows a very simple version of how this should be done.
private const int BufferSize = 1024;
private byte[] buffer = new byte[BufferSize];
private IoTHubConnection ioTHubConnection;
private NetworkStream stream;
private async Task Start()
{
listener = new TcpListener(IPAddress.Any, port);
listener.Start();
var client = await listener.AcceptTcpClientAsync();
ioTHubConnection = new IoTHubConnection("IoTHubName", deviceClientFactory, OnIoTHubMessage);
stream = client.GetStream();
// Read DeviceId and DeviceKey from some sort of StartConnection-message send by the TcpClient.
await ioTHubConnection.OpenAsync("DeviceId", "DeviceKey");
stream.BeginRead(buffer, 0, BufferSize, ReadTcpStreamCallback, null);
}
private void ReadTcpStreamCallback(IAsyncResult ar)
{
var bytesRead = stream.EndRead(ar);
if (bytesRead > 0)
{
var message = System.Text.Encoding.ASCII.GetString(result);
ioTHubConnection.SendMessage(message);
// Read again.
stream.BeginRead(buffer, 0, BufferSize, ReadTcpStreamCallback, null);
}
}
private async Task OnIoTHubMessage(string message)
{
// Potentially do some translation on the IoTHub message
// and send it to the Device
var byteData = Encoding.UTF8.GetBytes(message);
stream.BeginWrite(byteData, 0, byteData.Length, SendTcpCallback, null);
}
private void SendTcpCallback(IAsyncResult ar)
{
stream.EndWrite(ar);
}
I know I am late to this conversation. However I have interesting add on or might be a solution for some.
Azure IoT Gateway is now known as Azure IoT Edge, this is clearly mentioned in the following Azure github repo
https://github.com/Azure/iot-edge-modbus.git
On the other hand, Azure IoT Edge supports TCP for some protocols which can be found in the following links
https://learn.microsoft.com/en-us/azure/iot-edge/deploy-modbus-gateway
https://learn.microsoft.com/en-us/azure/iot-edge/iot-edge-as-gateway
Related
I have a project creating a websocket client side (Subscriber) to a MQTT publisher. I am quite new to C# and MQTT protocol. I follow some youtube video to make finish my very first lines connecting to this MQTT publisher to get all the train going in and out Helsinki station.
broker: "rata.digitraffic.fi"
Port: 80
Topic: trains-by-station/HKI (HKI abbr for Helsinki)
I use M2Mqtt library in dotnet to build the subscriber, somehow the client_MqttMsgPublishReceived function is never triggered. the client.IsConnected always returned false value!
You can find info of this mqtt protocol in the url below.
https://www.digitraffic.fi/rautatieliikenne/#websocket-mqtt
It gives me example in JavaScripts and it seems to run fine with the example of each MQTT. But when I tried to do it with my PC, it doesn't give me any thing, but
Hello World!!!
False
and the cmd window on hold.
SOOOO FRUSTRATING right now. it would be much appreciate if anyone can help me out.
BTW, I am using win10, I tried with dotnet 4/5/6 and m2mqtt 4.3.0.
using System.Text;
using uPLibrary.Networking.M2Mqtt;
using uPLibrary.Networking.M2Mqtt.Messages;
using System;
namespace m2qttSubscriber
{
internal class Program
{
static void Main(string[] args)
{
Console.WriteLine("Hello World!!!!");
MqttClient client = new MqttClient("rata.digitraffic.fi",
80,
false,
MqttSslProtocols.None,
null,
null);
client.MqttMsgPublishReceived += client_MqttMsgPublishReceived;
string clientID = "myclientid_" + RandomDigits(4);
client.Connect(clientID);
Console.WriteLine(client.IsConnected);
client.Subscribe(new string[] { "trains-by-station/HKI" },
new byte[] { MqttMsgBase.QOS_LEVEL_AT_MOST_ONCE}) ;
}
static void client_MqttMsgPublishReceived(object sender, MqttMsgPublishEventArgs e)
{
Console.WriteLine("SOme thing is received");
string payload = Encoding.Default.GetString(e.Message);
Console.WriteLine("Recevied {0} from", payload);
}
static public string RandomDigits(int length)
{
var random = new Random();
string s = string.Empty;
for (int i = 0; i < length; i++)
s = String.Concat(s, random.Next(10).ToString());
return s;
}
}
}
As per the comments the library used (uPLibrary.Networking.M2Mqtt) does not support MQTT over Websockets (which is what rata.digitraffic.fi:80 offers).
If you are able to use standard MQTT (over TCP/IP) then rata-mqtt.digitraffic.fi:1883 works (OP succeeded with MqttClient client = new MqttClient("rata-mqtt.digitraffic.fi", 1883, false, MqttSslProtocols.None, null, null);) and this is generally preferable to using WebSockets (there are some situations where you have to use WebSockets; e.g. code running in a browser or to bypass some filters/proxies).
Alternatively there are other libraries that do offer support for MQTT over Websockets.
I'm adding a multicast listening feature for an application I'm developing. A device I'm connecting to will send out a multicast packet every second in certain scenarios, and I'd like the user to be able to listen for that packet (if it's being sent).
I can see the multicast packets being continuously sent from the device using Wireshark, so I know they exist and I know I'm using the correct multicast group and port number. I've tried dozen of different ways without any luck to get my application to capture the packets. If I send a test multicast packet from the application itself it receives it no problem. I've tried to receive the packets both async and sync, no change. I'm really stumped on this one and not sure what I'm doing wrong. Every example I've found leads me to believe this should be working.
My multicast udp listener class:
class MulticastClient
{
private UdpClient client;
private IPEndPoint localEp;
private IPAddress multicastAddress;
public byte[] receivedData { get; private set; }
public MulticastClient(string multicastIP, int port)
{
localEp = new IPEndPoint(IPAddress.Any, port);
client = new UdpClient(AddressFamily.InterNetwork);
client.Client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
client.ExclusiveAddressUse = false;
client.Client.Bind(localEp);
this.multicastAddress = IPAddress.Parse(multicastIP);
client.JoinMulticastGroup(this.multicastAddress);
}
public async Task Listen()
{
UdpReceiveResult result = await client.ReceiveAsync();
receivedData = result.Buffer;
}
}
The button click that kicks it off:
MulticastClient client = new MulticastClient("239.255.0.1", 32768);
await Task.Run(async () =>
{
await client.Listen();
});
byte[] receivedData = client.receivedData;
if (receivedData != null)
{
//Here I display useful information about the packet to the user
}
Here is a snippet of a packet capture, showing the multicast packets that I can't seem to receive:
Wireshark packet capture
Got it to work using a variation of this post.
#jdweng made me realize it when he/she asked about multiple adapters, I had several virtual NICs for VMs that were likely joining ht multicast group instead of my actual hardware NIC.
I'm writing an application to send syslogs to a central server. The network connection to the server is expected to be unreliable. For this reason, I need to be able to check whether the messages have been received by the server, so that if the send is unsuccessful I can try sending them again later.
I have no control over the server, only the ability to form a TCP connection with it - most questions I've seen recommend writing custom ACK logic, but that's not an option for me. I need an entirely client-side approach. Is there any way I can access the TCP acknowledgements in order to mark a message as sent?
The TCP protocol guarantees that the data is properly delivered at the remote server. To quote from the original TCP specification in RFC 793:
An acknowledgment by TCP does not guarantee that the data has been
delivered to the end user, but only that the receiving TCP has taken
the responsibility to do so.
So if you do not get an error from your sending part, you have a confirm that the remove server TCP stack has properly received your data. TCP has built-in detection for damaged, lost, duplicated, or out of order delivered packets (using sequence numbers) and will retransmit packets when it detects problems.
As the other poster has mentioned, TCP is a reliable protocol. You can wrap this in some exception handling code to ensure that messages are re-sent when the connection is restored, for example:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.Net.Sockets;
namespace TCPClientTest
{
class SyslogMessage
{
public Guid Guid = Guid.NewGuid();
public byte[] MessageData;
}
class Program
{
static readonly int TCP_PORT = 1337;
static byte[] SendTCPMessage(String hostname, int port, byte[] data)
{
using (var client = new TcpClient(hostname, port))
{
using (var stream = client.GetStream())
{
stream.Write(data, 0, data.Length);
var responseData = new byte[1024];
var byteCount = stream.Read(responseData, 0, responseData.Length);
return responseData.Take(byteCount).ToArray();
}
}
}
static void SendSyslog(String hostname, int port, SyslogMessage m)
{
SendTCPMessage(hostname, port, m.MessageData);
}
static Queue<SyslogMessage> sysLogQueue = new Queue<SyslogMessage>(new List<SyslogMessage>()
{
new SyslogMessage() { MessageData = Encoding.UTF8.GetBytes("Test data 1")},
new SyslogMessage() { MessageData = Encoding.UTF8.GetBytes("Test data 2")}
});
static void Main(string[] args)
{
System.Threading.Timer timer = new System.Threading.Timer(SendLogs, null, 1000, 5000);
Console.WriteLine("Press return to continue...");
Console.Read();
}
static void SendLogs(object state)
{
while (sysLogQueue.Count > 0)
{
try
{
var m = sysLogQueue.Peek();
SendSyslog("localhost", TCP_PORT, m);
Console.WriteLine("Sent sys log: " + Encoding.UTF8.GetString(m.MessageData));
// Remove from queue.
sysLogQueue.Dequeue();
}
catch (SocketException e)
{
// Leave in the queue.
Console.WriteLine("Failed to send log: Socket exception occurred: {0}", e);
// Break until next attempt.
break;
}
}
}
}
}
I've used port 1337 for this, just set this to whatever you need.
I'm using the IBM websphere XMS API to connect and send messages to the mainframe. However, every message sent is sent through a new local port. Is there a way to set this to a fixed port?
A new port is created locally when the following line is hit:
var connContext = new XMSConnectionContext(connFactory.CreateConnection(), sendQ, replyQ, mqProfile, DateTime.Now.AddSeconds(qm.MqConfiguration.ConnectionPoolExpiryTime));
The code I'm using is
public IMQMessage Message { get; set; }
public void Initialise(IMQMessage message, QueueSet queueSet, QueueManager queueManager)
{
Message = message;
if (_connContext.ContainsKey(message.MessageId)) return;
_connContext.TryAdd(message.MessageId, ConnectQueueSet(queueSet, queueManager));
_connContext[message.MessageId].Connection.Start();
}
private XMSConnectionContext ConnectQueueSet(MQQueueSet queueSet, QueueManager qm)
{
var mqProfile = GetProfile(queueSet);
var xmsFactory = XMSFactoryFactory.GetInstance(XMSC.CT_WMQ);
var connFactory = xmsFactory.CreateConnectionFactory();
connFactory.SetStringProperty(XMSC.WMQ_HOST_NAME, mqProfile.ServerName);
connFactory.SetIntProperty(XMSC.WMQ_PORT, mqProfile.Port);
connFactory.SetStringProperty(XMSC.WMQ_CHANNEL, mqProfile.ChannelName);
connFactory.SetStringProperty(XMSC.WMQ_QUEUE_MANAGER, mqProfile.QueueManagerName);
connFactory.SetIntProperty(XMSC.WMQ_FAIL_IF_QUIESCE, 1);
connFactory.SetIntProperty(XMSC.WMQ_SHARE_CONV_ALLOWED, XMSC.WMQ_SHARE_CONV_ALLOWED_YES);
connFactory.SetIntProperty(XMSC.WMQ_CONNECTION_MODE, XMSC.WMQ_CM_CLIENT);
We've tried
connFactory.SetStringProperty(XMSC.XMSC_WMQ_LOCAL_ADDRESS,"(45000,45010)");
We've also tried
connFactory.SetStringProperty(XMSC.XMSC_WMQ_LOCAL_ADDRESS,"localhost(45000,45010)");
We've also tried
connFactory.SetStringProperty(XMSC.XMSC_WMQ_LOCAL_ADDRESS,"192.168.12.156(45000,45010)");
End of tests and the rest below is as it was.
IDestination sendQ = xmsFactory.CreateQueue(string.Format("queue://{0}/{1}?targetClient=1", mqProfile.QueueManagerName, mqProfile.RequestQueue));
IDestination replyQ = xmsFactory.CreateQueue(string.Format("queue://{0}/{1}?targetClient=1", mqProfile.QueueManagerName, mqProfile.ReplyQueue));
var connContext = new XMSConnectionContext(connFactory.CreateConnection(), sendQ, replyQ, mqProfile, DateTime.Now.AddSeconds(qm.MqConfiguration.ConnectionPoolExpiryTime));
QueueManager.Log.DebugFormat("XMSConnectionContext-Instantiated: ProfileName={0} SendQ={1}, ReplyQ={2}, ConnectionMetaData={3}", connContext.ProfileName, connContext.SendQ, connContext.ReplyQ, connContext.Connection);
return connContext;
}
public void Close()
{
if (_connContext != null)
{
_connContext[Message.MessageId].Connection.Stop();
}
}
Any help would be appreciated. Thanks.
XMS .NET has a connection factory property, XMSC_WMQ_LOCAL_ADDRESS that lets you specify a local port to be used while connecting to a queue manager. More details here
I want to preface this by saying my understanding of UDP Broadcasting and Multicasting is very limited. This is my first project working on this.
I have a C# desktop client running on a machine and a Windows phone 7 app.
The WP7 app is supposed to send a UDP broadcast over the network and the desktop client is supposed to listen for a UDP Multicast and respond accordingly. This is just meant for simple machine discovery over the network to find machines running the desktop client.
C# Desktop Client Code
public class ConnectionListener
{
private const int UDP_PORT = 54322;
private static readonly IPAddress MULTICAST_GROUP_ADDRESS = IPAddress.Parse("224.0.0.1");
private UdpClient _listener;
public ConnectionListener()
{
_listener = new UdpClient(UDP_PORT, AddressFamily.InterNetwork);
_listener.EnableBroadcast = true;
_listener.JoinMulticastGroup(MULTICAST_GROUP_ADDRESS);
_listener.BeginReceive(ReceiveCallback, null);
}
private void ReceiveCallback(IAsyncResult result)
{
IPEndPoint receiveEndpoint = new IPEndPoint(IPAddress.Any, UDP_PORT);
byte[] receivedBytes = _listener.EndReceive(result, ref receiveEndpoint);
byte[] response = Encoding.UTF8.GetBytes("WPF Response");
_listener.BeginSend(response, response.Length, receiveEndpoint, SendCallback, null);
}
private void SendCallback(IAsyncResult result)
{
int sendCount = _listener.EndSend(result);
Console.WriteLine("Sent Count is: " + sendCount);
}
}
The WP7 Server code:
public class MachineDetector
{
public const int UDP_PORT = 54322;
private const string MULTICAST_GROUP_ADDRESS = "224.0.0.1";
UdpAnySourceMulticastClient _client = null;
bool _joined = false;
private byte[] _receiveBuffer;
private const int MAX_MESSAGE_SIZE = 512;
public MachineDetector()
{
_client = new UdpAnySourceMulticastClient(IPAddress.Parse(MULTICAST_GROUP_ADDRESS), UDP_PORT);
_receiveBuffer = new byte[MAX_MESSAGE_SIZE];
_client.BeginJoinGroup(
result =>
{
_client.EndJoinGroup(result);
_client.MulticastLoopback = false;
SendRequest();
}, null);
}
private void SendRequest()
{
byte[] requestData = Encoding.UTF8.GetBytes("WP7 Multicast");
_client.BeginSendToGroup(requestData, 0, requestData.Length,
result =>
{
_client.EndSendToGroup(result);
Receive();
}, null);
}
private void Receive()
{
Array.Clear(_receiveBuffer, 0, _receiveBuffer.Length);
_client.BeginReceiveFromGroup(_receiveBuffer, 0, _receiveBuffer.Length,
result =>
{
IPEndPoint source;
_client.EndReceiveFromGroup(result, out source);
string dataReceived = Encoding.UTF8.GetString(_receiveBuffer, 0, _receiveBuffer.Length);
string message = String.Format("[{0}]: {1}", source.Address.ToString(), dataReceived);
Console.WriteLine(message);
Receive();
}, null);
}
}
I am able to receive data with the desktop client, but the WP7 app doesn't seem able to receive the response. I've been banging my head on this for a while now and don't know where else to look. Any help would be great.
So, any suggestions why the WP7 app isn't receiving a response?
I think the issue is with ConnectionListener:ReceiveCallback in C# Desktop Client.
_listener.BeginSend(response, response.Length,
receiveEndpoint, SendCallback, null);
Instead call
_listener.BeginSend(response, response.Length,
SendCallback, null);
This will cause the message to be sent to the multicast address. For further help with this refer to How to: Send and Receive Data in a Multicast Group for Windows Phone.
WP7 needs to multicast to the network in order to efficiently reach all desktop clients in one sweep. For the client response, the only intended destination is WP7. As such, a multicast has no real advantage here (as the desktop clients tuned into the multicast will effectively be ignoring it).
Instead, you could use receiveEndpoint populated by the call to EndReceive in ConnectionListener:ReceiveCallback to send a unicast response to the WP7 server. This is recommended consideration for creating multicast applications at MSDN.
This way WP7 no longer needs to join the multicast group for incoming multicast and the desktop client needn't send a multicast to respond.