We Keep Coding

TCP support in Azure IoT Hub

Azure IoT Hub Supports AMQP, MQTT, HTTP protocols. In order to customize these protocols we have Azure IoT protocol gateway. I can find good samples on MQTT protocol customization. I need some sample codes for TCP based protocol customization using Azure IoT Protocol Gateway.
EDIT (in order to get an answer): what the OP was asking, is an example using the Azure Protocol Gateway to support a proprietary TCP-based protocol. Currently the IoT hub only supports AMQP, MQTT and HTTP. While those protocols actually rely on TCP, the hub doesn't support direct TCP connection without the extra layer of AMQP, MQTT or HTTP.
As explained here, we need a basic example of a custom TCP based protocol.
Imagine a basic device that can only send some proprietary payload through TCP on a given IP address/port: we need an example of a gateway customization that allows this device to send data to the hub.
The current code of the protocol gateway is poorly designed, as it heavily relies on MQTT.
Adding some bounty too.

The default Protocol Gateway sample are indeed somewhat confusing because of all the MQTT code.
The protocol gateway works by 'simulating' a IoTHub connection for each custom protocol device you connect to the gateway.
To do this translation from the TCP device to an IoTHub device you first need to have a connection to the IoTHub on behalf of the device. This is the gateway part.
Below is the core essentials for this IoTHubConnection.
namespace GatewayTest
{
using System;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using DotNetty.Buffers;
using Microsoft.Azure.Devices.ProtocolGateway.Identity;
using Microsoft.Azure.Devices.ProtocolGateway.IotHubClient;
using Microsoft.Azure.Devices.ProtocolGateway.Messaging;
public class IoTHubConnection : IMessagingChannel<IMessage>
{
private readonly string iotHubHostName;
private readonly Func<IDeviceIdentity, Task<IMessagingServiceClient>> deviceClientFactory;
private readonly Func<string, Task> onMessage;
private IMessagingServiceClient deviceClient;
private IDeviceIdentity deviceIdentity;
public IoTHubConnection(
string iotHubHostName,
Func<IDeviceIdentity, Task<IMessagingServiceClient>> deviceClientFactory,
Func<string, Task> onMessage)
{
this.iotHubHostName = iotHubHostName;
this.deviceClientFactory = deviceClientFactory;
this.onMessage = onMessage;
}
public event EventHandler CapabilitiesChanged;
public async Task OpenAsync(string deviceId, string deviceKey)
{
this.deviceIdentity = this.GetDeviceIdentity(deviceId, deviceKey);
if (this.deviceIdentity != UnauthenticatedDeviceIdentity.Instance)
{
this.deviceClient = await this.deviceClientFactory(this.deviceIdentity);
this.deviceClient.BindMessagingChannel(this);
}
}
public async Task CloseAsync()
{
await this.deviceClient.DisposeAsync(null);
this.deviceClient = null;
}
public void Handle(IMessage message)
{
var messageBody = message.Payload.ToString(Encoding.UTF8);
this.onMessage(messageBody);
this.deviceClient.CompleteAsync(message.Id);
}
public Task SendMessage(string message)
{
var buffer = Unpooled.WrappedBuffer(Encoding.UTF8.GetBytes(message));
var deviceMessage = this.deviceClient.CreateMessage($"devices/{this.deviceIdentity.Id}/messages/events", buffer);
return this.deviceClient.SendAsync(deviceMessage);
}
protected virtual void OnCapabilitiesChanged(EventArgs e)
{
this.CapabilitiesChanged?.Invoke(this, e);
}
private IDeviceIdentity GetDeviceIdentity(string userName, string deviceKey)
{
IotHubDeviceIdentity ideviceIdentity;
if (!IotHubDeviceIdentity.TryParse($"{this.iotHubHostName}/{userName}", out ideviceIdentity))
{
return UnauthenticatedDeviceIdentity.Instance;
}
ideviceIdentity.WithDeviceKey(deviceKey);
return ideviceIdentity;
}
}
}
The deviceClientFactory callback method should be implemented as shown below and in this line in the ProtocolGateway repo in Github.
deviceClientFactory = IotHubClient.PreparePoolFactory(
"IotHubConnectionString",
400,
TimeSpan.FromMinutes(3),
iotHubClientSettings,
PooledByteBufferAllocator.Default,
new ConfigurableMessageAddressConverter("TopicNameConversion"));
When a Tcp Device connects to the protocol, you should create an instance of this IoTHubConnection and send messages from the Device to the IoTHubConnection and vica versa.
The code below shows a very simple version of how this should be done.
private const int BufferSize = 1024;
private byte[] buffer = new byte[BufferSize];
private IoTHubConnection ioTHubConnection;
private NetworkStream stream;
private async Task Start()
{
listener = new TcpListener(IPAddress.Any, port);
listener.Start();
var client = await listener.AcceptTcpClientAsync();
ioTHubConnection = new IoTHubConnection("IoTHubName", deviceClientFactory, OnIoTHubMessage);
stream = client.GetStream();
// Read DeviceId and DeviceKey from some sort of StartConnection-message send by the TcpClient.
await ioTHubConnection.OpenAsync("DeviceId", "DeviceKey");
stream.BeginRead(buffer, 0, BufferSize, ReadTcpStreamCallback, null);
}
private void ReadTcpStreamCallback(IAsyncResult ar)
{
var bytesRead = stream.EndRead(ar);
if (bytesRead > 0)
{
var message = System.Text.Encoding.ASCII.GetString(result);
ioTHubConnection.SendMessage(message);
// Read again.
stream.BeginRead(buffer, 0, BufferSize, ReadTcpStreamCallback, null);
}
}
private async Task OnIoTHubMessage(string message)
{
// Potentially do some translation on the IoTHub message
// and send it to the Device
var byteData = Encoding.UTF8.GetBytes(message);
stream.BeginWrite(byteData, 0, byteData.Length, SendTcpCallback, null);
}
private void SendTcpCallback(IAsyncResult ar)
{
stream.EndWrite(ar);
}

I know I am late to this conversation. However I have interesting add on or might be a solution for some.
Azure IoT Gateway is now known as Azure IoT Edge, this is clearly mentioned in the following Azure github repo
https://github.com/Azure/iot-edge-modbus.git
On the other hand, Azure IoT Edge supports TCP for some protocols which can be found in the following links
https://learn.microsoft.com/en-us/azure/iot-edge/deploy-modbus-gateway
https://learn.microsoft.com/en-us/azure/iot-edge/iot-edge-as-gateway

Multiple applications listening on the same UDP port: broadcast IP VS non-broadcast IP wrt which gets the message

Apps A and B are both listening to and sending from UDP port 1337 on the same computer.
App A was started first (so it bound to port 1337 first).
When they each send a message to 127.255.255.255:1337 or 255.255.255.255:1337, both apps A and B get the each others' messages, as well as their own echo.
When they each send a message to 127.0.0.1:1337 or 192.168.1.65:1337, a non-broadcast address, only A gets B's message and its own echo. B cannot even hear its own message. (192.168.1.65 is my computer's network IP where both apps are running)
If I stop app A, B can hear its own message. If I start A again, B can hear both messages and A can head nothing.
Why do messages get duplicated when sent to a broadcast IP, whereas they are distributed on a first-come-first-serve basis if they are sent to a non-broadcast address? Is there a why to get a consistent behavior (set some "always-duplicate" flag)?
Here is some C# code that demonstrates this:
public class Program
{
public static void Main()
{
var udp = new Udp("255.255.255.255", 1337);
Task.Run(() =>
{
while (true)
{
Console.WriteLine(udp.Receive());
}
});
Task.Run(() =>
{
while (true)
{
Thread.Sleep(1000);
udp.Send("(((1)))");
}
});
Console.ReadLine();
}
}
public class Udp
{
private readonly UdpClient _sender;
private readonly UdpClient _listener;
public Udp(string address, int port)
{
_sender = new UdpClient(address, port);
_listener = new UdpClient();
_listener.Client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
_listener.Client.Bind(new IPEndPoint(IPAddress.Any, port));
}
public string Receive()
{
var _ = null as IPEndPoint;
return $"{Encoding.Default.GetString(_listener.Receive(ref _))} from {_.Address}:{_.Port}";
}
public void Send(string message)
{
var dataAsBytes = Encoding.ASCII.GetBytes(message);
_sender.Send(dataAsBytes, dataAsBytes.Length);
}
}
You can run this app twice, change the number in udp.Send("(((1)))") to identify each. When you use new Udp("255.255.255.255", 1337) both instances can hear both messages, whereas using new Udp("127.0.0.1", 1337) causes the first instance only to be able to pick up messages. Thank you!

UDP in C# works on Windows but not Linux

I'm using mono to build a C# program that needs to send and receive using UDP. Currently my implementation works as expected on Windows but I have issues getting communication to work with my Ubuntu or Fedora systems.
Windows can broadcast and receive it's own datagrams.
Ubuntu can broadcast and receive it's own datagrams. It's broadcasts are received by Windows but it doesn't see datagrams broadcast by Windows.
Fedora can broadcast but does not receive datagrams from anywhere (not even itself). It's broadcasts are received by Windows.
When datagrams fail to reach either of the linux machines, the 'receive' function is never fired.
This is what I have so far:
int _port = 4568;
var server = new UdpClient(_port);
var send_UDP = new UdpClient();
The receive method uses the asynchronous calls of the UDPClient;
private static void receive()
{
server.BeginReceive(new AsyncCallback(receive), null);
}
private static void receive(IAsyncResult o)
{
try
{
// I'm told that port = 0 should receive from any port.
var sender = new IPEndPoint(IPAddress.Any, 0);
var data = server.EndReceive(o, ref sender);
receive();
var str = new string(Encoding.ASCII.GetChars(data));
postmessage(sender.Address.ToString() + ":" + sender.Port.ToString() + " > " + str);
}
catch {}
}
And the send method;
public static void send(string message)
{
var target = new IPEndPoint(IPAddress.Parse("255.255.255.255"), _port);
byte[] data = Encoding.ASCII.GetBytes(message);
send_UDP.Send(data, data.Length, target);
}
After some testing with Fedora, it seems to be an issue with the use of 255.255.255.255 to broadcast. Is there some other way to do this?

I already specified this in a comment but placing this as an answer since you may have overlooked it and no answers seem to be forthcoming.
Instead of using 255.255.255.255 for broadcast use your local IP subnet's broadcasting address (for instance 192.168.0.255 on a 192.168.0.1/24 subnet). The 255.255.255.255address will not be forwarded by a router (this is relevant if there are multiple subnets at your clients' sites) whereas a directed broadcast can be forwarded (if so configured). It used to be the case that routers would forward directed broadcasts per default but this was changed in RFC2644 so don't bet the farm on it ;).
Here's an example of calculating the directed IPV4 broadcast address per adapter:
public static void DisplayDirectedBroadcastAddresses()
{
foreach (var iface in NetworkInterface.GetAllNetworkInterfaces()
.Where(c => c.NetworkInterfaceType != NetworkInterfaceType.Loopback))
{
Console.WriteLine(iface.Description);
foreach (var ucastInfo in iface.GetIPProperties().UnicastAddresses
.Where(c => !c.Address.IsIPv6LinkLocal))
{
Console.WriteLine("\tIP : {0}", ucastInfo.Address);
Console.WriteLine("\tSubnet : {0}", ucastInfo.IPv4Mask);
byte[] ipAdressBytes = ucastInfo.Address.GetAddressBytes();
byte[] subnetMaskBytes = ucastInfo.IPv4Mask.GetAddressBytes();
if (ipAdressBytes.Length != subnetMaskBytes.Length) continue;
var broadcast = new byte[ipAdressBytes.Length];
for (int i = 0; i < broadcast.Length; i++)
{
broadcast[i] = (byte)(ipAdressBytes[i] | ~(subnetMaskBytes[i]));
}
Console.WriteLine("\tBroadcast: {0}", new IPAddress(broadcast).ToString());
}
}
}

Can't make TCP client code work without USB cable attached

I'm having a bit of trouble in getting a very simple TCP client working on my HTC Titan w/ Windows Phone 7.5.
When the USB cable is attached to the phone, the TCP client works like a charm, but as soon as the cable is unplugged, the client is unable to connect to a TCP server running on my development machine. The devices are on the same network and I'm using the explicit IP-address of my desktop machine to connect, so there's no name resolution going on afaik.
Here's the code I use. Most of it was taken from the Sockets samples on MSDN (can't seem to find the link now though).
private Socket _sock = null;
private ManualResetEvent _done = new ManualResetEvent(false);
private const int TIMEOUT = 5000;
//connect to server
public string Connect(string ip, int port) {
string result = string.Empty;
var host = new IPEndpoint(IPAddress.Parse(ip), port);
_sock = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
_sock.SetNetworkRequirement(NetworkSelectionCharacteristics.NonCellular);
var args = new SocketAsyncEventArgs();
args.RemoteEndPoint = host;
args.Completed += new EventHandler((s,e) => {
result = e.SocketError.ToString();
_done.Set();
});
_done.Reset();
_sock.ConnectAsync(args);
_done.WaitOne(TIMEOUT);
return result;
}
//send message
public string Send(string msg) {
string response = "Operation timeout";
if (_sock != null) {
var args= new SocketAsyncEventArgs();
args.RemoteEndPoint = _sock.RemoteEndPoint;
args.Completed += new EventHandler(s, e) => {
response = e.SocketError.ToString();
_done.Set();
});
var payload = Encoding.UTF8.GetBytes(data);
args.SetBuffer(payload, 0, payload.Length);
_done.Reset();
_sock.SendAsync(args);
_done.WaitOne(TIMEOUT);
}
return response;
}
//receive message
public string Receive() {
string response = "Operation timeout";
if (_sock != null) {
var args= new SocketAsyncEventArgs();
args.RemoteEndPoint = _sock.RemoteEndPoint;
args.SetBuffer(new Byte[MAX_BUFSIZE], 0, MAX_BUFSIZE);
args.Completed += new EventHandler((s,e) => {
if (e.SocketError == SocketError.Success) {
response = Encoding.UTF8.GetString(e.Buffer, e.Offset, e.BytesTransferred);
response = response.Trim('\0');
}
else {
response = e.SocketError.ToString();
}
_done.Set();
});
_done.Reset();
_sock.ReceiveAsync(args);
_done.WaitOne(TIMEOUT);
}
return response;
}
The code is then simply used like:
Connect(...);
Send(...);
Receive(...);
//and then close the socket
As I said before, the code works like a charm when the device is attached to my development machine. When the cable is unplugged, the connection phase just times out (regardless of the timeout interval I should say).
Also, the manifest contains the ID_CAP_NETWORKING capability which as I understand it should give the app permission to access the network.
Any ideas?
EDIT:
I discovered that switching to UDP communication works like a charm. Which means that the problem is that for some reason, the phone is unable to set up a persistant TCP connection to my dev machine. This is getting stranger by the minute.

Do you have a wireless ap nearby on which your phone is connected? because when you plug it in the pc it uses the pc's network connection.

You should check the IP address that you have on both the phone (from your code) and on the PC (which it looks like you've already found using ipconfig in your command prompt).
These should be in the same IP address range, and so start with the same digits (for IPv4, probably something link 192.168.0.*).
If this all matches up check your wireless router hasn't enabled a security setting which means that it doesn't allow TCP traffic from your phone to your PC.
If this is a consumer router you manage this should be fairly simple to verify (and potentially fix). If not, you're probably stuck...

Receive messages continuously using udpClient

I was looking for the best solution to receive and process messages via UdpClient class in C#. Does anyone have any solutions for this?

Try this code :
//Client uses as receive udp client
UdpClient Client = new UdpClient(Port);
try
{
Client.BeginReceive(new AsyncCallback(recv), null);
}
catch(Exception e)
{
MessageBox.Show(e.ToString());
}
//CallBack
private void recv(IAsyncResult res)
{
IPEndPoint RemoteIpEndPoint = new IPEndPoint(IPAddress.Any, 8000);
byte[] received = Client.EndReceive(res, ref RemoteIpEndPoint);
//Process codes
MessageBox.Show(Encoding.UTF8.GetString(received));
Client.BeginReceive(new AsyncCallback(recv), null);
}

For newer methods using TAP instead of Begin/End method you can use the following in .Net 4.5
Quite simple!
Asynchronous Method
private static void UDPListener()
{
Task.Run(async () =>
{
using (var udpClient = new UdpClient(11000))
{
string loggingEvent = "";
while (true)
{
//IPEndPoint object will allow us to read datagrams sent from any source.
var receivedResults = await udpClient.ReceiveAsync();
loggingEvent += Encoding.ASCII.GetString(receivedResults.Buffer);
}
}
});
}
Synchronous Method
As appose to the asynchronous method above, this can be also implemented in synchronous method in a very similar fashion:
private static void UDPListener()
{
Task.Run(() =>
{
using (var udpClient = new UdpClient(11000))
{
string loggingEvent = "";
while (true)
{
//IPEndPoint object will allow us to read datagrams sent from any source.
var remoteEndPoint = new IPEndPoint(IPAddress.Any, 0);
var receivedResults = udpClient.Receive(ref remoteEndPoint);
loggingEvent += Encoding.ASCII.GetString(receivedResults);
}
}
});
}

I can recommend two links about this solution which were helpful for me.
PC Related
Stack Overflow
The first one is really easy solution but be careful at modifying because the continual receiving will work only if some UDP packet is sent as first to the "remote" device.
For continual listening add the code line "udp.BeginReceive(new AsyncCallback(UDP_IncomingData), udp_ep);" after each reading of incomming data to enable new receiving of UDP packets.
The second one is nice solution for using the Multicast IP-addresses (239.255.255.255 - 240.0.0.0)