We Keep Coding

Unable to receive known reply when broadcasting UDP datagrams over LAN using Sockets or UdpClient

I have searched for 2 days and found many, many questions/answers to what appears to be this same issue, with some differences, however none really seem to provide a solution.
I am implementing a library for controlling a DMX system (ColorKinetics devices) directly without an OEM controller. This involves communicating with an Ethernet-enabled power supply (PDS) connected to my home LAN, through a router, which drives the lighting fixtures. The PDS operates on a specific port (6038) and responds to properly formatted datagrams broadcast over the network.
I can successfully broadcast a simple DMX message (Header + DMX data), which gets picked up by the PDS and applied to connected lighting fixtures, so one-way communication is not an issue.
My issue is that I am now trying to implement a device discovery function to detect the PDS(s) and attached lights on the LAN, and I am not able to receive datagrams which are (absolutely) being sent back from the PDS. I can successfully transmit a datagram which instructs the devices to reply, and I can see the reply coming back in WireShark, but my application does not detect the reply.
I also tried running a simple listener app on another machine, which could detect the initial broadcast, but could not hear the return datagram either, however I figure this wouldn't work since the return packet is addressed to the original sender IP address.
I initially tried implementing via UdpClient, then via Sockets, and both produce the same result no matter what options and parameters I seem to specify.
Here is my current, very simple code to test functionality, currently using Sockets.
byte[] datagram = new CkPacket_DiscoverPDSRequestHeader().ToPacket();
Socket sender = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
IPEndPoint ep = new IPEndPoint(IPAddress.Parse("192.168.1.149"), 6039);
public Start()
{
// Start listener
new Thread(() =>
{
Receive();
}).Start();
sender.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
sender.EnableBroadcast = true;
// Bind the sender to known local IP and port 6039
sender.Bind(ep);
}
public void Send()
{
// Broadcast the datagram to port 6038
sender.SendTo(datagram, new IPEndPoint(IPAddress.Broadcast, 6038));
}
public void Receive()
{
Socket receiver = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
receiver.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
receiver.EnableBroadcast = true;
// Bind the receiver to known local IP and port 6039 (same as sender)
IPEndPoint EndPt = new IPEndPoint(IPAddress.Parse("192.168.1.149"),6039);
receiver.Bind(EndPt);
// Listen
while (true)
{
byte[] receivedData = new byte[256];
// Get the data
int rec = receiver.Receive(receivedData);
// Write to console the number of bytes received
Console.WriteLine($"Received {rec} bytes");
}
}
The sender and receiver are bound to an IPEndPoint with the local IP and port 6039. I did this because I could see that each time I initialized a new UdpClient, the system would dynamically assign an outgoing port, which the PDS would send data back to. Doing it this way, I can say that the listener is definitely listening on the port which should receive the PDS response (6039). I believe that since I have the option ReuseAddress set to true, this shouldn't be a problem (no exceptions thrown).
Start() creates a new thread to contain the listener, and initializes options on the sending client.
Send() successfully broadcasts the 16-byte datagram which is received by the PDS on port 6038, and generates a reply to port 6039 (Seen in WireShark)
Receive() does not receive the datagram. If I bind the listener to port 6038, it will receive the original 16-byte datagram broadcast.
Here is the WireShark data:
Wireshark
I have looked at using a library like SharpPCap, as many answers have suggested, but there appear to be some compatibility issues in the latest release that I am not smart enough to circumvent, which prevent the basic examples from functioning properly on my system. It also seems like this sort of basic functionality shouldn't require that type of external dependency. I've also seen many other questions/answers where the issue was similar, but it was solved by setting this-or-that parameter for the Socket or UdpClient, of which I have tried every combination to no avail.
I have also enabled access permissions through windows firewall, allowed port usage, and even completely disabled the firewall, to no success. I don't believe the issue would be with my router, since messages are getting to Wireshark.
UPDATE 1
Per suggestions, I believe I put the listener Socket in promiscuous mode as follows:
Socket receiver = new Socket(AddressFamily.InterNetwork, SocketType.Raw, ProtocolType.IP);
receiver.SetSocketOption(SocketOptionLevel.IP, SocketOptionName.HeaderIncluded, true);
receiver.EnableBroadcast = true;
IPEndPoint EndPt = new IPEndPoint(IPAddress.Parse("192.168.1.149"), 0);
receiver.Bind(EndPt);
receiver.IOControl(IOControlCode.ReceiveAll, new byte[] { 1, 0, 0, 0 }, null);
This resulted in the listener receiving all sorts of network traffic, including the outbound requests, but still no incoming reply.
UPDATE 2
As Viet suggested, there is some sort of addressing problem in the request datagram, which is formatted as such:
public class CkPacket_DiscoverPDSRequest : BytePacket
{
public uint magic = 0x0401dc4a;
public ushort version = 0x0100;
public ushort type = 0x0100;
public uint sequence = 0x00000000;
public uint command = 0xffffffff;
}
If I change the command field to my broadcast address 192.168.1.149' or192.168.255.255`, my listener begins detecting the return packets. I admittedly do not know what this field is supposed to represent, and my original guess was to just put in a broadcast address since the point of the datagram is to discover all devices on the network. This is obviously not the case, though I am still not sure the exact point of it.
Either way, thank you for the help, this is progress.

So in actuality it turns out that my issue was with the formatting of the outgoing datagram. The command field needs to be an address on the local subnet 192.168.xxx.xxx, and not 255.255.255.255... for whatever reason this was causing the packet to be filtered somewhere before getting to my application, though WireShark could still see it. This may be common sense in this type of work but being relatively ignorant as to network programming as well as the specifics of this interface it wasn't something I had considered.
Making the change allows a simple UdpClient send/receive to function perfectly.
Much thanks to Viet Hoang for helping me find this!

As you've already noted, you don't need to bind to send out a broadcast but it uses a random source port.
If you adjust your code to not bind the sender, your listener should behave as expected again:
Socket sender = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
sender.EnableBroadcast = true;
Thread read_thread;
public Start()
{
// Start listener
read_thread = new Thread(Receive);
read_thread.Start();
}
The issue you've bumped into is that the operating system kernel is only delivering packets up to one socket binder (first come first serve basis).
If you want true parallel read access, you'll need to look into sniffing example such as: https://stackoverflow.com/a/12437794/8408335.
Since you are only looking to source the broadcast from the same ip/port, you simply need to let the receiver bind first.
If you add in a short sleep after kicking off the receive thread, and before binding the sender, you will be able to see the expected results.
public Start()
{
// Start listener
new Thread(() =>
{
Receive();
}).Start();
Thread.Sleep(100);
sender.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
sender.EnableBroadcast = true;
// Bind the sender to known local IP and port 6039
sender.Bind(ep);
}
Extra note: You can quickly test your udp sockets from a linux box using netcat:
# echo "hello" | nc -q -1 -u 192.168.1.149 6039 -
- Edit -
Problem Part #2
The source address of "255.255.255.255" is invalid.
Did a quick test with two identical packets altering the source ip:
https://i.stack.imgur.com/BvWIa.jpg
Only the one with the valid source IP was printed to the console.
Received 26 bytes
Received 26 bytes

Handling multiple UDP sockets listening on the same endpoint

I've been following (roughly) some example code here on writing a DTLS server that can handle multiple clients. In this example, which works quite well (I tried), the server listens on INADDR_ANY and port 0.
fd = socket(server_addr.ss.ss_family, SOCK_DGRAM, 0);
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, (const void*) &on, (socklen_t) sizeof(on));
bind(fd, (const struct sockaddr *) &server_addr, sizeof(struct sockaddr_in))
When a DGRAM is received, the server runs it through an OpenSSL method to generate a cookie and verify the client is actually at the address they say they are. Once that's done, the server creates a new socket bound to the same endpoint and does connect() to the new client's endpoint.
bind(fd, (const struct sockaddr *) &pinfo->server_addr, sizeof(struct sockaddr_in))
connect(fd, (struct sockaddr *) &pinfo->client_addr, sizeof(struct sockaddr_in))
So at this point, there are 2 UDP sockets bound to the same IP address and port. One of them is connect()ed and the other is only bound.
I've tried to do the same thing (in C#):
_sock = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
_sock.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
_sock.Bind(localEp);
var clientEndpoint = await DtlsListenAsync(ssl, _sock);
var clientSock = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
clientSock.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
clientSock.Bind(_localEp);
clientSock.Connect(clientEndpoint);
Keep in mind, I've edited out most of the code for brevity.
At this point, running my application fails miserably. It seems the socket created for the client never gets the messages. Basically any call to await clientSock.ReceiveAsync() hangs indefinitely.
However, if I debug and step through the code works. This leads me to believe that its a "race" between the sockets. Basically, can the client socket be created, bound and connected before the next DGRAM comes in and gets sent to the original socket.
AFAIK, UDP sockets work on a score system. The kernel calculates a score based on how specific the rout is for a given DGRAM and sends it to the highest scoring socket. So, it seems to me that my server should work as well as the example... But it doesn't.
So I'm not really asking if my code is right per se, but I'd like to know if what I'm trying to do makes any sense. Also if there are any considerations I haven't taken into account. Do sockets behave as I mentioned?
My alternative if this method doesn't pan out is to use a single socket and maintain a hashtable of endpoints and dispatch the data to the appropriate client as it comes in. But TBH, I really like the idea of leveraging sockets to do the "heavy lifting" for me.

TCPClient is not closed, and instead waits for a client to close it

i have few questions about programming a TcpListener.
First problem:
Once client is connected using browser, i see the request. it is all ok. but then i face the problem with writing. and client recieving that data. it basically never gets a reply from server. do i need that flush function ? how does it work ? and is there any others ways of doing it ?
Porblem number 2 which is even more weird. when i call client.close() client doesnt go anywhere. it is still there. browser is stillw ating for data. and when i kill connection in the browser, only then Tcp client gets closed and loop starts again.
namespace TestServer
{
class Program
{
public static void Main()
{
TcpListener server;
IPAddress addr = IPAddress.Parse("127.0.0.1");
server = new TcpListener(addr, 80);
server.Start();
byte[] buffer = new byte[1024];
while(true)
{
string data = null;
Console.WriteLine("Awaiting for connections");
TcpClient client = server.AcceptTcpClient();
Console.WriteLine("Connected...");
NetworkStream str = client.GetStream();
int msgCounter;
while ((msgCounter = str.Read(buffer, 0, buffer.Length)) != 0)
{
Console.WriteLine("Processing stream...");
data += System.Text.Encoding.ASCII.GetString(buffer, 0, msgCounter);
Console.WriteLine("Reciaved: {0}", System.Text.Encoding.ASCII.GetString(buffer, 0, msgCounter));
}
byte[] response = System.Text.Encoding.ASCII.GetBytes("200 OK");
str.Write(response, 0, response.Length);
str.Flush();
client.Close();
buffer = new byte[1024];
}
}
}
}

TCP as a bi-directional transport layer protocol does not denote any concept of the "client is done sending request" signal.
What it means for developers is that such signaling must be defined in the application (or any other higher level) protocol layer.
In your case it is declared by HTTP itself in the https://tools.ietf.org/html//rfc2616#section-5
So if you intend to implement an HTTP server you must parse the HTTP request that has a determined way to identify the end of the request (see the link above).
To summarise: you need to know somehow you've read the request entirely and you may start processing it and generating/sending the response.

I'd recommend you to start with a million times proven working MSDN example of TcpListener class. Additionally I can point to explicit wrong approach:
Do not recreate buffer, it is a waste of resources.
Do not use browser as a test client if you are working with TCP sockets. Any browser tries to correct somehow HTTP protocol errors and can do it in very unpredictable way. If you need HTTP level of debugging, use Fiddler, for lower levels - Microsoft Network Monitor, Wireshark of Netcat.
It could be useful to read some book about TCP/IP networking. Particularly, you will know, that there is not "close" operation or command for TCP connection by protocol definition, TcpClient just emulate it. Instead a peer can send "shutdown" to another one, it does mean it doesn't plan to send data anymore, but can read it. Connection can be considered as closed only after both peers have sent their "shutdown" and received "shutdown" from each other.

Tcp connection Keep alive

i am creating a client server application. the server is already design and in place waiting for connection from the client. Now in the client section i would like to keep the connection alive throughout th life of the application and the connection only closes when the main client application close's or shutdown or the server closes it.
Currently every 10 seconds Server closes the TCP connection.I tried with
socket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.KeepAlive, true);
but it doesn't work for me..
Below is my code block
public TCPStreamDevice(string RemoteIPAddress, int RemotePort, string SourceIPAddress, int SourcePortNo)
{
mIpAddress = RemoteIPAddress;
mPort = RemotePort;
mClient = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
System.Net.IPEndPoint LocalEndPoint = new System.Net.IPEndPoint(System.Net.IPAddress.Parse(SourceIPAddress), SourcePortNo);
mClient.Bind(LocalEndPoint);
mDataReceivedCallback = new AsyncCallback(DataReceivedTCPCallback_Handler);
mBuffer = new byte[1024];
Description = new DeviceDescription();
}
and in the handler I have:
private void DataReceivedTCPCallback_Handler(IAsyncResult ar)
{
try
{
Socket client = (Socket)ar.AsyncState;
int bytesReceived = client.EndReceive(ar);
if (bytesReceived > 0)
{
//to know transport level errors
//EngineInterface.reponseReceived(mBuffer, false);
ReceiveCallBackFunc(mBuffer, bytesReceived);
client.BeginReceive(mBuffer, 0, 1024, SocketFlags.None, DataReceivedTCPCallback_Handler, client);
}
else
{
//disconnect
/* when there is no datapacket means no TCP connection is alive now (how can i keep Tcp alive here) */
}
}
}

In the call to SetSocketOption(), KeepAlive is not valid at the SocketOptionLevel.Tcp level, instead use SocketOptionLevel.Socket.
SetSocketOption( SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true );

The comments and answer above are valid - sounds like a bad design choice to have a socket opened for the entire lifetime of the app AND expect things to work properly - you should build some sort of failsafe mechanism in case the connection gets dropped.
Back to keep-alives: You need them on both ends - server and client so check how the sockets are set up on both sides. I think that the default value for keep alives is 2 hours - that's a long time to wait for a keep-alive packet but it can be changed. Check Socket.IOControl method and use IOControlCode.KeepAliveValues with a structure that looks like this (unmanaged) http://msdn.microsoft.com/en-us/library/ms741621.aspx. More about control codes here: http://msdn.microsoft.com/en-us/library/system.net.sockets.iocontrolcode.aspx

The comment ("whrn there is no datapacket means no TCP connection") in your code is placed where you receive a disconnect (0 bytes) packet from the other side. There is no way to keep that connection alive because the other side choses to close it.
If the connection is being closed due to network issues, you would either get an exception, or it would seem as if the connection is valid but quiet.
Keep-alive mechanisms always work alongside with timeouts - the timeout enforces "if no data was received for x seconds, close the connection" where the keep-alive simply sends a dummy data packet to keep the timeout from occurring.
By implementing a protocol yourself (you're operating on the TCP/IP level) you only need to implement a keep-alive if you already have a timeout implemented on the other side.

How do I find the port number assigned to a UDP client (in .net/C#)?

If I create a socket using
var socket = new UdpClient(0,AddressFamily.InterNetwork);
How do I then find the port of the socket?
I'm probably being daft, but I'm not having luck in MSDN/Google (probably because it is 4:42 on a Friday and the sun is shining).
Background:
What I want to do is find an open port, and then report to another process to forward messages to me on that port. There may be multiple clients, so I don't want to used a fixed port.
Thanks.

UdpClient is a wrapper around the Socket class which exposes the endpoint it's bound to through the LocalEndPoint property. Since you're using an UDP/IP client it's an IPEndPoint which has the desired Port property:
int port = ((IPEndPoint)socket.Client.LocalEndPoint).Port;

For those (like me) who need to use a RAW socket, here is the workaround.
goal:
to create a RAW UDP socket on an arbitrary Port
learn what port the system chose.
expected: (socket.LocalEndPoint as IPEndPoint).Port
problems
whereas a DGRAM UDP Socket knows its (socket.LocalEndPoint as IPEndPoint).Port
a RAW UDP Socket always returns zero
solution:
create a normal DGRAM UDP socket
bind that socket
find out its Port
close that normal socket
create the RAW UDP socket
CAVEAT:
Use the modified local IPEndPoint variable to know the port, as the socket will always report zero.
code:
public Socket CreateBoundRawUdpSocket(ref IPEndPoint local)
{
if (0 == local.port)
{
Socket wasted = new Socket(local.AddressFamily,
SocketType.Dgram,
ProtocolType.Udp);
wasted.Bind(local);
local.Port = (wasted.LocalEndPoint as IPEndPoint).Port;
wasted.Close();
}
Socket goal = new Socket(local.AddressFamily,
SocketType.Raw,
ProtocolType.Udp);
goal.Bind(local);
return goal;
}