We Keep Coding

Unix domain socket: A socket operation encountered a dead network

I'm using Unix Domain Socket for the first time,
I have created the following console application using .NET 5.0 to learn about Unix Domain Sockets.
I found the code here (https://medium.com/codex/unix-domain-sockets-in-net-6-basics-and-real-world-examples-8982898ab293)
My expectation is that similar to TCP/IP socket connection, i will be able to talk between the client and server but using Unix Domain sockets instead.
Following is the code:
class Program
{
static void Main(string[] args)
{
const string Hostname = "localhost";
const string UnixSocketPath = "/tmp/foo.sock";
using var socket = new Socket(AddressFamily.Unix, SocketType.Stream, ProtocolType.IP);
var endpoint = new UnixDomainSocketEndPoint(UnixSocketPath);
socket.Connect(endpoint);
var requestBytes = System.Text.Encoding.UTF8.GetBytes($"GET / HTTP/1.0\r\nHost: {Hostname}\r\nAccept: */*\r\n\r\n");
socket.Send(requestBytes);
byte[] receivedBytes = new byte[1024];
socket.Receive(receivedBytes, 1024, SocketFlags.None);
Console.WriteLine(System.Text.Encoding.UTF8.GetString(receivedBytes));
}
}
I get the following error when i try to run my application:
"A socket operation encountered a dead network" at socket.Connect(endpoint) line.
I have the following questions:
Is it expected for the above console application to run in a windows system?
After reading couple of articles i understand that the "UnixSocketPath" is the file based path which is followed in a Unix machine. What does it refer to in case of a Windows machine? Or is it just a temporary file created in the output directory of the application.
I tried the following to resolve the error "A socket operation encountered a dead network" after reading up a bit on this:
Try to run the application in admin mode - does not work, same error
Switch to .net core 3.0 (even though i was not facing any compile time errors with .net 5.0)
does not work, same error.
Tried to change the "UnixSocketPath" to a local path (Eg: C:/Test/foo.sock ) - does not work, error : "No connection could be made because the target machine actively refused it. C:/Test/foo.sock"
Excuse me for these noob level questions, the entire socket level programming is new to me and i'm quite confused now .
Any suggestions are appreciated. Thanks

From your posted link:
To run the example on Windows-based systems, I can recommend WSL 2. Visual Studio and Visual Studio Code come with excellent remote integrations so you can run the code directly from the development environment.
WSL is "Windows subsystems for linux", i.e. a linux virtual machine that has some integrations with the windows host.
So you would need to run your program in this WSL environment, and have this configured correctly. See Debug .NET Apps in WSL with Visual Studio
However, if you just want your server and client to talk to each other I would consider if this really is the right approach. While unix domain sockets might have some performance advantages, you require both applications to be run on the same computer. I would just use some message based protocol and instead try to minimize the number and size of messages.

gRPC connection gets cut after 60 seconds if inactivity

I have been trying to set up a gRPC API capable of streaming events to a client. Basically, after a client has subscribed, the server will use gRPC's "Server Streaming" feature to send any new event to the client.
I expect there to be periods of inactivity, where the connection should remain active. However, with my current setup it seems Nginx is cutting the connection after 60 seconds of inactivity with the following exception at the client:
Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: The request was aborted. IOException: The response ended prematurely, with at least 9 additional bytes expected.", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The request was aborted.
---> System.IO.IOException: The response ended prematurely, with at least 9 additional bytes expected.
The question is why? and how can I prevent it?
My setup
The API is built in ASP.NET Core 3 (will probably upgrade to .NET 5 soon) and is running in a Docker container on a Digital Ocean server.
Nginx is also running in a Docker container on the server and works as a reverse proxy for the API (among other things).
The client is a simple C# client written in .NET Core and is run locally.
What have I tried?
I have tried to connect to the Docker image directly on the server using grpc_cli (bypassing Nginx) where the connection remain active for long periods of inactivity without any issues. So I can't see what else it can be, except Nginx. Also, most of Nginx default timeout values seem to be 60 seconds.
I have tried these Nginx settings and various combinations of them, yet haven't found the right one (or the right combination) yet:
location /commands.CommandService/ {
grpc_pass grpc://commandApi;
grpc_socket_keepalive on;
grpc_read_timeout 3000s; # These are recommended everywhere, but I haven't had any success
grpc_send_timeout 3000s; #
grpc_next_upstream_timeout 0;
proxy_request_buffering off;
proxy_buffering off;
proxy_connect_timeout 3000s;
proxy_send_timeout 3000s;
proxy_read_timeout 3000s;
proxy_socket_keepalive on;
keepalive_timeout 90s;
send_timeout 90s;
client_body_timeout 3000s;
}
The most common suggestion for people with similar issues is to use grpc_read_timeout and grpc_send_timeout, but they don't work for me. I guess it makes sense since I'm not actively sending/receiving anything.
My client code looks like this:
var httpClientHandler = new HttpClientHandler();
var channel = GrpcChannel.ForAddress("https://myapi.com", new GrpcChannelOptions()
{
HttpClient = new HttpClient(httpClientHandler) { Timeout = Timeout.InfiniteTimeSpan },
});
var commandService = channel.CreateGrpcService<ICommandService>();
var request = new CommandSubscriptionRequest()
{
HandlerId = _handlerId
};
var sd = new CancellationTokenSource();
var r = new CallContext(callOptions: new CallOptions(deadline: null, cancellationToken: sd.Token));
await foreach (var command in commandService.SubscribeCommandsAsync(request, r))
{
Console.WriteLine("Processing command: " + command.Id);
}
return channel;
To be clear, the call to the API works and I can receive commands from the server. If I just keep sending commands from the API, everything is working beautifully. But as soon as I stop for 60 seconds (I have timed it), the connection breaks.
A possible workaround would be to just keep sending a kind of heartbeat to keep the connection open, but I would prefer not to.
Does anyone know how I can fix it? Am I missing something obvious?
UPDATE: Turns out it wasn't Nginx. After I updated the API and the client to .NET 5 the problem disappeared. I can't say in what version this was fixed, but at least it's gone in .NET 5.

Not sure this is an Nginx issue, looks like a client connection problem.
Your results look very similar to an issue I had, that should have been fixed in .net 3.0 patch. Try updating to a newer version of .NET and see if that fixes the problem.
Alternatively, it could be a problem with the max number of connections. Try setting the MaxConcurrentConnections for the kestrel server (in appsettings.json):
{
"Kestrel": {
"Limits": {
"MaxConcurrentConnections": 100,
"MaxConcurrentUpgradedConnections": 100
}
}
}

How to detect a network was broken down with ZeroMQ monitor mechanism?

A. Description
I am using ZeroMQ monitor and I find that it works when logic disconnection but does not work when network broken down (unplug my cable).
For example:
I launch client app on an android pad, then launch a server app on my windows laptop. They are connected through a router with cables.
Everything will be OK with monitor if I close or open either client app or server app manually. Namely, the monitor on both sides can receive a 'Connect' or an 'Accept' and a 'Disconnect' event.
But If I unplug cable on the server side, while the client and server are connected and running, the monitors on both sides can not detect the 'Disconnect' event.
Is the monitor designed like this?
If so, are there any solutions to detect network broken down ( a cable unplug event ) except heartbeats?
If not, how to use the ZeroMQ's original monitor mechanism to solve this problem? Can a setTCPKeepAlive() interface be useful?
B. System environment
My scenario is as below.
Client
OS: Android, running on a pad, IDE: Android studio 2.3, lib:jeromq-0.4.3
// Java Code
String monitorAddr = "inproc://client.req";
ZContext ctx = new ZContext();
ZMQ.Socket clientSocket = ctx.createSocket(ZMQ.REQ);
clientSocket.monitor(monitorAddr,ZMQ.EVENT_ALL);
// Then start a montitor thread which is implemented by my own.
Server
OS: Windows 7 ( 64 bit ), running on my laptop, IDE: VS2013, lib: Clrzmq4
// C# Code
const string MonitorEndpoint = "inproc://server.rep";
var ctx = new ZContext();
var serverSocket = new ZSocket(ctx,ZSocketType.REP);
ZError error;
// Create serverSocket pair socket
if (!serverSocket.Monitor(MonitorEndpoint, ZMonitorEvents.AllEvents, out error))
{
if (error == ZError.ETERM)
return ; // Interrupted
throw new ZException(error);
}
// Create a monitor
ZMonitor _monitor = ZMonitor.Create(ctx, MonitorEndpoint);
_monitor.AllEvents += _monitor_AllEvents;
_monitor.Start();

AFAIK there is no built in heartbeat within ZeroMQ. I know there was some discussion on the topic within the ZMQ community some years ago, and that discussion may still be going on.
It is comparatively simple to incorporate your own heartbeat messaging in your application's use of ZeroMQ, especially if you use something like Google Protocol Buffers to encode different message types; the heartbeat is just another message.
Doing heartbeats in your application (rather than relying on some inbuilt mechanism) is ultimately more flexible; you can choose the heartbeat rate, you can choose what to do if the heartbeat fails, you can decide when heartbeating is important and not important, etc.
Consider heartbeats within a PUB/SUB pattern; it's a bit difficult for the ZMQ authors to decide on your behalf what connection / disconnection / connection-break events matter to you. And if they do build in a mechanism, but an application developer didn't want it, then it is a waste of bandwidth.
It's far easier for the ZMQ authors to leave that kind of application architectural issue to the application author (that's you!) to deal with.
With your specific example, an unplugged network cable simply looks (so far as any software can determine) like no traffic is flowing; it's the same as the application not sending anything. ZMQ doesn't send anything if the application hasn't sent anything.
If you look at the events that the socket monitor can report on, they're all the consequence of something flowing over the network connection, or something done to the socket by the application.

Trying to go lower than ZMQ protocol itself and access the TCP connection that specific ZeroMQ sockets use (while others do not) doesn't sound like a good idea; it would required to break encapsulation in multiple classes.
The answer #bazza gave in 2017 was entirely correct at the time.
However, newer versions of ZMQ (specifically ZMTP) include an heartbeat functionality.
Check ZMQ documentation for
socketOpt
Java functions
name
purpose
ZMQ_HEARTBEAT_IVL
get/setHeartbeatLvl()
heartbeat interval
milliseconds between ZMPT PINGs
ZMQ_HEARTBEAT_TIMEOUT
get/setHeartbeatTimeout()
local heartbeat timeout
how long the local socket waits between received packets until it considers the connection timed out
ZMQ_HEARTBEAT_TTL
get/setHeartbeatTtl()
remote heartbeat timeout
if and when remote side shall consider the connection timed out
ZMQ_HEARTBEAT_CONTEXT is still in draft state as of 2022. It is supposed to send an byte[] context with every ping.
Now, by design of ZMQ, quoting from chapter 2 of its documentation,
The network connection itself happens in the background, and ZeroMQ
will automatically reconnect if the network connection is broken
(e.g., if the peer disappears and then comes back).
Thus, answering your main question, I'd expect the monitor to give you ZMQ_EVENT_CONNECT_RETRIED / ZMQ_EVENT_CONNECTED events after the underlying connection was detected as disrupted.

C#-Server / C++ Client: Socket.Accept() never finishes

I'm writing a simple C++ client that should connect to a C# server. And, while the client runs throught fine and even states it sent a bunch of bites, the C# server never gets past listener.Acceppt():
Console.WriteLine("Waiting for a connection...");
// Program is suspended while waiting for an incoming connection.
Socket handler = listener.Accept();
I took the server source from here: Microsoft MSDN: Synchronous Server Socket Example
The client code I use can be found here: Microsoft MSDN: Complete Winsock Client Code
I also checked to server with the according C# client, which worked fine. I also checked the return values of the client, which also looked pretty plausible. So I'd guess the problem lies somewhere in how C++ / C# handle the details.
I tried Wireshark to find some hints, but mysteriously there was absolutely no trace of any communication on the used port (11000).
Where should I start looking to solve this issue?
Update: All communication at the moment runs just locally. I tried my IPAddress (from ipconfig) and 127.0.0.1 for both server and client.
I just have one network adapter installed, I use that IP adress. The client checks the getaddrinfo(argv[1], DEFAULT_PORT, &hints, &result) and returns valid information (e.g. TCP as protocol). connect() returns 0, which should be ok.
Update 2:
I tried different combinations of C++/C# Server-Client-Setups:
C++-Server, C++ Client: Works
C++-Server, C# Client: Works
C#-Server, C# Client: Works
C#-Server, C++ Client: Does not work
C#-Server, putty: Works
Update 3: I tried Wireshark on the other Server-Client constellations, but neither of them did show any traffic on tcp.port == 11000, although they did work (see Update 2). It looks like Wireshark does not show any results, because everything is just local (see Wireshark localhost traffic capture)

I had the same problem.
The connection with winsock is asynchronous. And the example in C# that you are using is synchronous.
You have to use a Asynchronous example to get communication with your C++ code.
I used this two examples!
C Sharp server
https://msdn.microsoft.com/en-us/library/fx6588te(v=vs.110).aspx
C plus plus client
https://msdn.microsoft.com/en-us/library/windows/desktop/ms737591(v=vs.85).aspx

An existing connection was forcibly closed by the remote host

I am working with a commercial application which is throwing a SocketException with the message,
An existing connection was forcibly closed by the remote host
This happens with a socket connection between client and server. The connection is alive and well, and heaps of data is being transferred, but it then becomes disconnected out of nowhere.
Has anybody seen this before? What could the causes be? I can kind of guess a few causes, but also is there any way to add more into this code to work out what the cause could be?
Any comments / ideas are welcome.
... The latest ...
I have some logging from some .NET tracing,
System.Net.Sockets Verbose: 0 : [8188] Socket#30180123::Send() DateTime=2010-04-07T20:49:48.6317500Z
System.Net.Sockets Error: 0 : [8188] Exception in the Socket#30180123::Send - An existing connection was forcibly closed by the remote host DateTime=2010-04-07T20:49:48.6317500Z
System.Net.Sockets Verbose: 0 : [8188] Exiting Socket#30180123::Send() -> 0#0
Based on other parts of the logging I have seen the fact that it says 0#0 means a packet of 0 bytes length is being sent. But what does that really mean?
One of two possibilities is occurring, and I am not sure which,
The connection is being closed, but data is then being written to the socket, thus creating the exception above. The 0#0 simply means that nothing was sent because the socket was already closed.
The connection is still open, and a packet of zero bytes is being sent (i.e. the code has a bug) and the 0#0 means that a packet of zero bytes is trying to be sent.
What do you reckon? It might be inconclusive I guess, but perhaps someone else has seen this kind of thing?

This generally means that the remote side closed the connection (usually by sending a TCP/IP RST packet). If you're working with a third-party application, the likely causes are:
You are sending malformed data to the application (which could include sending an HTTPS request to an HTTP server)
The network link between the client and server is going down for some reason
You have triggered a bug in the third-party application that caused it to crash
The third-party application has exhausted system resources
It's likely that the first case is what's happening.
You can fire up Wireshark to see exactly what is happening on the wire to narrow down the problem.
Without more specific information, it's unlikely that anyone here can really help you much.

Using TLS 1.2 solved this error.
You can force your application using TLS 1.2 with this (make sure to execute it before calling your service):
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
Another solution :
Enable strong cryptography in your local machine or server in order to use TLS1.2 because by default it is disabled so only TLS1.0 is used.
To enable strong cryptography , execute these commande in PowerShell with admin privileges :
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Wow6432Node\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type DWord
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type DWord
You need to reboot your computer for these changes to take effect.

This is not a bug in your code. It is coming from .Net's Socket implementation. If you use the overloaded implementation of EndReceive as below you will not get this exception.
SocketError errorCode;
int nBytesRec = socket.EndReceive(ar, out errorCode);
if (errorCode != SocketError.Success)
{
nBytesRec = 0;
}

Had the same bug. Actually worked in case the traffic was sent using some proxy (fiddler in my case). Updated .NET framework from 4.5.2 to >=4.6 and now everything works fine. The actual request was:
new WebClient().DownloadData("URL");
The exception was:
SocketException: An existing connection was forcibly closed by the
remote host

Simple solution for this common annoying issue:
Just go to your ".context.cs" file (located under ".context.tt" which located under your "*.edmx" file).
Then, add this line to your constructor:
public DBEntities()
: base("name=DBEntities")
{
this.Configuration.ProxyCreationEnabled = false; // ADD THIS LINE!
}

I've got this exception because of circular reference in entity.In entity that look like
public class Catalog
{
public int Id { get; set; }
public int ParentId { get; set; }
public Catalog Parent { get; set; }
public ICollection<Catalog> ChildCatalogs { get; set; }
}
I added [IgnoreDataMemberAttribute] to the Parent property. And that solved the problem.

If Running In A .Net 4.5.2 Service
For me the issue was compounded because the call was running in a .Net 4.5.2 service. I followed #willmaz suggestion but got a new error.
In running the service with logging turned on, I viewed the handshaking with the target site would initiate ok (and send the bearer token) but on the following step to process the Post call, it would seem to drop the auth token and the site would reply with Unauthorized.
Solution
It turned out that the service pool credentials did not have rights to change TLS (?) and when I put in my local admin account into the pool, it all worked.

I had the same issue and managed to resolve it eventually. In my case, the port that the client sends the request to did not have a SSL cert binding to it. So I fixed the issue by binding a SSL cert to the port on the server side. Once that was done, this exception went away.

For anyone getting this exception while reading data from the stream, this may help. I was getting this exception when reading the HttpResponseMessage in a loop like this:
using (var remoteStream = await response.Content.ReadAsStreamAsync())
using (var content = File.Create(DownloadPath))
{
var buffer = new byte[1024];
int read;
while ((read = await remoteStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
{
await content.WriteAsync(buffer, 0, read);
await content.FlushAsync();
}
}
After some time I found out the culprit was the buffer size, which was too small and didn't play well with my weak Azure instance. What helped was to change the code to:
using (Stream remoteStream = await response.Content.ReadAsStreamAsync())
using (FileStream content = File.Create(DownloadPath))
{
await remoteStream.CopyToAsync(content);
}
CopyTo() method has a default buffer size of 81920. The bigger buffer sped up the process and the errors stopped immediately, most likely because the overall download speeds increased. But why would download speed matter in preventing this error?
It is possible that you get disconnected from the server because the download speeds drop below minimum threshold the server is configured to allow. For example, in case the application you are downloading the file from is hosted on IIS, it can be a problem with http.sys configuration:
"Http.sys is the http protocol stack that IIS uses to perform http communication with clients. It has a timer called MinBytesPerSecond that is responsible for killing a connection if its transfer rate drops below some kb/sec threshold. By default, that threshold is set to 240 kb/sec."
The issue is described in this old blogpost from TFS development team and concerns IIS specifically, but may point you in a right direction. It also mentions an old bug related to this http.sys attribute: link
In case you are using Azure app services and increasing the buffer size does not eliminate the problem, try to scale up your machine as well. You will be allocated more resources including connection bandwidth.

I got the same issue while using .NET Framework 4.5. However, when I update the .NET version to 4.7.2 connection issue was resolved. Maybe this is due to SecurityProtocol support issue.

For me, it was because the app server I was trying to send email from was not added to our company's SMTP server's allowed list.
I just had to put in SMTP access request for that app server.
This is how it was added by the infrastructure team (I don't know how to do these steps myself but this is what they said they did):
1. Log into active L.B.
2. Select: Local Traffic > iRules > Data Group List
3. Select the appropriate Data Group
4. Enter the app server's IP address
5. Select: Add
6. Select: Update
7. Sync config changes

Yet another possibility for this error to occur is if you tried to connect to a third-party server with invalid credentials too many times and a system like Fail2ban is blocking your IP address.

I was trying to connect to the MQTT broker using the GO client,
broker address was given as address + port, or tcp://address:port
Example: ❌
mqtt://test.mosquitto.org
which indicates that you wish to establish an unencrypted connection.
To request MQTT over TLS use one of ssl, tls, mqtts, mqtt+ssl or tcps.
Example: ✅
mqtts://test.mosquitto.org

In my case, enable the IIS server & then restart and check again.

We are using a SpringBoot service. Our restTemplate code looks like below:
#Bean
public RestTemplate restTemplate(final RestTemplateBuilder builder) {
return builder.requestFactory(() -> {
final ConnectionPool okHttpConnectionPool =
new ConnectionPool(50, 30, TimeUnit.SECONDS);
final OkHttpClient okHttpClient =
new OkHttpClient.Builder().connectionPool(okHttpConnectionPool)
// .connectTimeout(30, TimeUnit.SECONDS)
.retryOnConnectionFailure(false).build();
return new OkHttp3ClientHttpRequestFactory(okHttpClient);
}).build();
}
All our call were failing after the ReadTimeout set for the restTemplate. We increased the time, and our issue was resolved.

This error occurred in my application with the CIP-protocol whenever I didn't Send or received data in less than 10s.
This was caused by the use of the forward open method. You can avoid this by working with an other method, or to install an update rate of less the 10s that maintain your forward-open-connection.