I have a simple pub-sub setup on a mid-sized network, using ZMQ 2.1. Although some subscribers are using C# bindings, others are using Python bindings, and the issue I'm having is the same for either.
If I pull the network cable from a machine running a subscriber, I get an un-catchable error that immediately terminates that subscriber.
Here's a very simple example of a subscriber in Python (not actual production code, but enough to reproduce the problem):
import zmq
def main(server_address, port):
context = zmq.Context()
sub_socket = context.socket(zmq.SUB)
sub_socket.connect("tcp://" + server_address + ":" + str(port))
sub_socket.setsockopt(zmq.SUBSCRIBE, "KITH1S2")
while True:
msg = sub_socket.recv()
print msg
if __name__ == "__main__": main("company-intranet", 4000)
In C# the program simply terminates silently. In Python I at least get this:
Assertion failed: rc == 0 (....\src\zmq_connector.cpp:48)
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
I've tried non-blocking versions, and poller versions, but in either case this instant termination problem persists. Is there something obvious I should be doing but I'm not? (That is, obvious to someone else :) ).
EDIT:
Found the following: https://zeromq.jira.com/browse/LIBZMQ-207
Seems as though it is/was a known issue.
That link further links to Github, where a change log for 2.1.10 has this note:
Fixed issue 207, assertion failure in zmq_connecter.cpp:48, when an
invalid zmq_connect() string was used, or the hostname could not be
resolved. The zmq_connect() call now returns -1 in both those cases.
Although connect() does indeed throw an Invalid Argument exception in Python (not C# apparently?), recv() still fails. If the subscriber machine suddenly loses the network, that subscriber will simply stop functioning.
So - I'm going to try using IP addresses instead of named addresses to see if this will bypass the issue. Not ideal, but better than insta-crash.
Original question: Is there something obvious I should be doing but I'm not?
No.
The workaround for now is to use IP addressing. This does not cause program failure upon network disconnect for ZMQ 2.1.x.
Related
I'm attempting to connect to a remote server using a specific local interface. My logs tell me everything is working as intended, but checking with netstat, every connection is using the default interface.
I'm using the following code to bind a TcpClient to a specific Local Endpoint
Console.WriteLine("Binding to {0}", connectionArgs.LocalBindingInterface.ToString());
client = new TcpClient(connectionArgs.LocalBindingInterface);
Console.WriteLine("Bound to {0}", client.Client.LocalEndPoint.ToString());
Where connectionArgs.LocalBindingInterface is an IPEndPoint specified as such
IPEndPoint[] localEndPoints = new IPEndPoint[2];
localEndPoints[0] = new IPEndPoint(IPAddress.Parse("192.168.0.99"), 0);
localEndPoints[1] = new IPEndPoint(IPAddress.Parse("192.168.0.100"), 0);
The IP addresses listed here are not the actual addresses.
When i check my logs, this is the info I get
Binding to 192.168.0.99:0
Bound to 192.168.0.99:59252
Binding to 192.168.0.100:0
Bound to 192.168.0.100:53527
But when i netstat -n -p --tcp -a I get
tcp 0 0 192.168.0.98:39948 remote_addr_here:443 ESTABLISHED 17857/mono
tcp 0 0 192.168.0.98:60009 remote_addr_here:443 ESTABLISHED 17857/mono
Clearly something's wrong here. None of the ports, nor the interfaces match. Netstat is run as sudo so I can't assume it's wrong. I also tried to manually create a socket, call it's bind method, and set the TcpClient's Client property to the manually bound socket, but I get the same result.
Is there something i'm doing wrong here? Is there a different way to force a Socket to use a specific Local EndPoint on mono?
I'm running this app as a non-root user, mono --version is Mono JIT compiler version 3.2.8 (Debian 3.2.8+dfsg-4ubuntu1.1), server's ubuntu version is Ubuntu 14.04.3 LTS
Edit 1:
Added an extra logging call after calling TcpClient.Connect()
Binding to 192.168.0.100:59000
Bound to 192.168.0.100:59000
After connect bound to 192.168.0.98:55484
Bottom line: you can't do this, not at the socket level.
The routing of outbound traffic is determined by the network routing configuration. You would have to create an explicit routing table entry for your destination to force a specific adapter to be used.
You can bind to a specific IP address, but this only causes inbound traffic to be filtered, i.e. you'll only receive traffic sent to that IP address.
There are related questions you may want to read as well:
How to stop behaviour: C++ Socket sendto changes interface — context is C++ and not constrained to Windows, but it has what is IMHO the most direct, most relevant notes on the topic.
Using a specific network interface for a socket in windows — fairly poor question and answer both, frankly. But it does contain some quotes and links that you might find useful anyway.
Arguably, this question might have been closed as a duplicate of one of those, or perhaps even another similar question. But those two don't really answer the question in an accurate, C#/.NET-specific way, and I didn't actually find any others that seemed any better.
I've read here: Error 10048 when trying to open TcpChannel
I am having what I thought to be a similar problem - apparently not. I took the advice of the first respondant to reset winsock (how does the winsock get corrupted, anyhow?) Anyway, here is my channel registration:
channel = new TcpChannel(channelPort);
ChannelServices.RegisterChannel(channel, false);
and the client call:
// Create a channel for communicating w/ the remote object
// Notice no port is specified on the client
TcpChannel channel = new TcpChannel();
ChannelServices.RegisterChannel(channel, false);
// Create an instance of the remote object
CommonDataObject obj = Activator.GetObject( typeof(CommonDataObject) ,
"tcp://localhost:49500/CommonDataObject") as CommonDataObject;
This seems all too straightforward to be such a hassle to use. But, the problem seems to be with the server's ChannelServices.RegisterChannel(...). Now, the reason I included the client portion is because the client instances, checks for the server object. If it can't find it, then it 'nudges' the server to instance itself. What I was wondering is if checking for the object's available first (a la: Activator.GetObject(...) ) would cause the ChannelServices to 'think' this tcp channel is already registered? It sounds dumb, but that is my only possible explanation. I have turned off the firewall, anti-fungal app, and rebooted. Still receive this
The channel 'tcp' is already
registered.
I looked at my stack trace and did notice:
at System.Runtime.Remoting.Channels.ChannelServices.RegisterChannelInternal(IChannel chnl, Boolean ensureSecurity)
at System.Runtime.Remoting.Channels.ChannelServices.RegisterChannel(IChannel chnl, Boolean ensureSecurity)
I wondered if the RegisterChannelInternal(...) might be what is causing the 'already registerd' issue. So, other than that, I am at a loss...
It's possible that the call I'm making to check for that Channel is causing it. If that is the consensus, then my question changes to: How can I poll for the Channel?
UPDATE:
After removing the initial check for the server from the client and 'assuming' that the server needs to be instanced, I did discover that the client checking is causing the problem. I've managed to get the server going, and the client did get a 'transparent proxy' object. But the question still remains: "How can I poll to discover if the server is instanced?"
The answer is evidently, yes...when the client is registering the channel, it keeps the server from registering another Tcp channel. I have removed the client instancing of a Tcp channel and the registration.
Since I haven't gotten an answer on pinging, I'm going through with a try/catch block on the obj = Activator.GetObject(...). If obj is returned null, then I 'nudge' the server, it fires up...and then the client connects with the CommonDataObject (derived from MarshalByRefObject).
So, in a sense, that is the polling technique I'm using. I'd like something more elegant - that is, an implementation that didn't work by causing a failure. To me, that's more of a hack work-around than a solution.
I found the answer here. Thanks to Abhijeet for the inadvertent solution!!! Btw...don't forget to declare:
using System.Linq;
I'm experiencing really weird behavior with the Socket.Connect method in C#. I am attempting a TCP Socket.Connect to a valid IP but closed port and the method is continuing as if I have successfully connected. When I packet sniffed what was going on I saw that the app was receiving RST packets from the remote machine. Yet from the tracing that is in place it is clear that the connect method is not throwing an exception. Any ideas what might be causing this?
The code that is running is basically this
IPEndPoint iep =
new IPEndPoint(System.Net.IPAddress.Parse(m_ipAddress), m_port);
Socket tcpSocket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp);
tcpSocket.Connect(iep);
To add to the mystery... when running this code in a stand alone console application, the result is as expected – the connect method throws an exception. However, when running it in the Windows Service deployment we have the connect method does not throw an exception.
Edit in response to Mystere Man's answer
How would the exception be swallowed? I have a Trace.WriteLine right above the .Connect method and a Trace.WriteLine right under it (not shown in the code sample for readability). I know that both traces are running. I also have a try catch around the whole thing which also does a Trace.Writeline and I don't see that in the log files anywhere. I have also enabled the internal socket tracing as you suggested. I don't see any exceptions. I see what appears to be successful connections.
I am trying to identify differences between the windows service app and the diagnostic console app I made. I am running out of ideas though
End edit
Thanks
Are you sure the exception isn't being caught and swallowed in the service, but not in the console app?
My first step would be to isolate the differences between the two implementations. You mention tracing, but you don't say whether this is Network tracing (part of the BCL) or your own tracing. If you're not using network tracing, then enable that.
see AppDomain.CurrentDomain.UnhandledException
I have never observed this again. It seems to me that something was corrupt somewhere. Either the OS on which the app was installed or the .NET framework.
The TIBCO EMS user's guide (pg 292) says:
The backup server will work indefinitely to either A) become the
primary server or B) reconnect to the primary server. It also says
clients may receive fail-over notification when the switch is successful (see also TIBCO EMS .NET reference pg 220).
I have some questions spinning off of these facts...
What kind of errors occur on the client side while the servers are attempting fail-over/reconnect?
What is the appropriate response from the client?
Get new Connection objects from the ConnectionFactory until one works?
Wait for fail-over notification? (are current Connection instances fixed at this time? or do I need to get a new instance?)
I hope the scenario is clear, any related information or advice would be appreciated too.
I can at least answer #1 above.
If you have enabled Tibems.SetExceptionOnFTSwitch(true); and have set up an exception handler to capture the messages the server sends to the client, you will see the following:
For single-server, non-fault tolerant connection failures:
"Connection has been terminated".
For fault-tolerant connection failures:
"Connection has performed fault-tolerant switch to "
If you attempt to publish while the connection is down, a TIBCO.EMS.IllegalStateException is thrown with the "Producer is closed" message.
for #2 above, I think the answer is to allow the EMS library to handle as much as possible. Once we got the EMS reconnect functionality to work, it gracefully tried to reconnect until the server became available again and once it reconnected, it was like there was never a problem. The only gotcha is probably if you try to publish a message before the ems connection is back. This is where the exception handler comes in, Once notified that you are in failover mode, you can adjust exception handling on the publisher side to suppress the error until the connection is back. The thing I don't know is how do you tell when you've exhausted all reconnect attempts.
Anyway, Seems like our two worlds are closely related when it comes to EMS - hope our findings (based on your comments on my questions) help you.
We use TEMS (Tibco EMS - a Tibco Product for WCF) So it becomes a custom binding. We tried to break it by doing things like bounce the server to force switch overs and it works really well. make sure you are using version 1.2 not 1.1 because you cannot do anything other then client acknowledgement.
Every now and then in a high volume .NET application, you might see this exception when you try to execute a query:
System.Data.SqlClient.SqlException: A transport-level error has
occurred when sending the request to the server.
According to my research, this is something that "just happens" and not much can be done to prevent it. It does not happen as a result of a bad query, and generally cannot be duplicated. It just crops up maybe once every few days in a busy OLTP system when the TCP connection to the database goes bad for some reason.
I am forced to detect this error by parsing the exception message, and then retrying the entire operation from scratch, to include using a new connection. None of that is pretty.
Anybody have any alternate solutions?
I posted an answer on another question on another topic that might have some use here. That answer involved SMB connections, not SQL. However it was identical in that it involved a low-level transport error.
What we found was that in a heavy load situation, it was fairly easy for the remote server to time out connections at the TCP layer simply because the server was busy. Part of the reason was the defaults for how many times TCP will retransmit data on Windows weren't appropriate for our situation.
Take a look at the registry settings for tuning TCP/IP on Windows. In particular you want to look at TcpMaxDataRetransmissions and maybe TcpMaxConnectRetransmissions. These default to 5 and 2 respectively, try upping them a little bit on the client system and duplicate the load situation.
Don't go crazy! TCP doubles the timeout with each successive retransmission, so the timeout behavior for bad connections can go exponential on you if you increase these too much. As I recall upping TcpMaxDataRetransmissions to 6 or 7 solved our problem in the vast majority of cases.
This blog post by Michael Aspengren explains the error message "A transport-level error has occurred when sending the request to the server."
To answer your original question:
A more elegant way to detect this particular error, without parsing the error message, is to inspect the Number property of the SqlException.
(This actually returns the error number from the first SqlError in the Errors collection, but in your case the transport error should be the only one in the collection.)
I had the same problem albeit it was with service requests to a SQL DB.
This is what I had in my service error log:
System.Data.SqlClient.SqlException: A transport-level error has occurred when sending the request to the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)
I have a C# test suite that tests a service. The service and DB were both on external servers so I thought that might be the issue. So I deployed the service and DB locally to no avail. The issue continued. The test suite isn't even a hard pressing performance test at all, so I had no idea what was happening. The same test was failing each time, but when I disabled that test, another one would fail continuously.
I tried other methods suggested on the Internet that didn't work either:
Increase the registry values of TcpMaxDataRetransmissions and TcpMaxConnectRetransmissions.
Disable the "Shared Memory" option within SQL Server Configuration Manager under "Client Protocols" and sort TCP/IP to 1st in the list.
This might occur when you are testing scalability with a large number of client connection attempts. To resolve this issue, use the regedit.exe utility to add a new DWORD value named SynAttackProtect to the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\ with value data of 00000000.
My last resort was to use the old age saying "Try and try again". So I have nested try-catch statements to ensure that if the TCP/IP connection is lost in the lower communications protocol that it does't just give up there but tries again. This is now working for me, however it's not a very elegant solution.
use Enterprise Services with transactional components
I have seen this happen in my own environment a number of times. The client application in this case is installed on many machines. Some of those machines happen to be laptops people were leaving the application open disconnecting it and then plugging it back in and attempting to use it. This will then cause the error you have mentioned.
My first point would be to look at the network and ensure that servers aren't on DHCP and renewing IP Addresses causing this error. If that isn't the case then you have to start trawlling through your event logs looking for other network related.
Unfortunately it is as stated above a network error. The main thing you can do is just monitor the connections using a tool like netmon and work back from there.
Good Luck.
You should also check hardware connectivity to the database.
Perhaps this thread will be helpful:
http://channel9.msdn.com/forums/TechOff/234271-Conenction-forcibly-closed-SQL-2005/
I'm using reliability layer around my DB commands (abstracted away in the repository interfaece). Basically that's just code that intercepts any expected exception (DbException and also InvalidOperationException, that happens to get thrown on connectivity issues), logs it, captures statistics and retries everything again.
With that reliability layer present, the service has been able to survive stress-testing gracefully (constant dead-locks, network failures etc). Production is far less hostile than that.
PS: There is more on that here (along with a simple way to define reliability with the interception DSL)
I had the same problem. I asked my network geek friends, and all said what people have replied here: Its the connection between the computer and the database server. In my case it was my Internet Service Provider, or there router that was the problem. After a Router update, the problem went away. But do you have any other drop-outs of internet connection from you're computer or server? I had...
I experienced the transport error this morning in SSMS while connected to SQL 2008 R2 Express.
I was trying to import a CSV with \r\n. I coded my row terminator for 0x0d0x0a. When I changed it to 0x0a, the error stopped. I can change it back and forth and watch it happen/not happen.
BULK INSERT #t1 FROM 'C:\123\Import123.csv' WITH
( FIRSTROW = 1, FIELDTERMINATOR = ',', ROWTERMINATOR = '0x0d0x0a' )
I suspect I am not writing my row terminator correctly because SQL parses one character at a time right while I'm trying to pass two characters.
Anyhow, this error is 4 years old now, but it may provide a bit of information for the next user.
I just wanted to post a fix here that worked for our company on new software we've installed. We were getting the following error since day 1 on the client log file: Server was unable to process request. ---> A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The semaphore timeout period has expired.) ---> The semaphore timeout period has expired.
What completely fixed the problem was to set up a link aggregate (LAG) on our switch. Our Dell FX1 server has redundant fiber lines coming out of the back of it. We did not realize that the switch they're plugged into needed to have a LAG configured on those two ports. See details here: https://docs.meraki.com/display/MS/Switch+Ports#SwitchPorts-LinkAggregation