I have a project that connects to a RMS file system through Attunity (Version 1.0.0.8). The RMS file is in a different server. The connection pool on both client and service is 10 (Max connection pool size). When we hit the server from the client, we sometimes get the error:
C014: Client connection limit reached - try later.
I would like to understand whether this error is related to Server overloaded or any issues on the Client side. I am sure that the client code that I am using to connect to the server is properly opening and disposing the connection.
This sounds like a problem we were having. We were running Attunity on OpenVms and we were maxing out the number of DECNET connections to Attunity between our nodes; the underlying problem was with our client. The clients in this case would induce a longstanding transaction or had problems releasing their connections back to the pool. We fixed the issue by eliminating the longstanding transactions and then finding the bug in the clients where the they would not release their connection. Unfortunately, all of our clients are implemented in Java and Cobol, so I don't have any .Net specific advice.
Related
I'm running a load test on my system. At a certain level of load, I start getting SQL errors in my log:
System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Prprovidererror: 40 - Could not operrorconnection to SQL Server) ---> System.ComponentModel.Win32Exception (0x80004005): The network path was not found
By running performance monitor on the SQL server in question, I found the following:
CPU level rarely exceeds 50%. (On a previous iteration I saw that it was maxing out at 100%, so I increased the specs of the VM, which helped push the problem to a higher load level.)
Number of user connections got to a shade over 8,000. The Sql Server has the default setting of 32,767 max connections.
The connection string specifies a max pool size of 1000 connections to each database, and there are 100 databases on the server. The load test is randomly distributed between the 100 databases, so there should be a fairly even distribution, meaning about 80 connections per database. Nowhere near the 1k limit.
What other factors might cause Sql Server to stop being able to accept connections?
UPDATE: extra info: I'm using Entity Framework Core (EF7) for my DB connections, if that helps any.
"Network Path Not Found" does not seem like an error related to SQL Server's capacity. As a former "IT Guy," I suspect that a firewall is dropping your packets. If this is during a stress test, the firewall could be interpreting the numerous requests as a denial of service attack, and using some sort of predefined rule to drop connections for a specified time period.
What is your network environment? If you have a hardware firewall or router with IPS capabilities, I would check those logs to see if you find a smoking gun. You might have to create a special rule to allow unlimited traffic to your SQL Server.
It's a bit curious that you are getting that many connections to the database. You should be utilizing connection pooling; even under intense load, the connection pooling should greatly reduce the number of active connections being used.
Can you provide the code that's accessing the database? Are you calling the dispose() method or closing the connection?
Also, have you looked to see if data datacaching would ease the db load? A 2-5 second datacache can greatly reduce database calls.
You are running into the TCP listen() backlog limit for the SQL-Server's listening port. When this happens, Windows platforms (but not *nix platforms) will issue 'connection refused' for further incoming connections.
I'm not an SQL-Server guy but there is bound to be a parameter somewhere by which you can increase its listen backlog.
Alternatively you should look into better or more connection pooling at the client.
It turns out the problem wasn't on SQL at all. The problem was on our API server, where some of the APIs were spinning off hundreds of parallel threads, each making its own connection to the database. The load was simply too much for the API server, and it started returning "Access Denied" exceptions without even really attempting to connect to the database.
Solution: we throttled the number of threads being spun off, using the pattern shown in this answer.
I observed that one of my Windows Services was not connecting to an FTP location on a Unix Server, I ran the executable on my PC as the dev didn't log any error and i was getting timeout error on trying to get response from an FTPWebRequest Object in C#.
On trying to access the FTP location using Filezilla I am getting the error
GnuTLS error -110: The TLS connection was non-properly terminated.
Using SFTP does not give this error and also using FTP in plain text(insecure) also does not.
I really do not understand this and note that the application has been running fine for years and this just suddenly happens on like 4 servers.
GnuTLS error -110: The TLS connection was non-properly terminated.
That just means that the peer just closed the socket and did not do a proper TLS shutdown. Some broken clients or servers do this. Assuming that this message relates to a data transfer you can usually ignore this because the transfer was finished anyway, so no data got lost.
Using SFTP does not give this error and also using FTP in plain text(insecure) also does not.
Of course you don't get it, since SFTP is using the SSH protocol instead of TLS and plain FTP does no encryption at all, so no TLS too. And if there is no TLS involved you can not get any TLS errors.
I really do not understand this and note that the application has been running fine for years and this just suddenly happens on like 4 servers.
It might simply be that the servers changed, i.e. either they never supported FTPS (i.e. FTP with TLS, not to be confused with SFTP) before or they now switched to a broken implementation.
If you are connecting to a cPanel server then you can temporarily fix this issue by enabling "Broken Client Compatibility" in Pure-FTP settings in WHM.
Older version of Filezilla may be responsible for the said error. Faced the same error on 3.4 version issue was resolved after downloading the 3.6 one.
I'm working on a C# WebSocket server (currently supported by https://datatracker.ietf.org/doc/html/draft-ietf-hybi-thewebsocketprotocol-17).
The server is working with the Socket object of the .NET for the server to listen and for each client to send and receive messages.
I built a web client that connect to the server, It can connect successfully and i can send messages between clients.
Everything is working great!
Now, if i'm connecting to the server and leave the client for a while without sending messages, the server throwing an exception that says:
Int32 Send(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags):An
existing connection was forcibly closed by the remote host.
The exception, as you can see is from the Send method of the client socket in the server, this is looks very wired because i didn't sent any data from the client and no one sending data to this client back so how can it be that the Send method can throw an exception and why this exception is thrown?
It's called a timeout!
WebSockets are just a wrapper around TCP/IP raw sockets (Socket class in .NET) - which timeout if nothing is sent, and nothing is keeping the connection alive.
AFAIK currently the WebSocket API isn't very well defined as far as how to keep the connection alive. I was experiencing the same and had to just switch over to using a ping (empty message) to keep the connection alive (I'm using the Microsoft sockets implementation).
If you're reinventing the wheel for a non final spec, just remember that you'll have to keep reinventing it every time the spec changes. I specifically chose to use the Microsoft sockets preview so that when it's released I'm pretty much not going to have to change any code. I don't run in IIS - I run as a console app and it's working mostly great so far but I have very very few users.
Note: The problem i was having that led me to find this question was if I send 10 messages without receiving a reply then the connection is closed. I'm still looking into why this is - whether its a bug / feature of WebSockets or a feature of the Socket class. it's possible I'm hitting a 65kb limit but my messages are small and I don't think that's why. Just be aware of this when testing whatever you're working on becasue it gives the same error you got.
I assume that you have exclude the usage of different protocols between the servers and the clients (silly assumption, but you never know).
If your code reaches the Send method without a prior Receive from the client, then it's obvious that something is wrong with the server code. Use trace and/or log to get more information even for abc's like entering wait to receive, receiving, received, exiting receiving etc.
I have developed a TCP server according to your advises: High performance TCP server in C#
It is based on asynchron pattern.
I also developed a stress test application to test its performance. My server can get thousands of connections paralelly from my stress test app, can parse data and save it to my database.
When I stress my server, I can get "System.Net.Sockets.SocketException "No connection could be made because the target machine actively refused it" error from my server, so I have to reconnect to it. If I test it with 5000 concurrent connections, I have to try connect again because of this problem 10-20% of the connections, if I test it with 10K concurrent connections, it can be 30-40%. Sometimes it can be - very rarely - more, than 50%. It seems it can not handle connection accepts: I make new connections from my stress test as heavily as my test machine can - about 120 connections/sec.
So, what can cause this kind of exception? How to handle it? What to do in server side implementation to avoid this problem? How to tune TCP connection accept?
Thanks in advance!
You might be running out of available ports every now and then. You can view this easily using SysInternals' TcpView utility.
On Windows, when you release a port, it doesn't immediately go into an available state, but instead sits in a TIME_WAIT state for some interval. Until it leaves this state, no app can use this port. The time delay, the max number of ports, and the available port ranges are all different to the OS, XP vs Win7 vs Win2008 Server.
There are two registry entries that can reduce this time interval:
HKLM/System/CurrentControlSet/Services/Tcpip/Parameters/TCPTimedWaitDelay
and increase the max number of ports that can be opened by an app:
HKLM/System/CurrentControlSet/Services/Tcpip/Parameters/MaxUserPort
EDIT: MaxFreeTcbs seems to be a third setting which could help (I haven't tried this yet), mentioned in this TechNet article which has more advice on tracking down odd network problems. HTH.
You are making connections faster than the software can listen for new connections, or in other words you are reaching the connections per second limit of that port. I think you can double the amount of connections per second by listening to a second port, client side you should just reconnect when you get the exception.
There are also limits applied to the amount of connection, for these see Chris O's answer.
I have a C# application that has been running fine for several years. It connects via a TCP/IP socket to a machine that sends me stock trade executions.
Recently, I've tried to deploy it to some machines in a new data center that is behind a hardware firewall, and I've started to see some weird dis-connects.
When a dis-connect happens, in my app (the client side), I see nothing unusual except that I stop receiving data over the socket. Wireshark confirms that no data is reaching the socket and my application's receive thread is blocking on the Receive() call when I stop it in the debugger. The socket shows as ESTABLISHED in netstat.
But from the server side, it looks like my client is dis-connecting. Looking at their logs, it looks like the socket on their end usually ends up with either (nRecvd=-1,errno=104) or (nRecvd=0,errno=11). (104 is connection reset by peer).
The dis-connect only seems to happen after a period of in-activity. I have solved this for now by implementing a heartbeat between my client and their server that just sends a short message every 20 seconds and gets a reply. This has caused the dis-connects to drop to 0 over the past few days.
At first, I figured that the hardware firewall was the problem. It was causing the socket to time out after in-activity. But the person in charge of the firewall claims that the timeout for connects on this port (8887) is 2160 minutes.
I am running Windows Server 2003 and .NET 3.5. The trades server is a linux machine (sles9 I believe though I'm not sure).
Any ideas on what might be going on? What could I do to debug this more given that I don't have any access to the firewall logs and no ability to change the code on the trade server?
Thanks,
Mike
What you describe is common, and it's common to implement a heartbeat to keep TCP sockets alive through such firewalls/gateways like you did.
That hardware might have hard 2160 minutes timeouts (in my experience 20-30 minutes is more common though) , but connections are usually dropped much more aggressively if there's any kind of load. Such firewalls have limited resources, and when they need more connection tracking they tend to drop the oldest connection tracked without any activity regardless of the hard timeout set.
If you want to debug this more, go sniff on the server side of the firewall and see what , if anyting, happens when the server gets a disconnect
I would setup wiresharp on both sides of the firewall to see what happens on TCP (and lower level).
And when the admin says the "timeout for connects" is something. Is that the timeout for an idle, established connection? Anything else does not make any sense I guess.
Also, are you using KeepAlive option for TCP? And is that forwarded by the firewall or not?
As I said, probably want to run wireshark on both sides of the firewall...