I'm running a load test on my system. At a certain level of load, I start getting SQL errors in my log:
System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Prprovidererror: 40 - Could not operrorconnection to SQL Server) ---> System.ComponentModel.Win32Exception (0x80004005): The network path was not found
By running performance monitor on the SQL server in question, I found the following:
CPU level rarely exceeds 50%. (On a previous iteration I saw that it was maxing out at 100%, so I increased the specs of the VM, which helped push the problem to a higher load level.)
Number of user connections got to a shade over 8,000. The Sql Server has the default setting of 32,767 max connections.
The connection string specifies a max pool size of 1000 connections to each database, and there are 100 databases on the server. The load test is randomly distributed between the 100 databases, so there should be a fairly even distribution, meaning about 80 connections per database. Nowhere near the 1k limit.
What other factors might cause Sql Server to stop being able to accept connections?
UPDATE: extra info: I'm using Entity Framework Core (EF7) for my DB connections, if that helps any.
"Network Path Not Found" does not seem like an error related to SQL Server's capacity. As a former "IT Guy," I suspect that a firewall is dropping your packets. If this is during a stress test, the firewall could be interpreting the numerous requests as a denial of service attack, and using some sort of predefined rule to drop connections for a specified time period.
What is your network environment? If you have a hardware firewall or router with IPS capabilities, I would check those logs to see if you find a smoking gun. You might have to create a special rule to allow unlimited traffic to your SQL Server.
It's a bit curious that you are getting that many connections to the database. You should be utilizing connection pooling; even under intense load, the connection pooling should greatly reduce the number of active connections being used.
Can you provide the code that's accessing the database? Are you calling the dispose() method or closing the connection?
Also, have you looked to see if data datacaching would ease the db load? A 2-5 second datacache can greatly reduce database calls.
You are running into the TCP listen() backlog limit for the SQL-Server's listening port. When this happens, Windows platforms (but not *nix platforms) will issue 'connection refused' for further incoming connections.
I'm not an SQL-Server guy but there is bound to be a parameter somewhere by which you can increase its listen backlog.
Alternatively you should look into better or more connection pooling at the client.
It turns out the problem wasn't on SQL at all. The problem was on our API server, where some of the APIs were spinning off hundreds of parallel threads, each making its own connection to the database. The load was simply too much for the API server, and it started returning "Access Denied" exceptions without even really attempting to connect to the database.
Solution: we throttled the number of threads being spun off, using the pattern shown in this answer.
Related
I use Entity Framework 6.4.4 in my multi-threads server application to connect to a SQL Server database on another server on the internet.
Before using DbContext, I open the connection by calling MyDbContext.Database.Connection.Open(). Multiple threads may try to open the connection at the same time. But sometimes I get this exception of type InvalidOperationException with Message: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
I am sure I close all the connections after being used, and no connection remains open for more than about 2 seconds. And also, I am sure there are not many simultaneous connections.
The SQL Server can support about 1500 connections at the same time (I have tested), but I think the problem is with opening a connection not having many opened connections.
I tested my application on a server with a CPU of 40 logical processors. It works fine. But when I move my application to a server with 4 logical processors, it works correctly for a while, but it can not open even one single connection after a period. I limited the number of threads tries to open the connections simultaneously to even 3. Still, it didn't help, and I get that exception continuously. CPU usage is consistently below 50%, and there is free memory.
I have an application that makes many connections to both SQL and Oracle DBs. Oracle connection throws this exception many times. I did a search and found this link
http://www.forumtopics.com/busobj/viewtopic.php?p=925784
It looks like a .net issue in Windows as described below
"[The ORA-12542 TNS address in use error] is caused due to the fact that the free ports in the windows client has been exhausted and it is trying to make use of a port which is in TIME_WAIT state which gives the error TNS-12542: Address already in use."
My questions are:
1> Is it possible in .NET to specify the local port range for making connections, e.g., one range of ports dedicate to SQL server connection, and another range of ports to Oracle.
2> How ports are allocated to connections in .Net? So, Time_Wait ports will not be reallocated.
3> For oracle connections, my local ports are allocated with port numbers starting from very high numbers, e.g., 50000, but with SQL Server connections, my local ports start from 5000. Why is that?
4> How to solve this problem?
Thanks in advance.
Edit: We also experience Timed out exception in the SQLServer connections. I think they are related.
This problem can occur if you have connection pooling turned off, and ephemeral ports are exhausted. Using connection pooling reduces the number of connections required to satisfy the same request throughput, and may resolve the problem.
Local port usage is commonly restricted to particular ranges, even for ephemeral ports, to make it easier to identify traffic and manage firewall rules for example.
Connection pooling is generally in place for database connections as connection is a relatively expensive operation. For this reason it is good to reuse connections.
As to why this is preventing you from connecting, you may want to look at this:
http://technet.microsoft.com/en-us/library/cc938196.aspx
Perhaps MaxUserPorts has been modified to a non-default value? By default there should be 3976 ports available allowing 3976 simultaneous connections to the same remote IPaddress/port combination. (The port only has to be unique if the remote end is the same - the combination remoteIP:port plus localIP:port must be unique).
Alternatively, some components manage their own local port allocation rather than delegating to the OS, so perhaps this is the case with SqlConnection.
I have a project that connects to a RMS file system through Attunity (Version 1.0.0.8). The RMS file is in a different server. The connection pool on both client and service is 10 (Max connection pool size). When we hit the server from the client, we sometimes get the error:
C014: Client connection limit reached - try later.
I would like to understand whether this error is related to Server overloaded or any issues on the Client side. I am sure that the client code that I am using to connect to the server is properly opening and disposing the connection.
This sounds like a problem we were having. We were running Attunity on OpenVms and we were maxing out the number of DECNET connections to Attunity between our nodes; the underlying problem was with our client. The clients in this case would induce a longstanding transaction or had problems releasing their connections back to the pool. We fixed the issue by eliminating the longstanding transactions and then finding the bug in the clients where the they would not release their connection. Unfortunately, all of our clients are implemented in Java and Cobol, so I don't have any .Net specific advice.
I have developed a TCP server according to your advises: High performance TCP server in C#
It is based on asynchron pattern.
I also developed a stress test application to test its performance. My server can get thousands of connections paralelly from my stress test app, can parse data and save it to my database.
When I stress my server, I can get "System.Net.Sockets.SocketException "No connection could be made because the target machine actively refused it" error from my server, so I have to reconnect to it. If I test it with 5000 concurrent connections, I have to try connect again because of this problem 10-20% of the connections, if I test it with 10K concurrent connections, it can be 30-40%. Sometimes it can be - very rarely - more, than 50%. It seems it can not handle connection accepts: I make new connections from my stress test as heavily as my test machine can - about 120 connections/sec.
So, what can cause this kind of exception? How to handle it? What to do in server side implementation to avoid this problem? How to tune TCP connection accept?
Thanks in advance!
You might be running out of available ports every now and then. You can view this easily using SysInternals' TcpView utility.
On Windows, when you release a port, it doesn't immediately go into an available state, but instead sits in a TIME_WAIT state for some interval. Until it leaves this state, no app can use this port. The time delay, the max number of ports, and the available port ranges are all different to the OS, XP vs Win7 vs Win2008 Server.
There are two registry entries that can reduce this time interval:
HKLM/System/CurrentControlSet/Services/Tcpip/Parameters/TCPTimedWaitDelay
and increase the max number of ports that can be opened by an app:
HKLM/System/CurrentControlSet/Services/Tcpip/Parameters/MaxUserPort
EDIT: MaxFreeTcbs seems to be a third setting which could help (I haven't tried this yet), mentioned in this TechNet article which has more advice on tracking down odd network problems. HTH.
You are making connections faster than the software can listen for new connections, or in other words you are reaching the connections per second limit of that port. I think you can double the amount of connections per second by listening to a second port, client side you should just reconnect when you get the exception.
There are also limits applied to the amount of connection, for these see Chris O's answer.
I have a web site which uses one SQL database but the hosting company is very slow sometimes and I got database timeout, login and similar errors. Can I implement my code to use two databases simultaneously? I have stored procedures and the data is updated periodically.
EDIT:
Simply: When dbDefault is down and inaccessible I need to use dbSecondary so the web app keeps running. Ant these two databases must be always same.
EDIT:
Some errors:
A transport-level error has occurred when sending the request to the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
Cannot open database "db" requested by the login. The login failed. Login failed for user 'root'.
Load balancing and/or fail-over clustering database servers typically involves a lot of work.
You will need to make sure ALL data is merge replicated between the two database servers. Hosting providers rarely provide this option unless you have a dedicated server.
Allowing for merge replication might involve redesigning parts of your database; which may not be feasible.
Unless you are willing to invest a lot of time and money, you are much better off just switching hosting providers to one that has better db support. Considering there are literally thousands upon thousands of such companies out there this is an easy fix.
UPDATE
Almost of all the errors you identified in your edit are generally attributable to failing to properly dispose of connections, commands, and readers. You might want to go through your code to make sure you are accessing the sql server correctly. Every connection, command, and reader should be wrapped in a using clause in order to make sure they are properly released back to the connection pool.
If you provide a data access code sample (new question please) we can help you rewrite it.
Not really.
Data consistency and integrity:
How do you decide what data or what call to make at what time?
What happens on write?
Firewalls, remote server etc:
If you use another hosting company, how will you connect?
Misconception:
Two databases on one server = absolutely no advantage.
The server is probably overloaded, a 2nd db will make it worse
A database timeout could be code related of course and it may not help you to have 2 databases with the same poor code or design...
Not a nice answer, but if your host is providing poor service then your options are limited
ist of all find the reasons of the Timeout if it is in your code than rectify the code by optimizing query etc.
i think what you need is a Failover Server , where you can switch if the one server is down.
Alternatively
you can maintain two connection string in web.config and can switch to other server if one is down.
in both the method , you need to devise an strategey to sync the servers.
If both your database are in synch (which is an obvious requirement for what you are trying to do), the best solution is to rely on a loadbalancer but if you can't, I guess you goal is to run the query against both database in the same time and returns the first result otherwise you will have to wait for the timeout to run the request against the second server.
SO what you need is asynchronous sql command right ?