Long Running Application Causes SQL TimeOut - c#

We are terribly confused with a recurring error on our client's machine. I have a long running application that processes ~6million records over about 20 hours. On our dev environment locally everything works, the process completes successfully each day. On our client's machine, however, we continue to get a timeout.
The program simply uses a Parallel.Foreach on each record to spawn a thread and process the record, performing a few queries and inserts.
In the client's sql log we are seeing:
2013-11-21 09:02:12.10 spid4s SQL Server has encountered 4 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [E:\Program Files...
2013-11-21 09:02:12.10 spid4s SQL Server has encountered 5 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [E:\Program Files...
2013-11-21 09:18:02.35 spid4s SQL Server has encountered 10 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [E:\Program Files...
In my C# application's log file I recieve the error:
------ERROR------
Time : 11/21/2013 9:05:14 AM
Source : .Net SqlClient Data Provider
Module : SqlInternalConnection
Method : Void OnError(System.Data.SqlClient.SqlException, Boolean)
Message : Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Occurring At :
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
at System.Data.SqlClient.TdsParserStateObject.ReadByte()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
And in the few traces the client has been able to give us, nothing looks out of the ordinary, until the queries start timing out. We don't see a steady climb in query time or any noticable pattern.
We've added with (nolock) to our queries to ensure we aren't deadlocking, even though the sql log didn't report any. The machine has 16GB of memory on it, and sql's max is set ~14GB.
Essentially I am wondering if anyone can assist me in beginning to locate the problem. It doesn't occur at the same time throughout the day, and never at the same record (between 200k and 1.5 million have been processed before a crash on different runs).

Related

Website loses connection to database sporadically

I've been struggling with this error for just over a month now and can't get it figured out.
Background/Architecture:
MVC 4 application
NHibernate
FluentNHibernate
SQL Server 2008 R2
Azure website
Sporadically throughout the day, my views are throwing errors because when they try to read/display properties on my objects, it says:
NHibernate.Exceptions.GenericADOException: could not initialize a
collection: [WrestleStat.BL.School.Schedule#54][SQL: SELECT
...(removed to keep this short)....] --->
System.Data.SqlClient.SqlException: Timeout expired. The timeout
period elapsed prior to completion of the operation or the server is
not responding. This failure occurred while attempting to connect to
the routing destination. The duration spent while attempting to
connect to the original server was - [Pre-Login] initialization=3;
handshake=23; [Login] initialization=0; authentication=0; [Post-Login]
complete=1; ---> System.ComponentModel.Win32Exception: The wait
operation timed out
--- End of inner exception stack trace --- at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
exception, Boolean breakConnection, Action`1 wrapCloseInAction) at
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject
stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at
System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject
stateObj, UInt32 error) at
System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync() at
System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket() at
System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer() at
System.Data.SqlClient.TdsParserStateObject.TryReadByteArray(Byte[]
buff, Int32 offset, Int32 len, Int32& totalRead) at
System.Data.SqlClient.TdsParserStateObjec
2 weeks ago, I was running my website on AppHarbor, and I thought it was because I was on their "basic" (free) plan. The problem seemed to go away if I'd up my worker threads from 1 to 2. Because of that, I decided to switch over to Azure (I was going to anyway, but not this early). Since the switchover to Azure, the problem has not gone away.
I get approximately a dozen of these errors per day on the site. It appears as though that the errors always occur when the razor views are trying to access properties of an object that it's trying to read/display, as opposed to throwing the error inside the controller trying to retrieve the data. Maybe it has something to do with the lazy loading??? But why would that matter, why would the connection be lost in that extremely short timeframe from retrieving the data, then presenting it on the page?
I'm at a loss on how to debug this issue, because it obviously never happens in my development environment. I'm going to switch over to Dapper eventually, but I'm not able to do/start that for at least another 3 months.
Any ideas here?
Edit: here's the website http://www.wrestlestats.com
These are transient errors normal in a cloud environment. You need to implement a retry logic/policy.
Check this answer: How do I add Retry Logic in NHibernate to handle Transient Failures in SQL Azure?

SQL Session state server in own database

I've made a website and uploaded it to my hosting-server (some smalltime hosting provider). It runs great but there is a problem with the Session; It keeps logging out my user.
It happens on complete random, so I phoned the provider to ask what kind of server my website is hosted in. They told me it was a shared hosting IIS farm. So what is happening is that the loadbalancer is re-routing my session to another server, so my session is lost and I get a login screen.
To solve this I need to use a Session State Service or SQL Session State database.
The provider said there is no way to run a dedicated Session State Service in the server-farm, so that leaves me with the SQL Session State database option.
For this I need to run the "aspnet_regsql" tool on my SQL Server, but when I do this I get this error:
Microsoft (R) ASP.NET SQL Registration Tool version 4.0.30319.33440
Administrative utility to install and uninstall ASP.NET features on a SQL server.
Copyright (C) Microsoft Corporation. All rights reserved.
Start adding session state.
.
An error occurred during the execution of the SQL file 'InstallSqlState.sql'. The SQL error number is 262 and the SqlException message is: CREATE DATABASE permission denied in database 'master'.
SQL Server: ................
Database: aspnetdb
SQL file loaded:
InstallSqlState.sql
Commands failed:
/* Create and populate the session state database */
IF DB_ID(N'ASPState') IS NULL BEGIN
DECLARE #cmd nvarchar(500)
SET #cmd = N'CREATE DATABASE [ASPState]'
EXEC(#cmd)
END
SQL Exception:
System.Data.SqlClient.SqlException (0x80131904): CREATE DATABASE permission denied in database 'master'.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler,TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at System.Web.Management.SqlServices.ExecuteFile(String file, String server, String database, String dbFileName, SqlConnection connection, Boolean sessionState, Boolean isInstall, SessionStateType sessionStatetype)
ClientConnectionId:............................................
Error Number:262,State:1,Class:14
So I don't have permissions to create my own database, so there is no way to create the Dedicated "ASPState" database.
Can anyone help? Is there a way to maybe install the SessionState service on the database that is assigned to my account, instat of a dedicated ASPState database?
Use aspnet_regsql command line utility then specify connection string and database name to create tables if db name exists in server it does not create database and will create tables
[Msdn link] https://msdn.microsoft.com/en-us/library/ms229862.aspx
For example command line:
aspnet_regsql -A all -C "Data Source=.\SQLEXPRESS;Integrated Security=True;" -d yourbusinessdbname

What account to use in sessionstate connectionstring

I am working on a Website that was donated to our institution, right now I'm trying to decide exactly how to set up two connection strings (CS) in the Web.config file (some of the details in these connection strings were redacted by the donator).
The CS I have a question about is referenced in sessionState:
<sessionState allowCustomSqlDatabase="true" mode="SQLServer" sqlConnectionString="DevelopmentSessionState" cookieless="false" timeout="20"/>
I'm trying to determine what type of Account I should be using for the UserID in the SessionState CS. In order to get the Website functional I used the sa account, but I know this isn't a good idea - I'm looking for an account to replace sa. I have no idea if this is a user I should define in SQL Server, or if I should use a pre-existing account, or what type of access this user should have.
I am also fuzzy on the idea of why a separate connection string is needed for Session State. The Website's c# code uses the other CS explicitly to connect to the db, but the only reference to the SessionState CS is in Web.config. I have read a lot about Session State and SQLServer mode that past few days, but I haven't found any recommendations for what account to use or why a separate CS is needed for SessionState.
Thanks,
Chris
Update: This is the stack trace from the abend that happens when I substitute my UserID for sa in the Session State CS (I replaced my actual User ID with USERID):
Server Error in '/' Application.
Failed to login to session state SQL server for user 'USERID'.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Web.HttpException: Failed to login to session state SQL server for user 'USERID'.
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[SqlException (0x80131904): Login failed for user 'USERID'.]
System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection) +4876535
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj) +194
System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) +1121
System.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK) +35
System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, Boolean ignoreSniOpenTimeout, Int64 timerExpire, SqlConnection owningObject) +144
System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(String host, String newPassword, Boolean redirectedUserInstance, SqlConnection owningObject, SqlConnectionString connectionOptions, Int64 timerStart) +367
System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(SqlConnection owningObject, SqlConnectionString connectionOptions, String newPassword, Boolean redirectedUserInstance) +225
System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, Object providerInfo, String newPassword, SqlConnection owningObject, Boolean redirectedUserInstance) +189
System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection) +185
System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnection owningConnection, DbConnectionPool pool, DbConnectionOptions options) +31
System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject) +431
System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject) +66
System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject) +499
System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection) +65
System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory) +117
System.Data.SqlClient.SqlConnection.Open() +122
System.Web.SessionState.SqlStateConnection..ctor(SqlPartitionInfo sqlPartitionInfo) +93
[HttpException (0x80004005): Failed to login to session state SQL server for user 'USERID'.]
Version Information: Microsoft .NET Framework Version:2.0.50727.5485; ASP.NET Version:2.0.50727.5491
I did finally succeed in creating a user that worked in the Session State CS.
To begin with, rather than running InstallSqlState.sql I ran aspnet_regsql.exe. The SQL file had a warning not to run it standalone and that I should instead use aspnet_regsql.exe, so I did that. I used the following options:
aspnet_regsql.exe -ssadd -sstype p -E -S servername
This created db ASPState and meant that I had to use mode="SQLServer" in in web.config.
I also had to add role 'public' to ASPState as suggested here:
http://forums.asp.net/t/969708.aspx
This post included giving 'public' permissions but did not talk about which ones would be needed, so I had to experiment with that until the CS worked. Permissions were added by right clicking on db ASPState, choosing Properties, selecting Permissions, highlighting database role 'public', then checking off permissions under the 'Grant' heading.
Finally I logged in to SQL Server Management Studio using SQL Server Authentication and created a new user S1. I only gave 'Connect' permission to S1 using the same navigation described above. S1 replaced sa in the session state connection string. Here is the web.config entry for the session state CS. Obviously I changed the pw and servername, but you can see the format I used:
<add name="DVSessionState" connectionString="Data Source=servername;User Id='S1';Password='password'"/>
The following steps describe how to run the InstallSqlState.sql and the UninstallSqlState.sql script files to configure SQL Server mode session state management.
In SQL Query Analyzer, on the File menu, click Open.
In the Open Query File dialog box, browse to the InstallSqlState.sql script file, and then click Open. By default, InstallSqlState.sql is located in one of the following folders:
system drive\WINNT\Microsoft.NET\Framework\version\
system drive\Windows\Microsoft.NET\Framework\version\
After InstallSqlState.sql opens in SQL Query Analyzer, click Execute on the Query menu to run the script.
Before you run the UninstallSqlState.sql script file to uninstall SQL Server mode session state management configuration, you must stop the w3svc process. To do this, follow these steps:
On the Windows Start menu, click Run, type cmd, and then click OK to open a command prompt.
At the command prompt, type net stop w3svc. You receive confirmation that the w3svc process is stopped.
In SQL Query Analyzer, on the File menu, click Open.
In the Open Query File dialog box, browse to the UninstallSqlState.sql script file, and then click Open. By default, UninstallSqlState.sql is located in one of the following folders:
system drive\WINNT\Microsoft.NET\Framework\version\
system drive\Windows\Microsoft.NET\Framework\version\
After UninstallSqlState.sql opens in SQL Query Analyzer, click Execute on the Query menu to run the script.
After you uninstall SQL Server mode session state management configuration, you must restart the w3svc service. To restart the w3svc process, type net start w3svc at a command prompt.
Modify the Web.config File of Your Application
To implement ASP.NET SQL Server mode session state management, you must modify the <sessionState> element of your application's Web.config file as follows:
Set the mode attribute of the <sessionState> element to SQLServer to indicate that session state is stored in SQL Server.
Set the sqlConnectionString attribute to specify the connection string for SQL Server. For example:
sqlConnectionString="data source=MySQLServer;user id=;password="
Note: The user, <user name>, must have permissions to perform this operation on the database.
The modified <sessionState> element should appear as follows:
<sessionState
mode="SQLServer"
sqlConnectionString="data source=127.0.0.1;user id=<username>;password=<strongpassword>"
cookieless="false"
timeout="20"
/>
Note: Ensure that you use the correct case when you specify the <sessionState> element and the associated attribute values. This code is case sensitive.

windows service doesn't start automatically on reboot

I have made one simple service for retrieving backup from database.
I have set the start up type to "Automatic".
The problem is when i restart the PC, that particular service doesn't start.
I have checked the Event log. It display Error message as given below:
"Service cannot be started. System.Data.SqlClient.SqlException: Cannot open database "Smart_Tracker" requested by the login. The login failed. Login failed for user 'sa'. at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) at System.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK) at System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, Boolean ignoreSniOpenTimeout, Int64 timerExpire, SqlConnection owningObject) at System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(String host, String newPassword, Boolean redirectedUserInstance, SqlConnection owningObject, SqlConnectionString connectionOptions, Int64..."
I have checked the Connection string in Config file of that service and it is correct.
Please help me to solve this problem.
Try to log in from SSMS using 'sa' and password defined in your configuration file. If you fail - check SQL server settings, make sure that both Windows and SQL server authentication modes are enabled (Security). If you can connect using credentials defined in configuration file, change the account under which service is running (Services->Properties->"Log on" tab). Start switching from "Local system account" to our own. Then you can change this to another user with granted privilege to log in into SQL server.

SQLConnection Leakage Issue (Timeout Expired due to Pooling - ASP.NET / SQL Server)

I am developing a web application which is based on ASP.NET 4.0, jQuery, AJAX, Javascript using SQL Server 2008 database. Our application is an ERP application which has almost around 400 to 500 users using the application. We have made it live since a month ago.
Now, I'm facing a timeout issue and application crashed literally which is occurred after some hours of application usage.
Server Error in '/' Application.
--------------------------------------------------------------------------------
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
Source Error:
Line 36: param[1].Value = Password;
Line 37:
Line 38: return SqlHelper.ExecuteDataset(CDSGlobal.ConnectionString, CommandType.StoredProcedure, "ValidateLogin_sp", param);
Line 39: }
Line 40:
Source File: e:\abcd\App_Code\User.cs Line: 38
Stack Trace:
[InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.]
System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection) +6352273
System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory) +6352606
System.Data.SqlClient.SqlConnection.Open() +300
DBConn.SqlHelper.ExecuteDataset(String connectionString, CommandType commandType, String commandText, SqlParameter[] commandParameters) +270
User.ValidateLogin(String Employee_Code, String Password) in e:\abcd\App_Code \User.cs:38
User_UserLogin.btnLogin_Click(Object sender, EventArgs e) in e:\abcd \User\UserLogin.ascx.cs:79
System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) +154
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +3707
--------------------------------------------------------------------------------
Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.272
For temporary resolution, we are restarting the server and restarting the MSSQLSERVER service.
After searching for above, I came to know it is due to connection leakage. Some opened connections are never get closed.
I am using ready made Microsoft Data Access Application Block for .NET i.e. SQLHelper Class which in my application for data access using Dataset and all.
Note: I am NOT using SqlDataReader anywhere.
I have examined the class and its closing the connections and I have NEVER used manual connection code instead used SQLHelper everywhere.
I also have the following nested transactions structure in stored procedures as follows:
BEGIN TRY
BEGIN TRANSACTION
-- Every below stored procedure has the transation structure as this SP i.e. Parent SP. (BEGIN TRY, BEGIN TRANSACTION etc)
exec sp1
exec sp2
exec sp3
---
---
COMMIT TRANSACTION
END TRY
BEGIN CATCH
IF ##TRANCOUNT>0
ROLLBACK TRANSACTION
PRINT 'ERROR OCCURED In SP'
END CATCH
Now, The problem is why connection leakage is occurring.
Is above transaction structure is responsible i.e it is blocking the transactions and requests.
If request is in process and user has closed the browser, does the application closes the connection?
Any help will be greatly appreciated.
Solved !!
It was not due to Nested Transactions but it was due to Bad Indexes Strategy used. Used normal Indexes and Indexes with include to solve the performance issues.
Also ran SQL Profiler to examine the most read and writes on columns and accordingly applied indexes and review maintenance and execution plans. !!
You can also use this,
SqlConnection.ClearAllPools();
The above code then closes all of the pools upon execution. I had the same problem and cleared the pools each time and then it should work successfully due to the pools being cleared.

Categories