This is prior to EF 6. My company has a process that works with all of our other clients expect one. The process opens a connection to the clients database, reads out 1000 records at a time, and commits it to our database.
For this client, we read and commit the first 1000 records just fine. When it starts to read again I get "Underlying provider failed on Open". I understand that EF transactions open and close for each read, so when it tries to reopen the connection to do the next read is when it is failing.
Details: We connect through a VPN to the client database.
The code flow is:
connection.open()
create datareader
while datareader.read()
get 1000 records
bulk commit
db.SaveChanges
get next 1000 records
and so on until it gets all records
After the first SaveChanges is when we get the error.
Any help is appreciated.
Prior to EF6 the DbContext was closing the connection when it was getting disposed regardless whether it owned it or not. Starting from EF6 the context honors the contextOwnsConnection flag passed to the constructor (see here). It's not clear from your pseudocode how you instantiate the connection and the context, so presume you create the context in the loop and pass the opened connection. If that's the case you have a few options:
Upgrade to EF6, or
Use only one DbContext for all the saves, or
Load all the records into memory and process them in chunks each in it's own DbContext, or
Load them in chunks and process them in chunks
If you avoid using the same context for your processing for performance reasons, you can use .AsNoTracking(). There is an article on MSDN on EF performance tuning in case you need more.
Thanks for everyone's help. Turns out that the connection being lost was to our database and not the client's. Not entirely sure why, but what seemed to help was putting our BulkInsert method to create the SqlBulkCopy object inside of a using block. We also, at the point at which it failed, reestablished the connection. It's a little hacky, but it's working.
Related
I have an EntityFrameworkDataService implemented on a server application where client applications can query from it. I googled around for a while but couldn't find out how to close the database connection once we're done with a query. Is there a way to do this or is it done automatically?
See here:
The Entity Framework opens connections only when required, for example to execute a query or to call SaveChanges, and then closes the connection when the operation is complete.
I've no reason at all to assume that EntityFrameworkDataService<T> will force its DbContext to keep connections open. Of course you can always check by profiling the database.
I have a long running transaction performing a lot of delete queries on a database; the issue is that the mysql connection (to the server on the same machine) will be dropped for no reason every now and then.
Currently, my retry logic will detect the disconnection, reconnect, and restart the whole transaction from the beginning, which may never succeed if the connection's "dropping frequency" is too high.
Is it possible at all to reopen a lost connection to continue the transaction?
I am using MySQL Connector for .NET.
What you are asking is not possible for a Transaction. A transaction is to make sure that either each and every action performed on DataBase is completed or None are.
If your Connection Dropping frequency is too high and you don't have a control on fixing it then what you should do is to make simple queries without a transaction or Better Make the Number of Actions in your Transaction fewer and Send a Batch of Transactions instead of a Single Big Transaction.
And also add some data validation check codes to make sure every thing is right with entries.
Theoretically you can do exactly what you need with the XA transactions... but the limitations of mysql are rather drastic and make XA transactions on mysql a joke do be honest: both resume & join on start and end with suspend are not working (since 2006 when this was first released). So to answer you question no! No chance with mysql, forget it! Try increasing timeouts(both on client and server), memory pools, optimize the queries etc... mysql won't help you here.
I currently have a working program that is monitoring a few SQL tables and transferring data to MySQL tables. Essentially, I have a loop that checks every 30 seconds. My main concern is that I currently need to close and open the connection every time I loop. The reason for this is because I was getting errors about multiple transactions. When I close my connections, I also need to dispose the connection. I thought that disposing the transactions would have solved this problem, but I was still getting errors about multiple transactions.
This all seems to be working fine but I was wondering if there was a better way to do this without closing the connection.
I am not sure about your errors but it seems that you have to increase the number of connections to the remote computer. Have a look here http://msdn.microsoft.com/en-us/library/system.net.configuration.connectionmanagementelement.maxconnection.aspx
Also you can try to do is use only one connection to realize multiple SQLs.
If it is doesn't help then please provide your code to check it...
Were you committing your transactions in your loop? transaction.Commit() ... that could have been the issue... Hard to say with no code. No need to worry about opening and closing connections anyways since ADO.NET uses connection pooling behind the scenes. You only actually 'open' a connection the first time, after that is kept open in the pool to be used again. As others have said though, Post some code!
What are the rules for how a linq-to-sql datacontext keeps the database connection open?
The question came up when we made a few tests on performance for one SubmitChanges() per updated entity instead of one SubmitChanges() for the entire batch of entities. Results:
Inserting 3000 items in one SubmitChanges() call... Duration: 1318ms
Inserting 3000 items in one SubmitChanges() call, within
transactionscope... Duration: 1280ms
Inserting 3000 items in individual SubmitChanges() calls... Duration:
4377ms
Inserting 3000 items in individual SubmitChanges() calls within a
transaction... Duration: 2901ms
Note that when doing individual SubmitChanges() for each changed entity, putting everything within a transaction improves performance, which was quite unexpected to us. In the sql server profiler we can see that the individual SubmitChanges() calls within the transaction do not reset the DB connection for each call, as opposed to the one without the transaction.
In what cases does the data context keep the connection open? Is there any detailed documentation available on how linq-to-sql handles connections?
You aren't showing the entire picture; LINQ-to-SQL will wrap a call to SubmitChanges in a transaction by default. If you are wrapping it with another transaction, then you won't see the connection reset; it can't until all of the SubmitChanges calls are complete and then when the external transaction is committed.
There may be a number of factors that could be influencing the timings besides when connections are opened/closed.
edit: I've removed the bit about tracked entities after realizing how linq2sql manages the cached entities and the dirty entities separately.
You can get a good idea how the connections are managed under the covers by using Reflector or some other disassembler to examine the methods on the SqlConnectionManager class. SubmitChanges will call ClearConnection on its IProvider (typically SqlProvider which then uses SqlConnectionManager) after the submit if it wrapped the submit in its own transaction, but not if the SubmitChanges is part of a larger transaction. When the the connection is opened and closed depends on whether there is other activity making use of the SqlConnectionManager.
I messed about with this lately also. Calling SubmitChanges 3000 times will not be a good idea, but then depending on how critical it is that each record gets inserted, you may want to do this, after all it only takes 1000ms.
The transaction scope and multiple SubmitChanges is what i'd expect to see. Since your still within one transaction i'd expect to see SQL server handle this better, which it seems to. One SubmitChanges and using a explicit/implicit TransactionScope seems to yield the same result, which is to be expected. There shouldn't be any/much of a performance difference there.
I think connections are created when needed, but you have to remember this will be pooled within your provider so unless your connection string is changing, you should hook onto the same connection pool which will yield the same performance regardless of approach. Since LINQ-SQL uses SqlConnection behind the scenes, some information about it is at the following:
http://msdn.microsoft.com/en-us/library/8xx3tyca(VS.80).aspx
If your after brute force performance, look at moving into a Stored Proceedure for insert with an explicit TransactionScope. If that isn't fast enough, look at using SqlBulkCopy. 3000 rows should insert faster than 1000ms.
Have you tried opening and close the connection yourself:
Force the Opening of the DataContext's Connection (LINQ)
I think in that case you do not need the extra transaction.
We have a web service coded in C# that makes many calls to MS SQL Server 2005 database. The code uses Using blocks combined with C#'s connection pooling.
During a SQL trace, we saw many, many calls to "sp_resetconnection". Most of these are short < 0.5 sec, however sometimes we get calls lasting as much as 9 seconds.
From what I've read sp_resetconnection is related to connection pooling and basically resets the state of an open connection. My questions:
Why does an open connection need its state reset?
Why so many of these calls!
What could cause a call to sp_reset connection to take a non-trivial amount of time.
This is quite the mystery to me, and I appreciate any and all help!
The reset simply resets things so that you don't have to reconnect to reset them. It wipes the connection clean of things like SET or USE operations so each query has a clean slate.
The connection is still being reused. Here's an extensive list:
sp_reset_connection resets the following aspects of a connection:
It resets all error states and numbers (like ##error)
It stops all EC's (execution contexts) that are child threads of a parent EC executing a parallel query
It will wait for any outstanding I/O operations that is outstanding
It will free any held buffers on the server by the connection
It will unlock any buffer resources that are used by the connection
It will release all memory allocated owned by the connection
It will clear any work or temporary tables that are created by the connection
It will kill all global cursors owned by the connection
It will close any open SQL-XML handles that are open
It will delete any open SQL-XML related work tables
It will close all system tables
It will close all user tables
It will drop all temporary objects
It will abort open transactions
It will defect from a distributed transaction when enlisted
It will decrement the reference count for users in current database; which release shared database lock
It will free acquired locks
It will releases any handles that may have been acquired
It will reset all SET options to the default values
It will reset the ##rowcount value
It will reset the ##identity value
It will reset any session level trace options using dbcc traceon()
sp_reset_connection will NOT reset:
Security context, which is why connection pooling matches connections based on the exact connection string
If you entered an application role using sp_setapprole, since application roles can not be reverted
The transaction isolation level(!)
Here's an explanation of What does sp_reset_connection do? which says, in part "Data access API's layers like ODBC, OLE-DB and SqlClient call the (internal) stored procedure sp_reset_connection when re-using a connection from a connection pool. It does this to reset the state of the connection before it gets re-used." Then it gives some specifics of what that system sproc does. It's a good thing.
sp_resetconnection will get called everytime you request a new connection from a pool.
It has to do this since the pool cannot guarantee the user (you, the programmer probably :)
have left the connection in a proper state. e.g. Returning an old connection with uncommited transactions would be ..bad.
The nr of calls should be related to the nr of times you fetch a new connection.
As for some calls taking non-trivial amount of time, I'm not sure. Could be the server is just very busy processing other stuff at that time. Could be network delays.
Basically the calls are the clear out state information. If you have ANY open DataReaders it will take a LOT longer to occur. This is because your DataReaders are only holding a single row, but could pull more rows. They each have to be cleared as well before the reset can proceed. So make sure you have everything in using() statements and are not leaving things open in some of your statements.
How many total connections do you have running when this happens?
If you have a max of 5 and you hit all 5 then calling a reset will block - and it will appear to take a long time. It really is not, it is just blocked waiting on a pooled connection to become available.
Also if you are running on SQL Express you can get blocked due to threading requirements very easily (could also happen in full SQL Server, but much less likely).
What happens if you turn off connection pooling?