I realize that there is no way to atomically guarantee:
if(hasInternet)
doDatabaseCall();
However, what is the correct way of handling connection problems when dealing with DbConnection and DbCommand objects in the .NET world? I'm specifically interested in the MySqlConnection and MySqlCommand objects but assume (hope) its pattern of exceptions is the same as SQL Server's classes.
I'd assume that if the internet goes down before calling, conn.Open(), a MySqlException gets raised. I'd also assume the same happens if you call ExecuteReader or ExecuteNonQuery and the internet has gone down.
I'm not sure because the docs for the MySql objects don't say. The SqlServer objects just say that it might raise a SqlException which means:
An exception occurred while executing the command against a locked row. This exception is not generated when you are using Microsoft .NET Framework version 1.0.
That doesn't seem to cover connection issues... What I'd like to do is handle the exception, wait for some amount of time, and start over again. My application's sole purpose is to execute these database calls and its running on a dedicated system so retrying forever is really the best option I believe. That said, I would love to differentiate between connection exceptions and other kinds of database exceptions, is that possible?
I've done some testing and it appears to work as I assume but does it work in all edge cases? Such as: the command was successfully sent to the database server but the connection goes down before or while the results are being returned? If it doesn't work in all edge cases then I'm going to have to: execute a command, query for the desired state change, execute the next command, etc. It's important that each command goes through.
I am connecting to a port on localhost that is forwarded via SSH to a remote server if that makes a difference.
As for the SqlDataProvider:
The SqlException exception has a several properties that give you detailed information why your operation failed.
For your use case the "Class" property might be a good choice. It's a byte indicating the severity of the exception.
See: http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlexception.class.aspx
If that is not specific enough, you can examine that individual errors in the Errors collection.
http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlexception.errors.aspx
Based on that information you can decide whether to retry.
Related
My program makes use of NServiceBus as the service bus.
Now, when I run a some part of my program, it fires a command to initiate a process. This process involves data look up from a database (with A lot of data) by 3 separate handlers (classes) in the program. So, in some way, they are happening in parallel. As these 3 classes receives and handles the same command, then starts work
Searching through similar posts on stack overflow, I've come across a number of suggestions. Including 'increasing the timeout time' in both the config and machine.config. Done this to know avail.
This post! made me realise it could be an issue with NServiceBus and MSDTC.
I've also attached visual studio debugger to the program process and witnessed the exception taking place at every point where I'm querying a repository class - which queries the database.
System.Transactions.TransactionException occurred
HResult=-2146233087
Message=The operation is not valid for the state of the transaction.
Source=System.Transactions
StackTrace:
at
System.Transactions.TransactionState.EnlistVolatile(InternalTransaction
tx, IEnlistmentNotification enlistmentNotification, EnlistmentOptions
enlistmentOptions, Transaction atomicTransaction)
InnerException:
I'm tempted to just have a try catch everywhere. But that's me getting desperate. And, I'm ignoring a lot of data.
Please, any ideas?
All response will be appreciated.
I wonder whether there is a way to determine the Connection ID and LocalEndpoint's port used by MySql Connection on the C# side.
I know that I can run:
select * from information_schema.processlist
where id = connection_id();
However, my question is whether I can get these details (or at least the connection ID) without executing this SQL statement, that is, right from the Connection object itself (as this information DOES exist within the connection's underlying type and TCP socket)
Thanks :)
The documentation for the connection_id() function says that it returns the thread ID of the current connection. If you check the documentation for MySqlConnection you may find that the MySqlConnection equivalent of ClientConnectionId is ServerThread - I have done some testing & found that it contains the same value as returned by the query - but that may not be guaranteed.
I have not found a way of determining the TCP port from publicly accessible properties/methods.
The connection string is parsed into Properties of the Connection class. It might have ceased existing once it was processed. Look at the Propeties you have availible: https://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlconnection.aspx#Properties
As for getting Connection details - that is problematic. Internally the SQLConnection class is doing Connection Pooling. So the same low-level connection might be re-used for a request that comes soon after. Like Thread ID's, it is the kind of thing you should not really access in your code.
These values could change even during execution without any obvious forwarning, rhyme or reason (there is still a reason, but it is hard to even get a decent guess). The Behavior Might change between Framework Versions, Specific Circumstances in that Millisecond, or even if your Grandmother visits or wheter you have a full moon or not today.
Morning all,
I am working on a project where I am saving data to Mongo DB and also to Sql Server (Entity Framework code first). There are a number of scenarios where I carry out a write to each database in a single block of code. I'm wondering, can any of you suggest a way to handle this in something similar to a transaction ? such that if any part of the code block fails, the whole thing should fail and rollback ?
I don't think there's any bullet-proof way of doing this since you not only have two separate connections but also two different architectures.
Assuming your SQL Server has the more complicated data model (the one that's more likely to fail for some reason) I came up with an approach in the past that worked for me:
Execute the operations sequentially not both at the same time
Execute the SQL satement first, if it fails don't execute the MongoDB statement and you'll be consistent
Should it succeed, execute the MongoDB statement next
If the MongoDB statement fails write an error log. Make sure the log is not on a remote machine so that the possibility that the logging could fail is as small as possible.
You can later use the error log to either manually or automatically salvage your data. In any case you should implement a retry policy for all statements, since the most likely reason for a failed operation is (given your code is correct) a timing issue and retrying solves this in general. If you're doing it right there will be maybe like one exotic error a month.
Of course in step 4 you could try to revert the SQL operation, instead of (or in addition to) writing a log entry. But this is mostly cumbersome and leaves you with the question what to do should the revert fail.
Maybe there still is some magic middleware to integrate MongoDB statements into an SQL transaction but for the time being I would just acknowledge that data consistency and using different databases are opposing goals. Try to monitor errors closely and reduce the error potential.
Two Phase Commit will suit your scenario. In a single transaction we can hit any number(normally we use two) of DB's and maintain our Data synchronized across the DB's.
More info on Two Phase Commit
https://lostechies.com/jimmybogard/2013/05/09/ditching-two-phased-commits/
https://www.coursera.org/learn/data-manipulation/lecture/mXS0H/two-phase-commit-and-consensus-protocols
https://sankarsan.wordpress.com/tag/two-phase-commit/
Read this post
How to force only one transaction within multiple DbContext classes?
I'm hitting weird behavior under load testing scenario: backend (sql server 2012) is getting overloaded and some of the commands times out (this is still expected as the backend server is half-intentionally slow HW); but our platform is regularly (with increasing delay) retrying the timeouted operation - and after few retries it suddenly starts receiving 'cannot insert duplicate key' SqlException.
I verified that only a single row with a specific unique key can be generated and is attempted to be inserted (first insert and all possible retries always happens on the same thread).
I also altered the SP so that it uses explicit transaction:
BEGIN TRY
BEGIN TRANSACTION;
-- Insert into table A
-- Insert into table B
COMMIT TRANSACTION;
END TRY
BEGIN CATCH
ROLLBACK TRANSACTION;
THROW
END CATCH
Yet the issue is still occurring.
Are there any ideas why this can be happening?
How can I find out where is the timeout coming from (backend vs. client side)?
Is there a way how to make sure that the operation either successful finishes or fails (basically transaction - but probably from client side code)?
EDIT01:
I believe one way of solving this is leverage of ado.net integration of SQL server distributed transactions - e.g:
using (TransactionScope scope = new TransactionScope())
{
//Perform the sql commands
//if above statements throws (e.g. due to timeout) - than the transaction is not commited and it will be rolled back
scope.Complete()
}
HOWEVER: I agree that it only adds complexity and actuly might be still object to the same problem (Two Generals problem as outlined by usr).
The best approach therefore is likely to code client and server side to count on such an option - again as noted by usr in his answer
This is expected behavior. When the communication between the client and the server is interrupted the client does not know the result of the operation. It might never have been sent, or it was sent but not received, or it was received but failed, or it was received but the success response did not come through.
This is the Two Generals Problem. It is unsolvable (when defining it strictly).
You must work around it. Either check for existence before insert or handle the duplicate key exception.
Or, simply increase the timeout. It does not do you any good to abort an otherwise working command that would have succeeded eventually. Aborting and restarting it does not make it go faster (except by coincidence). Timeouts are mostly useful for network errors or run-away queries (bugs).
Per the documentation SqlException Class.
The exception that is thrown when SQL Server returns a warning or
error. This class cannot be inherited.
My experience is that if you got a SQL exception then it came from SQL.
Timeout is a SQL setting. You can set it SSMS.
OK I am now believe usr +1 that this comes from the client side
SqlCommand.CommandTimeout
Gets or sets the wait time before terminating the attempt to execute a
command and generating an error.
That just is not how I would want it to be implemented.
You could have a crazy query and lose connection SQL would just keep on running.
I have a windows service which receives messages via RabbitMQ, this triggers an event handler which does some work and then attempts to persist the result to the database. It's threaded using:
ThreadPool.QueueUserWorkItem(ProcessMessageOnThread, messageReceived);
where ProcessMessageOnThread is a method which does the work on the messageReceived which is a representation of the message dequeued from RabbitMQ.
Under normal circumstances the windows service operated as expected, that is dequeue, process and persist.
I want to ensure that all of my messages are processed and given a fair change to be processed so if I can't open a connection to SQL Server I simply requeue the message for it to be processed again (hopefully that time the SQL Server will be back, otherwise this continues - and I'm fine with that).
Now the problem comes when the process has been running as expected for a period of time, the SQL Server connection pool has filled up and then SQL Server is disconnected, now this is when things get a bit unstable.
One of two things can happen:
An exception is thrown on connection.Open() - however I'm catching this and so not worried about it
An exception is thrown on cmd.ExecuteNonQuery() - which is where I'm executing a stored procedure
It is the second option that I need to figure out how to handle. Previously I assumed that any exception here meant that there was a problem with the data I was passing into the stored procedure and therefore should just move it out of the queue and have something else analyse it.
However, now I think I need a new approach to handle the cases where the exception is to do with the connection not actually being established.
I've had a look at the SqlException class and noticed a property called Class which has this description Gets the severity level of the error returned from SQL Server, now the info on this says:
Messages with a severity level of 10 or less are informational and indicate problems caused by mistakes in information that a user has entered. Severity levels from 11 through 16 are generated by the user, and can be corrected by the user. Severity levels from 17 through 25 indicate software or hardware errors. When a level 17, 18, or 19 error occurs, you can continue working, although you might not be able to execute a particular statement.
Does this mean to fix my exception handling I can just check if (ex.Class > 16) then requeue message because the problem is with the connection else throw it away as it is most likely to do with malformed data being send to the stored procedure?
So the question is, how should I do exception handling and how can I detect when calling cmd.ExecuteNonQuery() if the exception thrown is because of a disconnected connection.
Update:
I've experienced problems previously with connections not being returned to the pool (this was due to threading issues) and have fixed those problems, so I'm confident the issue isn't to do with connections not going back into the pool. Also, the logic around what the connections are being used for is so simple also I'm ensuring they are closed consistently...so I'm more interested in answers to do with the disconnection of the Sql Server and then the capturing the behaviour of cmd.ExecuteNonQuery()
Connections in the connection pool can get into a weird state for various reasons, all of which have to do with poor application design:
Closing the connection before its associated data reader
Change a setting (like transaction isolation level) that the pool does not reset
Starting an asynchronous query (BeginOpenReader) and then returning the connection to the pool before the asynchronous handler fires
You should investigate your application and make sure connections are properly returned to the pool. One thing that can help debugging is reducing the size of the application pool in a development setting. You change the size of the pool in the connection string:
...;Integrated Security=SSPI;Max Pool Size=2;Pooling=True;
This makes pooling issues much easy to reproduce.
If you can't find the cause, but still need to deploy a fix, you could use one of ClearPool or ClearAllPools. A good place to do that is when you detect one of the suspicious exceptions after Open() or ExecuteNonQuery(). Both are static methods on the SqlConnection class:
SqlConnection.ClearPool(yourConnection);
Or for an even rougher approach:
SqlConnection.ClearAllPools()
Note that this is basically Pokémon Exception Handling. If it works, you'll have no idea why. :)