Due to a few restrictions I can't use entity Framework and thus need to use SQL Connections, commands and Transactions manually.
While writing unit tests for the methods calling these data layer operations I stumbled upon a few problems.
For the unit tests I NEED to do them in a Transaction as most of the operations are changing data by their nature and thus doing them outside a Transaction is problematic as that would change the whole base data. Thus I need to put a Transaction around these (with no commit fired at the end).
Now I have 2 different variants of how These BL methods work.
A few have Transactions themselves inside of them while others have no Transactions at all. Both of these variants cause problems.
Layered Transaction: Here I get errors that the DTC cancelled the distributed Transaction due to timeouts (although the timeout is being set to 15 minutes and it is running for only 2 minutes).
Only 1 Transaction: Here I get an error about the state of the Transaction when I come to the "new SQLCommand" line in the called method.
My question here is what can I do to correct this and get unit testing with manual normal and layered Transactions working?
Unit testing method example:
using (SqlConnection connection = new SqlConnection(Properties.Settings.Default.ConnectionString))
{
connection.Open();
using (SqlTransaction transaction = connection.BeginTransaction())
{
MyBLMethod();
}
}
Example for a Transaction using method (very simplified)
using (SqlConnection connection = new SqlConnection(Properties.Settings.Default.ConnectionString))
{
connection.Open();
using (SqlTransaction transaction = connection.BeginTransaction())
{
SqlCommand command = new SqlCommand();
command.Connection = connection;
command.Transaction = transaction;
command.CommandTimeout = 900; // Wait 15 minutes before a timeout
command.CommandText = "INSERT ......";
command.ExecuteNonQuery();
// Following commands
....
Transaction.Commit();
}
}
Example for a non Transaction using method
using (SqlConnection connection = new SqlConnection(Properties.Settings.Default.ConnectionString))
{
connection.Open();
SqlCommand command = new SqlCommand();
command.Connection = connection;
command.CommandTimeout = 900; // Wait 15 minutes before a timeout
command.CommandText = "INSERT ......";
command.ExecuteNonQuery();
}
On the face of it, you have a few options, depending upon what you want to test and your ability to spend money / change your code base.
At the moment, you’re effectively writing integration tests. If the database isn’t available then your tests will fail. This means the tests can be slow, but on the plus side if they pass you’re pretty confident that you code can hit the database correctly.
If you don’t mind hitting the database, then the minimum impact to changing your code / spending money would be for you to allow the transactions to complete and verify them in the database. You can either do this by taking database snapshots and resetting the database each test run, or by having a dedicated test database and writing your tests in such a way that they can safely hit the database over and over again and then verified. So for example, you can insert a record with an incremented id, update the record, and then verify that it can be read. You may have more unwinding to do if there are errors, but if you’re not modifying the data access code or the database structure that often then this shouldn’t be too much of an issue.
If you’re able to spend some money and you want to actually turn your tests into unit tests, so that they don’t hit the database, then you should consider looking into TypeMock. It’s a very powerful mocking framework that can do some pretty scary stuff. I believe it using the profiling API to intercept calls, rather than using the approach used by frameworks like Moq. There's an example of using Typemock to mock a SQLConnection here.
If you don’t have money to spend / you’re able to change your code and don’t mind continuing to rely on the database then you need to look at some way to share your database connection between your test code and your dataaccess methods. Two approaches that spring to mind are to either inject the connection information into the class, or make it available by injecting a factory that gives access to the connection information (in which case you can inject a mock of the factory during testing that returns the connection you want).
If you go with the above approach, rather than directly injecting SqlConnection, consider injecting a wrapper class that is also responsible for the transaction. Something like:
public class MySqlWrapper : IDisposable {
public SqlConnection Connection { get; set; }
public SqlTransaction Transaction { get; set; }
int _transactionCount = 0;
public void BeginTransaction() {
_transactionCount++;
if (_transactionCount == 1) {
Transaction = Connection.BeginTransaction();
}
}
public void CommitTransaction() {
_transactionCount--;
if (_transactionCount == 0) {
Transaction.Commit();
Transaction = null;
}
if (_transactionCount < 0) {
throw new InvalidOperationException("Commit without Begin");
}
}
public void Rollback() {
_transactionCount = 0;
Transaction.Rollback();
Transaction = null;
}
public void Dispose() {
if (null != Transaction) {
Transaction.Dispose();
Transaction = null;
}
Connection.Dispose();
}
}
This will stop nested transactions from being created + committed.
If you’re more willing to restructure your code, then you might want to wrap your dataaccess code in a more mockable way. So, for example you could push your core database access functionality into another class. Depending on what you’re doing you’ll need to expand on it, however you might end up with something like this:
public interface IMyQuery {
string GetCommand();
}
public class MyInsert : IMyQuery{
public string GetCommand() {
return "INSERT ...";
}
}
class DBNonQueryRunner {
public void RunQuery(IMyQuery query) {
using (SqlConnection connection = new SqlConnection(Properties.Settings.Default.ConnectionString)) {
connection.Open();
using (SqlTransaction transaction = connection.BeginTransaction()) {
SqlCommand command = new SqlCommand();
command.Connection = connection;
command.Transaction = transaction;
command.CommandTimeout = 900; // Wait 15 minutes before a timeout
command.CommandText = query.GetCommand();
command.ExecuteNonQuery();
transaction.Commit();
}
}
}
}
This allows you to unit test more of your logic, like the command generation code, without having to actually worry about hitting the database and you can test your core dataaccess code (the Runner) against the database once, rather than for every command you want to run against the database. I would still write integration tests for all dataaccess code, but I’d only tend to run them whilst actually working on that section of code (to ensure column names etc have been specified correctly).
Related
I am writing SQL Server integration tests using xUnit.
My tests look like this:
[Fact]
public void Test()
{
using (IDbConnection connection = new SqlConnection(_connectionString))
{
connection.Open();
connection.Query(#$"...");
//DO SOMETHING
}
}
Since I am planning to create multiple tests in the same class I was trying to avoid creating a new SqlConnection every time. I know that xUnit framework per se creates an instance of the class per each running test inside it.
This means I could make the creation of the SqlConnection in the constructor since anyway a new one will be created every time a new test run.
The problem is disposing it. Is it good practice or do you see any problem in disposing manually the SqlConnection at the end of each test?
Such as:
public class MyTestClass
{
private const string _connectionString = "...";
private readonly IDbConnection _connection;
public MyTestClass()
{
_connection = new SqlConnection(_connectionString));
}
[Fact]
public void Test()
{
_connection.Open();
_connection.Query(#$"...");
//DO SOMETHING
_connection.Dispose();
}
}
I was trying to avoid creating a new SqlConnection every time.
It's okay to create and dispose a new SqlConnection over and over. The connection instance is disposed, but the underlying connection is pooled. Under the hood it may actually use the same connection repeatedly without closing it, and that's okay.
See SQL Server Connection Pooling.
So if that's your concern, don't worry. What you are doing in the first example is completely harmless. And however your code is structured, if you're creating a new connection when you need it, using it, and disposing it, that's fine. You can do that all day long.
If what concerns you is the repetitive code, I find this helpful. I often have a class like this in my unit tests:
static class SqlExecution
{
public static void ExecuteSql(string sql)
{
using (var connection = new SqlConnection(GetConnectionString()))
{
using (var command = new SqlCommand(sql, connection))
{
connection.Open();
command.ExecuteNonQuery();
}
}
}
public static T ExecuteScalar<T>(string sql)
{
using (var connection = new SqlConnection(GetConnectionString()))
{
using (var command = new SqlCommand(sql, connection))
{
connection.Open();
return (T)command.ExecuteScalar();
}
}
}
public static string GetConnectionString()
{
// This may vary
}
}
How you obtain the connection string may vary, which is why I left that method empty. It might be a configuration file, or it could be hard-coded.
This covers the common scenarios of executing something and retrieving a value, and allows me to keep all the repetitive database code out of my tests. Within the test is just one line:
SqlExecution.ExecuteSql("UPDATE WHATEVER");
You can also use dependency injection with xUnit, so you could write something as an instance class and inject it. But I doubt that it's worth the effort.
If I find myself writing more and more repetitive code then I might add test-specific methods. For example the test class might have a method that executes a certain stored procedure or formats arguments into a query and executes it.
You can also do dependency injection with xUnit so that dependencies are injected into the class like with any other class. There are lots of answers about this. You may find it useful. Maybe it's the reason why you're using xUnit. But something simpler might get the job done.
Someone is bound to say that unit tests shouldn't talk to the database. And they're mostly right. But it doesn't matter. Sometimes we have to do it anyway. Maybe the code we need
to unit test is a stored procedure, and executing it from a unit test and verifying the
results is the simplest way to do that.
Someone else might say that we shouldn't have logic in the database that needs testing, and I agree with them too. But we don't always have that choice. If I have to put logic in SQL I still want to test it, and writing a unit test is usually the easiest way. (We can call it an "integration test" if it makes anyone happy.)
There's a couple different things at play here. First, SqlConnection uses connection pooling so it should be fine to create/dispose the connections within a single test.
Having said that disposing of the connections on a per test class basis would also be doable. XUnit will dispose of test classes that are IDisposable. So either would work. I'd suggest it's cleaner to create/dispose them within the test.
public class MyTestClass : IDisposable
{
const string ConnectionStr = "";
SqlConnection conn;
public MyTestClass()
{
this.conn = new SqlConnection(ConnectionStr);
}
public void Dispose()
{
this.conn?.Dispose();
this.conn = null;
}
public void Test()
{
using (var conn = new SqlConnection(ConnectionStr))
{
}
}
}
i'm stuck within a problem that is going to happen to everyone in ADO.NET mixed with Entity Framework contexts.
I've got a big procedure that handles and saves data into database by using multiple ways like ADO.NET dataadapters and direct CRUD commands against DB. All the procedure is wrapped by 2 using() blocks that creates and releases a DBConnection/DBTransaction and a try/catch block to commit or rollback the transaction. Unfortunately, in the middle of this routine, i have to recall a saving procedure implemented by using Entity Framework. This leads me to a problem:
According to the official documentation, Entity Framework 5 allows me to pass a connection with an transaction associated transaction (it should work, in debug mode, when i call SaveChanges() i don't receive any TimeOutException due to deadlocks, conversely if i pass a new connection it does), but unfortunately after SaveChanges() kicks in the connection is closed and associated transaction committed! Even if i set the flag 'contextOwnsConnection'!
As far as i know, if i migrate EF5 to EF6, things should work (am i right?), but unfortunately i can't, because the project i'm working on is very large and involves a lot of dependecies and it would take a large amount of time.
How can i make it work with EF5? Is there any trick or pattern to achieve the desired result? Am i right about the behavior of EF6? Does it worth EF6 migration?
Here you are a simple example of how does my code looks like.
For privacy reasons i can't post the original code but just imagine a situation like this with a lot of more complexity:
using(DbConnection conn = DBProvider.CreateConnection()){
//Open the created connection
conn.Open();
//Create a new transaction
using(DbTransaction tr = DBProvider.CreateTransaction()){
//Begin a new transaction
tr.Begin();
bool saveOk;
try{
//Updates customers by using dataadapter
dataAdapterCustomers.InsertCommand.Transaction = tr;
dataAdapterCustomers.UpdateCommand.Transaction = tr;
dataAdapterCustomers.DeleteCommand.Transaction = tr;
dataAdapterCustomers.Update();
//Updates stock items by using dataadapter
stockAdapterCustomers.InsertCommand.Transaction = tr;
stockAdapterCustomers.UpdateCommand.Transaction = tr;
stockAdapterCustomers.DeleteCommand.Transaction = tr;
stockAdapterCustomers.Update();
//...Many other DB accessing here...
//Updates stock quantity by using simple DBCommand
quantityUpdateCmd.Transaction = tr;
quantityUpdateCmd.ExecuteNonQuery();
//Updates stock statistics by using a simple DBCommand
updateStockStatsCmd.Transaction = tr;
updateStockStatsCmd.ExecuteNonQuery();
//...Many other DB accessing here...
//HERE:
//Creates a new activity and save it using EF.
//I use a UnitOfWork and i pass to it my connection and 'false' as contextOwnsConnection parameter
//(it 'll be used by the DBContext contained in my Unit of work)
using(ActivityUoW uow = new ActivityUoW(conn, false)){
Activity act = new Activity();
act.Name = "Saving activity";
act.Description = "Done by user";
act.Date = DateTime.Now;
uow.Activities.Add(act);
uow.SaveChanges();
}
//Based on activity result, launch a store procedure that makes other complex things.
UNFORTUNATELY THE CONNECTION HAS BEEN CLOSED AND TRANSACTION COMMITTED, SO THE FOLLOWING INSTRUCTION WILL FAIL.
launchActivityUpdateSpCmd.Transaction = tr;
launchActivityUpdateSpCmd.ExecuteNonQuery();
//...Many other DB accessing here...
//Data saved correctly
saveOk = true;
}
catch(Exception ex){
//There was an error during save
saveOk = false;
}
//Commit or rollback transaction according to save procedure result
if(saveOk)
tr.Commit();
else
tr.Rollback();
}
}
I didn't quite follow your question(s) and wasn't sure if your issue was related to how to handle transactions or if you had a question about EF5 to EF6 migrations. That being said, you have an interesting mixture of data access code.
Regarding transactions - I would look into using the TransactionScope which is part of the System.Transactions namespace.
For example:
try
{
using (var scope = new TransactionScope())
{
using (var conn = new SqlConnection("your connection string"))
{
conn.Open();
// your EF and ADO.NET code
}
scope.Complete();
}
}
catch (TransactionAbortedException ex)
{
}
catch (ApplicationException ex)
{
}
I'm writing a very simple web application that serves as an endpoint for uploading money transactions from customers and saving them in SQL Server DB. It accepts requests with just 2 params: userid: 'xxx', balancechange: -19.99. If the user ID exists in the app database, then the balance is changed; if not - a new row is created for this ID.
The difficult part in all this is that the numer of requests is enormous and I have to implement the app in such a way that it works as fast as possible and resolves concurrency issues (if 2 requests for the same ID arrive simultaneously).
The app is a ASP.NET MVC WebAPI. I chose to use plain old ADO.NET for speed, and this is what I currently have:
private static readonly object syncLock = new object();
public void UpdateBalance(string userId, decimal balance)
{
lock (syncLock)
{
using (var sqlConnection = new SqlConnection(this.connectionString))
{
var command = new SqlCommand($"SELECT COUNT(*) FROM Users WHERE Id = '{userId}'", sqlConnection);
if ((int)command.ExecuteScalar() == 0)
{
command = new SqlCommand($"INSERT INTO Users (Id, Balance) VALUES ('{userId}', 0)", sqlConnection);
command.ExecuteNonQuery();
}
command = new SqlCommand($"UPDATE Users SET Balance = Balance + {balance} WHERE Id = {userId}", sqlConnection);
command.ExecuteNonQuery();
}
}
}
Called from a controller like this:
[HttpPost]
public IHttpActionResult UpdateBalance(string id, decimal balanceChange)
{
UpdateBalance(id, balanceChange);
return Ok();
}
The thing I'm concernred with is concurrency control using lock (syncLock). This would slow the app down under high load and doesn't allow multiple instances of the app to be deployed on different servers. What are ways to properly implement concurrency control here?
Note: I'd like to use a fast and DB-independent way of implementing concurrency control, as the current storage mechanism (SQL Server) can change in the future.
First, DB-independent code:
For this, you will want to look at DbProviderFactory. What this allows is passing the provider name (MySql.Data.MySqlClient, System.Data.SqlClient), then using the abstract classes (DbConnection, DbCommand) you interact with your DB.
Second, using transactions and paramaterized queries:
When you are working with a database, you ALWAYS want to have your queries paramaterized. If you use String.Format() or any other type of string concatenation, you open your query up to injection.
Transactions ensure all or nothing with your queries, and they can also lock down the table so that only queries within the transaction can access those tables. Transactions have two commands, Commit which will save the changes (if any) to the DB, and Rollback which discards any changes to the DB.
The following will assume that you already have an instance of DbProviderFactory in a class variable _factory.
public void UpdateBalance(string userId, decimal balanceChange)
{
//since we might need to execute two queries, we will create the paramaters once
List<DbParamater> paramaters = new List<DbParamater>();
DbParamater userParam = _factory.CreateParamater();
userParam.ParamaterName = "#userId";
userParam.DbType = System.Data.DbType.Int32;
userParam.Value = userId;
paramaters.Add(userParam);
DbParamater balanceChangeParam = _factory.CreateParamater();
balanceChangeParam.ParamaterName = "#balanceChange";
balanceChangeParam.DbType = System.Data.DbType.Decimal;
balanceChangeParam.Value = balanceChange;
paramaters.Add(balanceChangeParam);
//Improvement: if you implement a method to clone a DbParamater, you can
//create the above list in class construction instead of function invocation
//then clone the objects for the function.
using (DbConnection conn = _factory.CreateConnection()){
conn.Open(); //Need to open the connection before you start the transaction
DbTransaction trans = conn.BeginTransaction(System.Data.IsolationLevel.Serializable);
//IsolationLevel.Serializable locks the entire table down until the
//transaction is commited or rolled back.
try {
int changedRowCount = 0;
//We can use the fact that ExecuteNonQuery will return the number
//of affected rows, and if there are no affected rows, a
//record does not exist for the userId.
using (DbCommand cmd = conn.CreateCommand()){
cmd.Transaction = trans; //Need to set the transaction on the command
cmd.CommandText = "UPDATE Users SET Balance = Balance + #balanceChange WHERE Id = #userId";
cmd.Paramaters.AddRange(paramaters.ToArray());
changedRowCount = cmd.ExecuteNonQuery();
}
if(changedRowCount == 0){
//If no record was affected in the previous query, insert a record
using (DbCommand cmd = conn.CreateCommand()){
cmd.Transaction = trans; //Need to set the transaction on the command
cmd.CommandText = "INSERT INTO Users (Id, Balance) VALUES (#userId, #balanceChange)";
cmd.Paramaters.AddRange(paramaters.ToArray());
cmd.ExecuteNonQuery();
}
}
trans.Commit(); //This will persist the data to the DB.
}
catch (Exception e){
trans.Rollback(); //This will cause the data NOT to be saved to the DB.
//This is the default action if Commit is not called.
throw e;
}
finally {
trans.Dispose(); //Need to call dispose
}
//Improvement: you can remove the try-catch-finally block by wrapping
//the conn.BeginTransaction() line in a using block. I used a try-catch here
//so that you can more easily see what is happening with the transaction.
}
}
The goal is simple - rollback data inserted by a unit test. Here is how it goes. In a unit test, a method is called that creates a new connection and inserts some data. After that a unit test creates a new connection and tries to find what has been inserted and assert that. I was hoping to wrap these two things with TransactionScope, not call Complete and see inserted data rolled back. That's not happening. Am I doing something wrong or I am just missing the point?
using (new TransactionScope())
{
// call a method that inserts data
var target = new ....
target.DoStuffAndEndupWithDataInDb();
// Now assert what has been added.
using (var conn = new SqlConnection(connectionString))
using (var cmd = conn.CreateCommand())
{
// Just read the data from DB
cmd.CommandText = "SELECT...";
conn.Open();
int count = 0;
using (var rdr = cmd.ExecuteReader())
{
// Read records here
...
count++;
}
// Expecting, say, 3 records here
Assert.AreEqual(3, count);
}
}
EDIT: I don't think I had DTC running and configured on my machine. So I started the service and tried to configure DTC but I am getting this error.
are you using MSTest ? then you can use MsTestExtensions
you unit test needs to derive from MSTestExtensionsTestFixture and your test needs to have TestTransaction Attribute, it uses AOP to automatically start a transaction and roll it back.
I don't think you're missing the point but just attacking the problem incorrectly.
In NUnit terms, the concepts are [SetUp] and [TearDown] methods. You've already defined the setup method in your description and your tear down method should just undo what the setup method did (assuming what you're unit testing has no residual side effects).
Do you have Distributed Transaction Coordinator properly configured? This is a big gotcha when trying to use TransactionScope like this... if it isn't configured, sometimes you'll get an error, but other times the transaction will just commit and not rollback.
I'd recommend looking at this article, which shows you all the various steps that need to be done in order to rollback your unit tests using MSDTC.
Your code should work as you expect. How are you adding data in DoStuffAndEndupWithDataInDb()? I'm wondering whether the data initialization is not participating in the transaction.
For reference, the following console application correctly outputs 3 rows, and does not commit the rows to the database (checked using SSMS).
public class Program
{
private static void Main(string[] args)
{
using (var trx = new TransactionScope())
{
InitializeData();
using (var connection = new SqlConnection("server=localhost;database=Test;integrated security=true"))
using (var command = connection.CreateCommand())
{
command.CommandText = "select count(*) from MyTable";
connection.Open();
Console.WriteLine("{0} rows", command.ExecuteScalar());
}
}
Console.ReadLine();
}
private static void InitializeData()
{
using (var connection = new SqlConnection("server=localhost;database=Test;integrated security=true"))
using (var command = connection.CreateCommand())
{
command.CommandText = "insert into MyTable values (1),(2),(3)";
connection.Open();
command.ExecuteNonQuery();
}
}
}
Which pattern is better for SqlConnection object? Which is better in performance?
Do you offer any other pattern?
class DataAccess1 : IDisposable
{
private SqlConnection connection;
public DataAccess1(string connectionString)
{
connection = new SqlConnection(connectionString);
}
public void Execute(string query)
{
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = query;
command.CommandType = CommandType.Text;
// ...
command.Connection.Open();
command.ExecuteNonQuery();
command.Connection.Close();
}
}
public void Dispose()
{
connection.Dispose();
}
}
VS
class DataAccess2 : IDisposable
{
private string connectionString;
public DataAccess2(string connectionString)
{
this.connectionString = connectionString;
}
public void Execute(string query)
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
SqlCommand command = connection.CreateCommand();
command.CommandText = query;
command.CommandType = CommandType.Text;
// ...
command.Connection.Open();
command.ExecuteNonQuery();
command.Connection.Close();
}
}
public void Dispose()
{
}
}
There's no real way to answer this question. The short, canonical answer is that the connection should stay alive for the lifetime of your unit of work. Because we have no way of knowing how DataAccess is used (does it exist for the lifetime of your application, or do you instantiate it and dispose it whenever you do something?), it's impossible to give a concrete answer.
That being said, I would recommend the first pattern, but instantiate and dispose of your DataAccess object as needed; don't keep it around longer than necessary.
Suggest going with DataAccess2. It's a personal preference though. Some might even suggest your class be static. It'd be difficult to say that one is more performant than the other. You're on the path of IDisposable, which is great.
I'd be happy to read and maintain both styles shown above in your question.
Consider having your DAL be able to read the connection string from a .config as well, rather than exclusively allowing the value to be passed in the constructor.
public DataAccess2(string connStr)
{
this.connectionString = connStr;
}
public DataAccess2()
{
this.connectionString =
ConfigurationManager.ConnectionStrings["foo"].ConnectionString;
}
Consider wrapping your SqlCommand in a using as well.
using (var conn = new SqlConnection(connectionString))
{
using(var cmd = conn.CreateCommand())
{
}
}
I think it depends on how your DataAccess object is intended to be used, if it's used within a 'using' clause then the connection is guaranteed to be disposed of after it's done.
But in general I prefer the second pattern as the sql connection is created and disposed of within the Execute method so it's less likely to be left open when you forget to dispose of your DataAccess object.
Considering that sql connection can be a scarse resource I think every attempt should be made to ensure that they're not wasted.
The first will result in errors if you make concurrent calls.
The second will ensure you use a clean connection for each command resulting in more connections being made.
I agree with the statements above that it depends on the scenario for use, to get over the problem related to the first I have a wrapper that needs to use such a pattern so I set a field value boolean to show that a command is being executed on the connection already then "queue" the next command for execution.
There will of course be situations where you may prefer to use multiple connections ...