Unit Testing Amazon S3 - c#

I have a fairly simple class that I'm trying to unit test. I'm very new to unit testing in general, and I'm not sure what I should be testing here.
The only test case that I can figure out how to code is a null argument of stream. Besides that, I'm not sure how to test the results of a PutObjectRequest or what else. If I should be using mocks here, how?
public class AmazonS3Service : IAmazonS3Service
{
private readonly Uri baseImageUrl;
private readonly Uri s3BaseUrl;
private readonly string imageBucket;
public AmazonS3Service()
{
imageBucket = ConfigurationManager.AppSettings["S3.Buckets.Images"];
s3BaseUrl = new Uri(ConfigurationManager.AppSettings["S3.BaseAddress"]);
baseImageUrl = new Uri(s3BaseUrl, imageBucket);
}
public Image UploadImage(Stream stream)
{
if (stream == null) throw new ArgumentNullException("stream");
var key = string.Format("{0}.jpg", Guid.NewGuid());
var request = new PutObjectRequest
{
CannedACL = S3CannedACL.PublicRead,
Timeout = -1,
ReadWriteTimeout = 600000, // 10 minutes * 60 seconds * 1000 milliseconds
InputStream = stream,
BucketName = imageBucket,
Key = key
};
using (var client = new AmazonS3Client())
{
using (client.PutObject(request))
{
}
}
return new Image
{
UriString = Path.Combine(baseImageUrl.AbsoluteUri, key)
};
}
}

You are having trouble unit testing UploadImage because it is coupled to many other external services and state. Static calls including (new) tightly couple the code to specific implementations. Your goal should be to refactor those so that you can more easily unit test. Also, keep in mind that after unit testing this class, you will still need to do the big tests involving actually using the Amazon S3 service and making sure the upload happened correctly without error or fails as expected. By unit testing thoroughly, hopefully you reduce the number of these big and possibly expensive tests.
Removing the coupling to the AmazonS3Client implementation is probably going to give you the biggest bang for your testing buck. We need to refactor by pulling out the new AmazonS3Client call. If there is not already an interface for this class, then I would create one to wrap it. Then you need to decide how to inject the implementation. There are a number of options, including as a method parameter, constructor parameter, property, or a factory.
Let's use the factory approach because it is more interesting than the others, which are straight-forward. I've left out some of the details for clarity and read-ability.
interface IClientFactory
{
IAmazonS3Client CreateAmazonClient();
}
interface IAmazonS3Client
{
PutObjectResponse PutObject(PutObjectRequest request); // I'm guessing here for the signature.
}
public class AmazonS3Service : IAmazonS3Service
{
// snip
private IClientFactory factory;
public AmazonS3Service(IClientFactory factory)
{
// snip
this.factory = factory;
}
public Image UploadImage(Stream stream)
{
if (stream == null) throw new ArgumentNullException("stream");
var key = string.Format("{0}.jpg", Guid.NewGuid());
var request = new PutObjectRequest
{
CannedACL = S3CannedACL.PublicRead,
Timeout = -1,
ReadWriteTimeout = 600000, // 10 minutes * 60 seconds * 1000 milliseconds
InputStream = stream,
BucketName = imageBucket,
Key = key
};
// call the factory to provide us with a client.
using (var client = factory.CreateAmazonClient())
{
using (client.PutObject(request))
{
}
}
return new Image
{
UriString = Path.Combine(baseImageUrl.AbsoluteUri, key)
};
}
}
A unit test might look like this in MSTest:
[TestMethod]
public void InputStreamSetOnPutObjectRequest()
{
var factory = new TestFactory();
var service = new AmazonS3Service(factory);
using (var stream = new MemoryStream())
{
service.UploadImage(stream);
Assert.AreEqual(stream, factory.TestClient.Request.InputStream);
}
}
class TestFactory : IClientFactory
{
public TestClient TestClient = new TestClient();
public IAmazonS3Client CreateClient()
{
return TestClient;
}
}
class TestClient : IAmazonS3Client
{
public PutObjectRequest Request;
public PutObjectResponse Response;
public PutObjectResponse PutObject(PutObjectRequest request)
{
Request = request;
return Response;
}
}
Now, we have one test verifying that the correct input stream is sent over in the request object. Obviously, a mocking framework would help cut down on a lot of boilerplate code for testing this behavior. You could expand this by starting to write tests for the other properties on the request object. Error cases are where unit testing can really shine because often they can be difficult or impossible to induce in production implementation classes.
To fully unit test other scenarios of this method/class, there are other external dependencies here that would need to be passed in or mocked. The ConfigurationManager directly accesses the config file. Those settings should be passed in. Guid.NewGuid is basically a source of uncontrolled randomness which is also bad for unit testing. You could define an IKeySource to be a provider of key values to various services and mock it or just have the key passed from the outside.
Finally, you should be weighing all the time taken for testing/refactoring against how much value it is giving you. More layers can always be added to isolate more and more components, but there are diminishing returns for each added layer.

Things I would look at:
Mock your configuration manager to return invalid data for the bucket and the URL. (null, invalid urls, invalid buckets)
Does S3 support https ? If so mock it, if not, mock it and verify you get a valid error.
Pass different kinds of streams in (Memory, File, other types).
Pass in streams in different states (Empty streams, streams that have been read to the
end, ...)
I would allow the timeouts to be set as parameters, so you can test with really low
timeouts and see what errors you get back.
I would also test with duplicate keys, just to verify the error message. Even though you are using guids, you are storing to an amazon server where someone else could use the S3 API to store documents and could theoretically create a file that appears to be a guid, but could create a conflict down the road (unlikely, but possible)

Related

.NET Core how to unit test service?

I have build a WebAPI and want to create a unit test project to have my services tested automatically.
The flow of my WebAPI is simple:
Controller (DI Service) -> Service (DI Repository) -> _repo CRUD
Suppose I have a service like:
public int Cancel(string id) //change status filed to 'n'
{
var item = _repo.Find(id);
item.status = "n";
_repo.Update(item);
return _repo.SaveChanges();
}
And I want to build a unit test, which just use InMemoryDatabase.
public void Cancel_StatusShouldBeN() //Testing Cancel() method of a service
{
_service.Insert(item);
int rs = _service.Cancel(item.Id);
Assert.Equal(1, rs);
item = _service.GetByid(item.Id);
Assert.Equal("n", item.status);
}
I've searched other related question, found that
You can't use dependency injections on test classes.
I just want to know if there is any other solution to achive my unit test idea?
When unit testing, you should just supply all the dependencies of the class you are testing explicitly. That is dependency injection; not having the service construct its dependencies on its own but making it rely on the outer component to provide them. When you are outside of a dependency injection container and inside a unit test where you are manually creating the class you are testing, it’s your responsibility to provide the dependencies.
In practice, this means that you either provide mocks or actual objects to the constructor. For example, you might want to provide a real logger but without a target, a real database context with a connected in-memory database, or some mocked service.
Let’s assume for this example, that the service you are testing looks like this:
public class ExampleService
{
public ExampleService(ILogger<ExampleService> logger,
MyDbContext databaseContext,
UtilityService utilityService)
{
// …
}
// …
}
So in order to test ExampleService, we need to provide those three objects. In this case, we will do the following for each:
ILogger<ExampleService> – we will use a real logger, without any attached target. So any call on the logger will work properly without us having to provide some mock, but we do not need to test the log output, so we do not need a real target
MyDbContext – Here, we’ll use the real database context with an attached in-memory database
UtilityService – For this, we will create a mock which just setups the utility method we need inside the methods we want to test.
So a unit test could look like this:
[Fact]
public async Task TestExampleMethod()
{
var logger = new LoggerFactory().CreateLogger<ExampleService>();
var dbOptionsBuilder = new DbContextOptionsBuilder().UseInMemoryDatabase();
// using Moq as the mocking library
var utilityServiceMock = new Mock<UtilityService>();
utilityServiceMock.Setup(u => u.GetRandomNumber()).Returns(4);
// arrange
using (var db = new MyDbContext(dbOptionsBuilder.Options))
{
// fix up some data
db.Set<Customer>().Add(new Customer()
{
Id = 2,
Name = "Foo bar"
});
await db.SaveChangesAsync();
}
using (var db = new MyDbContext(dbOptionsBuilder.Options))
{
// create the service
var service = new ExampleService(logger, db, utilityServiceMock.Object);
// act
var result = service.DoSomethingWithCustomer(2);
// assert
Assert.NotNull(result);
Assert.Equal(2, result.CustomerId);
Assert.Equal("Foo bar", result.CustomerName);
Assert.Equal(4, result.SomeRandomNumber);
}
}
In your specific Cancel case, you want to avoid using any methods of the service you are not currently testing. So if you want to test Cancel, the only method you should call from your service is Cancel. A test could look like this (just guessing the dependencies here):
[Fact]
public async Task Cancel_StatusShouldBeN()
{
var logger = new LoggerFactory().CreateLogger<ExampleService>();
var dbOptionsBuilder = new DbContextOptionsBuilder().UseInMemoryDatabase();
// arrange
using (var db = new MyDbContext(dbOptionsBuilder.Options))
{
// fix up some data
db.Set<SomeItem>().Add(new SomeItem()
{
Id = 5,
Status = "Not N"
});
await db.SaveChangesAsync();
}
using (var db = new MyDbContext(dbOptionsBuilder.Options))
{
// create the service
var service = new YourService(logger, db);
// act
var result = service.Cancel(5);
// assert
Assert.Equal(1, result);
}
using (var db = new MyDbContext(dbOptionsBuilder.Options))
{
var item = db.Set<SomeItem>().Find(5);
Assert.Equal(5, item.Id);
Assert.Equal("n", item.Status);
}
}
Btw. note that I’m opening up a new database context all the time in order to avoid getting results from the cached entities. By opening a new context, I can verify that the changes actually made it into the database completely.

Unit testing Void method which calls another void, starts Task()

I'm looking for some advice on writing some unit tests for the code below. Implementation aside (it's not my code, but I've been tasked to retroactively write some tests for it) could someone suggest how I might test this? I'm not using nUnit or a similar framework; I am using the testing tools built into Visual Studio.
I'm fairly new to writing unit tests, but I imagine I should at least test the following:
Valid response passed into SaveFormBrokerResponse() method
Test for valid exceptions thrown by the catch()
Testing the started Task, but not sure how to do this
I've stripped just a bit out of this function, mostly to do with instantiation and population of some objects:
public void SaveResponse(IForm form, bool isLive, HttpRequestBase request)
{
try
{
var response = new FormBrokerResponses();
// Initialize some vars on response
using (var memory = new MemoryStream())
{
var serializer = new DataContractSerializer(typeof(FormKeyValue[]));
serializer.WriteObject(memory, request.Form.AllKeys.Select(r => new FormKeyValue(r, request.Form[r])).ToArray());
memory.Flush();
memory.Seek(0, SeekOrigin.Begin);
response.Values = Encoding.UTF8.GetString(memory.ToArray());
}
_dataHandler.SaveFormBrokerResponses(response);
}
catch (Exception ex)
{
throw new Exception("boom explosions");
}
Task.Factory.StartNew(() => DispatchFormResponseViaEmail(form, isLive, request.Form.AllKeys.ToDictionary(r => r, r => (object)request.Form[r])));
}
I realize that testing void implementations is tricky and questionable and that there are some integration test concerns here, but that said I can't (currently) change the implementation and need to write tests for what I have.
You can't. You've created a method that fires off an asynchronous operation and then doesn't expose any means of observing the completion/results of that operation to the caller. There are lots of ways of doing this (returning a task, accepting a callback, an event, etc.) but you need to do something for the caller to be able to observe the results of the asynchronous operation. If the method doesn't expose anything, then there is nothing that the caller can reliably do.
If you are allowed to make slight modifications to the code I would do the following which is just a small change anyway :
public void SaveResponse(IForm form, bool isLive, HttpRequestBase request)
{
try
{
var response = new FormBrokerResponses();
// Initialize some vars on response
using (var memory = new MemoryStream())
{
var serializer = new DataContractSerializer(typeof(FormKeyValue[]));
serializer.WriteObject(memory, request.Form.AllKeys.Select(r => new FormKeyValue(r, request.Form[r])).ToArray());
memory.Flush();
memory.Seek(0, SeekOrigin.Begin);
response.Values = Encoding.UTF8.GetString(memory.ToArray());
}
_dataHandler.SaveFormBrokerResponses(response);
}
catch (Exception ex)
{
throw new Exception("boom explosions");
}
Dispatch(form,isLive,request);
}
virtual void Dispatch(IForm form, bool isLive, HttpRequestBase request){
Task.Factory.StartNew(() => DispatchFormResponseViaEmail(form, isLive, request.Form.AllKeys.ToDictionary(r => r, r => (object)request.Form[r])));
}
I don't know what this class is named so suppose the class is named DutClass, you can now derive a different implementation of that class as following:
public class UnitTestClass : DutClass{
override Dispatch(){
//don't do anything or set a state variable that this method was called
}
}
Then instead of testing the DutClass you test the UnitTextClass which has a different implementation of the Dispatch method and does not start a Task at all. You can then test that in fact this method was called, test for the exceptions and so on.

How to set HttpRequest.InputStream value using reflection

I am writing a unit test for a WebApi controller that reads a POST body from Request.InputStream. I need to set the inputstream property of HttpContext.Current.Request.InputStream, or set the inputstream's contents. Here is my unit test code so far, but it keeps throwing exceptions:
var originalStream = HttpContext.Current.Request.InputStream;
Stream newStream = new MemoryStream(ASCIIEncoding.Default.GetBytes("Test String"));
var propInfo = originalStream.GetType().GetProperty("CanWrite");
propInfo.SetValue(originalStream, true);
newStream.CopyTo(originalStream);
propInfo.SetValue(originalStream, false);
I get the following exception on the SetValue line:
ArgumentException: Property set method not found
Am I going about this all wrong? My controller reads the input stream and deserializes it into JSON, so I need to be able to insert data into that stream. I just don't know how to do it. Many thanks.
public abstract bool CanWrite { get; }. There is not setter on that property.. hence your error.
In your example, originalStream will be of type Stream. Wrap it in another stream for your test. You aren't testing the HttpRequest.InputStream, you're testing the deserialization ...
var originalStream = new StreamReader(HttpContext.Current.Request.InputStream);
var content = originalStream.ReadToEnd();
.. etc.
You may even consider skipping using the request input altogether since you aren't really testing that.
EDIT:
To expand a bit more. You should move this out of the controller action.. do something like the below:
public class YourController : Controller {
private readonly IStreamWrapper _streamWrapper;
public YourController(IStreamWrapper wrapper) {
_streamWrapper = wrapper;
}
public ActionResult MethodYouAreTesting() {
var result = _streamWrapper.Process(HttpRequest.InputStream);
}
}
public class Tests {
public void YourTestMethod() {
var controller = new YourController(new FakeStreamWrapper()); // mock perhaps?
// Asserts here for the controller action
}
public void YourWrapperTester() {
var wrapper = new RealStreamWrapper();
// test Process method here..
}
}
Then you can test the stream reading and the deserialization in isolation.
Does that make sense or have I made it more complex? :/
The InputStream is a read-only stream.
There are several ways. To start with the simplest:
Create an instance of a StreamReader and call ReadToEnd() on InputStream to get the data being sent.
Once you have the contents, you can modify as needed before doing your JSON conversion.
Again, this is the simplest, least elegant way to do what you want, but it should work.

Mocking a StreamWriter/determining when to mock

I have a class which uses a StreamWriter to write to a file.
public void CreateLog(string errorLogFilePath, StringBuilder errorLogBuilder, string errorMessage)
{
using (StreamWriter sw = new StreamWriter(errorLogFilePath, true)
{
errorLogBuilder.Apend("An error was discovered.");
//....
sw.Write(errorLogBuilder.ToString());
}
}
[Questions]
1: Is it possible to check that the .Write() method is called?
2: Do i need to wrap a MemoryStream inside the StreamWriter in order to test it, without actually accessing the hard drive. One of StreamWriters constructors accepts a stream but it states the following + will the UTF-8 encoding affect this?
Initializes a new instance of the StreamWriter class for the specified stream by using UTF-8 encoding and the default buffer size.
3: How do you determine if a class is actually accessing the hd and thus needs to be mocked? (sorry if this last question sounds stupid, but im genuinely a little puzzled by this.)
Have the method write to a TextWriter rather than a StreamWriter. Then test the method by passing it a mock TextWriter. In the "real" code, of course, you'll pass in a StreamWriter that was created using new StreamWriter(errorLogFilePath, true).
This yields the following answers to your questions:
Yes
No
You can't generally determine that without decompiling its code.
A little more detail:
Refactor the method into two methods:
public void CreateLog(string errorLogFilePath, StringBuilder errorLogBuilder, string errorMessage)
{
using (StreamWriter sw = new StreamWriter(errorLogFilePath, true)
{
CreateLog(sw, errorLogBuilder, errorMessage);
}
}
public void CreateLog(TextWriter writer, StringBuilder errorLogBuilder, string errorMessage)
{
errorLogBuilder.Apend("An error was discovered.");
//....
writer.Write(errorLogBuilder.ToString());
}
Test the first method to ensure that it calls the second method with an appropriately-constructed StreamWriter. Test the second method to ensure that it calls Write on the passed TextWriter, with appropriate arguments. Now you've abstracted away the dependency on the hard drive. Your tests don't use the hard drive, but you're testing everything.
Generally speaking, you could :
Use a well tested logging library (like NLog, MS Logging Application Block), and spare you developping and maintaining your own.
Refactor your logging logic (or code calling messageboxes, open file dialogs, and so on) into a service, with its interface. This way you can split your testing strategy :
when testing consumers of the loggin service : mock the logging interface to make sure the log method is called. This will ensure that the logging is correctly called by consumers of your logging service
when testing the logging service implementation, just make sure that expected output matches given input : if you want to write "FOO" to bar.log, effectively call
IE :
// arrrange
File.Delete("bar.log")
// act
CreateLog("bar.log", errorLogBuilder, "FOO")
// assert
Assert.IsTrue( File.Exists("bar.log") )
Assert.IsTrue( File.ReadAllLines("bar.log").First() == "FOO")
The point is making sure that the component is called, done by mocking.
Then you can check that the component works as expected.
I know this is a very old question, but I came across this while trying to solve a similar problem.. namely, how to fake the StreamWriter.
The way I went about this was by not having the StreamWriter created inside the method as part of the using statement, but created up front, within the ctor (make your class extend from IDisposable and then destroy the StreamWriter in the Dispose method instead). Then inject a fake over the top of it while under test:
internal class FakeStreamWriter : StreamWriter
{
public List<string> Writes { get; set; } = new List<string>();
public FakeStreamWriter() : base(new MemoryStream()) { }
public override void Write(string value)
{
WriteLine(value);
}
public override void WriteLine(string value)
{
Writes.Add(value);
}
public override void Flush()
{
}
}
My unit test method then looks like this:
public void SmtpStream_Negotiate_EhloResultsCorrectly()
{
var ctx = new APIContext();
var logger = new FakeLogger();
var writer = new FakeStreamWriter();
var reader = new FakeStreamReader { Line = "EHLO test.com" };
var stream = new SmtpStream(logger, ctx, new MemoryStream())
{
_writer = writer,
_reader = reader
};
Exception ex = null;
try
{
stream.Negotiate(ctx);
}
catch (Exception x)
{
ex = x;
}
Assert.IsNull(ex);
Assert.IsTrue(writer.Writes.ElementAt(0) == "250 Hello test.com");
Assert.IsTrue(writer.Writes.ElementAt(1) == "250 STARTTLS");
}

What is a good approach to get rid of dependency on a StreamReader/FileStream for Unit Tests?

Here's the scenario:
I have a method that reads in a file via a FileStream and a StreamReader in .NET. I would like to unit test this method and somehow remove the dependency on the StreamReader object.
Ideally I would like to be able to supply my own string of test data instead of using a real file. Right now the method makes use of the StreamReader.ReadLine method throughout. What is an approach to modifying the design I have now in order to make this test possible?
Depend on Stream and TextReader instead. Then your unit tests can use MemoryStream and StringReader. (Or load resources from inside your test assembly if necessary.)
Note how ReadLine is originally declared by TextReader, not StreamReader.
The simplest solution would be to have the method accept a Stream as a parameter instead of opening its own FileStream. Your actual code could pass in the FileStream as usual, while your test method could either use a different FileStream for test data, or a MemoryStream filled up with what you wanted to test (that wouldn't require a file).
Off the top of my head, I'd say this is a great opportunity to investigate the merits of Dependency Injection.
You might want to consider redesigning your method so that it takes a delegate that returns the file's contents. One delegate (the production one) might use the classes in System.IO, while the second one (for unit testing), returns the contents directly as a string.
I think the idea is to dependency inject the TextReader and mock it for unit testing. I think you can only mock the TextReader because it is an abstract class.
public class FileParser
{
private readonly TextReader _textReader;
public FileParser(TextReader reader)
{
_textReader = reader;
}
public List<TradeInfo> ProcessFile()
{
var rows = _textReader.ReadLine().Split(new[] { ',' }).Take(4);
return FeedMapper(rows.ToList());
}
private List<TradeInfo> FeedMapper(List<String> rows)
{
var row = rows.Take(4).ToList();
var trades = new List<TradeInfo>();
trades.Add(new TradeInfo { TradeId = row[0], FutureValue = Convert.ToInt32(row[1]), NotionalValue = Convert.ToInt32(row[3]), PresentValue = Convert.ToInt32(row[2]) });
return trades;
}
}
and then Mock using Rhino Mock
public class UnitTest1
{
[Test]
public void Test_Extract_First_Row_Mocked()
{
//Arrange
List<TradeInfo> listExpected = new List<TradeInfo>();
var tradeInfo = new TradeInfo() { TradeId = "0453", FutureValue = 2000000, PresentValue = 3000000, NotionalValue = 400000 };
listExpected.Add(tradeInfo);
var textReader = MockRepository.GenerateMock<TextReader>();
textReader.Expect(tr => tr.ReadLine()).Return("0453, 2000000, 3000000, 400000");
var fileParser = new FileParser(textReader);
var list = fileParser.ProcessFile();
listExpected.ShouldAllBeEquivalentTo(list);
}
}
BUT the question lies in the fact whether it is a good practice to pass such an object from the client code rather I feel it should be managed with using within the class responsible for processing. I agree with the idea of using a sep delegate for the actual code and one for the unit testing but again that is a bit of extra code in production. I may be a bit lost with the idea of dependency injection and mocking to even file IO open/read which actually is not a candidate for unit testing but the file processing logic is which can be tested by just passing the string content of the file (AAA23^YKL890^300000^TTRFGYUBARC).
Any ideas please! Thanks

Categories