I'm writing a Tiger compiler in C# and I'm going to translate the Tiger code into IL.
While implementing the semantic check of every node in my AST, I created lots of unit tests for this. That is pretty simple, because my CheckSemantic method looks like this:
public override void CheckSemantics(Scope scope, IList<Error> errors) {
...
}
so, if I want to write some unit test for the semantic check of some node, all I have to do is build an AST, and call that method. Then I can do something like:
Assert.That(errors.Count == 0);
or
Assert.That(errors.Count == 1);
Assert.That(errors[0] is UnexpectedTypeError);
Assert.That(scope.ExistsType("some_declared_type"));
but I'm starting the code generation in this moment, and I don't know what could be a good practice when writing unit tests for that phase.
I'm using the ILGenerator class. I've thought about the following:
Generate the code of the sample program I want to test
Save generated code as test.exe
Execute text.exe and store the output in results
Assert against results
but I'm wondering if there is a better way of doing it?
That's exactly what we do on the C# compiler team to test our IL generator.
We also run the generated executable through ILDASM and verify that the IL is produced as expected, and run it through PEVERIFY to ensure that we're generating verifiable code. (Except of course in those cases where we are deliberately generating unverifiable code.)
I've created a post-compiler in C# and I used this approach to test the mutated CIL:
Save the assembly in a temp file, that will be deleted after I'm done with it.
Use PEVerify to check the assembly; if there's a problem I copy it to a known place for further error analysis.
Test the assembly contents. In my case I'm mostly loading the assembly dynamically in a separate AppDomain (so I can tear it down later) and exercising a class in there (so it's like a self-checking assembly: here's a sample implementation).
I've also given some ideas on how to scale integration tests in this answer.
You can think of testing as doing two things:
letting you know if the output has changed
letting you know if the output is incorrect
Determining if something has changed is often considerably faster than determining if something is incorrect, so it can be a good strategy to run change-detecting tests more frequently than incorrectness-detecting tests.
In your case you don't need to run the executables produced by your compiler every time if you can quickly determine that the executable has not changed since a known good (or assumed good) copy of the same executable was produced.
You typically need to do a small amount of manipulation on the output that you're testing to eliminate differences that are expected (for example setting embedded dates to a fixed value), but once you have that done, change-detecting tests are easy to write because the validation is basically a file comparison: Is the output the same as the last known good output? Yes: Pass, No: Fail.
So the point is that if you see performance problems with running the executables produced by your compiler and detecting changes in the output of those programs, you can choose to run tests that detect changes a stage earlier by comparing the executables themselves.
Related
I started playing with Roslyn. It’s relatively easy to parse code and do static analysis.
I wonder if it’s possible to use it for runtime analysis? I want to call a method with parameters and check which branches were executed. In other words, I need a runtime execution plan.
Is it something which could be done with Roslyn?
I don't know what the best solution is and I would defer to anything SLaks recommends in most cases.
However...
If you want to do this with Roslyn you certainly can. In fact at my company does something similar (we map unit tests to the methods they invoke).
Here's a high level overview of our approach.
Rewrite every single function in the solution to log when it is hit in some global static lookup/data-structure. You can iterate over every file one at a time and use the CSharpSyntaxRewriter on each one. (In your case you'll be rewriting on a branch or line-by-line basis)
Run each unit test one at a time and see what gets run by analzying your global lookup.
Aggregate the results across all your unit tests to understand your complete code coverage.
I suspect the answer is no, but I'll ask anyway...
TL;DR
I know I can exclude a class or method from coverage analysis with the [ExcludeFromCodeCoverage] attribute, but is there a way to exclude only part of a method?
Concrete example
I have a method that lazily generates a sequence of int.MaxValue elements:
private static IEnumerable<TElement> GenerateIterator<TElement>(Func<int, TElement> generator)
{
for (int i = 0; i < int.MaxValue; i++)
{
yield return generator(i);
}
}
In practice, it's never fully enumerated, so the end of the method is never reached. Because of that, DotCover considers that 20% of the method is not covered, and it highlights the closing brace as uncovered (which corresponds to return false in the generated MoveNext method).
I could write a test that consumes the whole sequence, but it takes a very long time to run, especially with coverage enabled.
So I'd like to find a way to tell DotCover that the very last instruction doesn't need to be covered.
Note: I know I don't really need to have all the code covered by unit tests; some pieces of code can't or don't need to be tested, and I usually exclude those with the [ExcludeFromCodeCoverage] attribute. But I like to have 100% reported coverage for the code that I do test, because it makes it easier to spot untested parts of the code. Having a method with 80% coverage when you know there is nothing more to test in it is quite annoying...
No, there is no way to exclude "part of a method" from coverage analysis with dotCover.
In the general sense you got a couple of options:
Extract the uncovered part into its own method, so you can properly ignore that method from analsysis
Ignore the problem
In this case there may be a third options. Since your test code exercises the majority of your method, perhaps you should just write a test method that makes sure the code runs to completion?
First and foremost, while "code coverage" can be an important metric, one must realize that it just might not be possible to have 100% "code coverage". 100% Code coverage is one of those metrics that you should aspire to attain, but which you never will; i.e. get as close as you possibly can.
OTOH, don't go crazy trying to get 100% code coverage. More importantly, is your code readable? Is it testable (I presume so since you're looking at code coverage)? Is it maintainable? Is it SOLID? Do you have passing unit, integration, and end-to-end tests? These things are more important than achieving 100% code coverage. What code coverage will tell you is how extensive your testing is (I'm not sure if the built-in code coverage analysis engine includes only unit tests, or includes all types of tests when calculating its statistics), which gives you an indication of whether or not you have enough tests. Also, while it will tell you how extensive your tests are (i.e. how many lines of code are executed by your tests), it won't tell you if your tests are any good (i.e. are your tests really testing what needs to be tested to ensure your application is working correctly).
Anyway, this may be not an answer, but food for thought.
Imagine there's a mission-critical process that'll be used in a business which handles sensitive information (think of Credit Card, social security, patient records...etc). I would think this unit ideally should do whatever it has to do on-the-fly, meaning it won't intentionally write files to disk containing sensitive information. The idea here is that if the computer that runs this process is compromised, no sensitive information can be leaked, at least not by means of files.
What approaches could be taken to, say, come up with a unit test that will fail if the unit under test tries to write any file to disk?
There is the FileSystemWatcher (http://www.c-sharpcorner.com/uploadfile/puranindia/filesystemwatcher-in-C-Sharp/) however this requires you to know a specific directory. In your case this probably isn't very helpful since the program could write anything to disk any where. This introduces a unique problem. However, I have also found something called Detours from Microsoft. This appears to intercept all native win32 api calls. http://research.microsoft.com/en-us/projects/detours/ The issue with this is that its kind of hard to test, and integrating it into unit testing will be a challenge.
When you have to treat your software as "untrusted" in the sense that you need to prove it doesn't do something, testing becomes a complex task that requires you to run them on very controlled environments. When hooking in to the Win32 API, you will be deluged with API calls that need to be processed quickly. This can result in unintentional side effects because the application is not running in a truly native environment.
My suggestion to you (having worked several years doing software testing for Pharma automation to the exacting standards of the FDA) is to create a controlled environment, eg a virtual machine, that has a known starting state. This can be accomplished by never actually saving vmdk changes to disk. You have to take a snapshot of the file system. You can do this by writing a C# app to enumerate all files on the virtual drive, getting their size, some timestamps and maybe even a hash of the file. This can be time consuming so you may want (or be able) to skip the hashing. Create some sort of report, easiest would be by dropping them in a CSV or XML export. You then run your software under normal circumstances for a set period of time. Once this is complete, you run a file system analysis again and compare the results. There are some good apps out there for comparing file contents (like WinMerge). When taking these snap shots, the best way to do it would be to mount the vmdk as a drive in the host OS. This will bypass any file locks the guest OS might have.
This method is time intensive but quite thorough. If you don't need something of this depth, you can use something like Process Monitor and write the output to a file and run a report against that. However in my work I would have to prove that Process Monitor shows all IO before I could use it which can be just as hard as the method I spoke of above.
Just my 2 cents.
UPDATE:
I've been thinking about it, and you might be able to achieve fairly reliable results if you remove all references to System.IO from your code. Write a library to wrap around System.IO that either does not implement a write method, or only implements one that also writes to a log file. In this case, you simply have to validate that every time a write occurs using your library, it gets logged. Then validate using reflection that you don't reference System.IO outside of this new wrapper library. Your tests can then simply look at this log file to make sure only approved writes are occurring. You could make use of a SQL Database instead of a flat log file to help avoid cases of tampering or contaminated results. This should be much easier to validate than trying to script a virtual machine setup like I described above. This, of course, all requires you to access to the source code of the "untrusted" application, although since you are unit testing it, I assume you do.
1st option:
Maybe you could use Code Access Security, but the "Deny" is obsolete in .NET 4 (but should works in previous version):
[FileIOPermission(SecurityAction.Deny)]
public class MyClass
{
...
}
You may reactivate this behavior in .NET 4 using NetFx40_LegacySecurityPolicy
2nd option:
reducing the level of privilege may also works, as I know that downloaded app can't write on the disk and must use a special storage area.
3rd option:
Remove any reference to System.IO and replace by an interface that your code must use to write data to disk.
Then write an implementation that use System.IO (in a separate project)
In the nunit test, mock this interface and throw an exception when a method id called.
Problem is to ensure any developers will not call System.IO anymore. You can try to do this by enforcing coding rules using FxCop (or other similar tools)
I want to avoid my program being simple to have the license-verifier part removed from.
I don't want to use a commercial obfuscator because:
Of the cost. And though they can do a better job than I – they
too don't make it impossible to crack, just harder.
It seems that sometimes obfuscators cause bugs in the generated
code.
Obviously, I will be keeping an un-obfuscated copy for maintenance.
I once had to hide a license verifier in code that the customer could modify. Conceivably, they could have removed it if they knew where to look. Here are some tricks that I used at the time.
Give your verifier classes, assembly names, and variable names that look like they actually do something else.
Call the verifier from multiple parts of the code.
Add a randomizer to the call for verification so that sometimes it runs, and sometimes it doesn't. This will make it harder to know where the verification code is actually coming from.
I should add that all of this is defeatable and could cause serious maintenance headaches, but in my particular scenario it worked.
If your intent is to make it harder, but not impossible, one way is to have multiple code points that check your licence file is valid.
Lets say you have a licence file with some key like so
abc-def-fhi-asdf
So, four parts to the key. We would then create four different methods that check for the various parts of the key.
By doing this, and varying the methods used through the code (ideally, randomly choosing the verification method at runtime), you make it significantly more difficult to remove the validation.
on top of this, one method would be to have a publish process that inlined your verification method, subtly changing it each time it is called.
for example something like this:
*user clicks a common function
// [VALIDATION STUB]
*perform user action
The new publish process runs through the code, pulling out // [VALIDATION STUB] and replacing it with your validation code (before the code is compiled), which as I say should vary as much as possible each time.
The main thing to pull from my answer really is that obfuscation is hard, but not impossible. Especially if you resign yourself to the reality that the malevolent user will always break it eventually
I have some suggestions that you may find usefull.
First of course you can use free obfuscators like the one that comes with VisualStudio. It's better than nothing.
Second you can write your license verification code and once it's working fine, refactor it as much as you can, change class names, member variables, local variables and methods to something like c1, v1, l1, m1 and so on. That's basically what obfuscators do.
Third, do all of the above.
Fourth, write your licence verification in unmanaged code (C++, Delphi) and make it a DLL named something important like core.dll, net.dll etc. You can also put some decoy methods in there that would do nothing important. Make many calls to that DLL from multiple places of your code and pretend that you do something with the results of those calls.
I have one class which talks to DataBase.
I have my integration-tests which talks to Db and asserts relevant changes. But I want those tests to be ignored when I commit my code because I do not want them to be called automatically later on.
(Just for development time I use them for now)
When I put [Ignore] attribute they are not called but code-coverage reduces dramatically.
Is there a way to keep those tests but not have them run automatically
on the build machine in a way that the fact that they are ignored does
not influence code-coverage percentage?
Whatever code coverage tool you use most likely has some kind of CoverageIgnoreAttribute or something along those lines (at least the ones I've used do) so you just place that on the method block that gets called from those unit tests and you should be fine.
What you request seems not to make sense. Code-Coverage is measured by executing your tests and log which statements/conditions etc. are executed. If you disable your tests, nothing get executed and your code-coverage goes down.
TestNG has groups so you can specify to only run some groups, automatically and have the others for usage outside of that. You didn't specify your unit testing framework but it might have something similar.
I do not know if this is applicable to your situation. But spontaneously I am thinking of a setup where you have two solution files (.sln), one with unit/integration tests and one without. The two solutions share the same code and project files with the exception that your development/testing solution includes your unit tests (which are built and run at compile time), and the other solution doesn't. Both solutions should be under source control but only the one without unit tests are built by the build server.
This kind of setup should not need you to change existing code (too much). Which I would prefer over rewriting code to fit your test setup.