Dennis Burton's Develop Using .NET

Change is optional. Survival is not required.
Tags: tdd

In my last post, I mentioned the new ValuesAttribute that can be used as a test factory to generate a series of tests with many permutations of parameters. Looking into that feature led me to look at the feature set in NUnit 2.5 which is currently in Alpha 4. Some of these features address scenarios that I have run into in my test code. I wanted to mention them here so others could start benefiting as well.

The source of the new feature set that I am pulling from is the current release notes.

Movin' on up

One of the items that I think is most important is not a new feature, just a change in location. The change in mind set is what stands out as important to me. The Is, Has, Text, and List constraints have been moved into the NUnit.Framework namespace. They were formerly off in the more obscure NUnit.Framework.SyntaxHelpers namespace. The assertions that are created using these constraints line up much better with BDD and have a more readable feel to them. I am happy to see them becoming part of the mainstream namespace. If you are not familiar with the constraint model, check out code below.

// Classic model
Assert.AreEqual(expected, actual);
// Constraint model
Assert.That(actual, Is.EqualTo(expected));

I think the second reads much more like the English phrase that the constraint would represent. Check out the docs; you will find that almost everything you can do with the "classic" model can also be done in the constraint model.

Chained Setup and TearDown

Test code should be crafted with the same level of care as production intent code. This means that it should be DRY and carefully designed. Many times you will be testing a related set of classes, leading to base classes in your test code. Don't be afraid to capture common test code functionality in base classes. Prior to this release, the Setup and TearDown attributes could be applied in your base class, but if you applied the Setup or Teardown attributes in your derived class, the calls were not chained. You had to remember to call the base class version from the derived class. In 2.5, the base Setup method will be call prior to the derived Setup method by the framework. One more win for automation versus developer discipline.

New constraints

Chris Marinos recently blogged about disliking the [ExpectedException] style of tests and wrote a helper class to support the syntax of Throws.Exception. As of the 2.5 release Throws.Exception is included. This allows for much more focused testing. Putting the ExpectedExceptionAttribute on a method simply says that somewhere in the method the exception will get thrown. The exception may well be thrown 2 lines prior to the call you were intending to test. This yields a false positive test result. Throws.Exception (and its set of related Throws) takes a delegate so that you can check a specific call for the presence of an exception.

[Test]
[ExpectedException]
public void PassesButShouldnt()
{
    int importantPreWork = int.Parse("abcd");
    DoStuff();
}

[Test]
public void FailsProperly()
{
    int importantPreWork = int.Parse("abcd"); 
    Assert.Throws<Exception>( DoStuff );
}
public void DoStuff()
{ throw new Exception("Something is terribly wrong"); }

In the example, the PassesButShouldnt test passes due to the expected exception being thrown on the Parse call. Unfortunately, the test was intended to ensure that DoBadStuff threw an exception. The FailsProperly test more accurately checks that only the DoBadStuff call throws an exception. In addition, it also reads better as it places the constraint on the same line as the call just like any other Assert method.

Attribute testing

With the Has.Attribute constraint, you can verify an object (yes, object not class) is decorated with an attribute. A common example of where this could be used is validating that a class has been marked with the SerializableAttribute.

[Serializable]
public class testclass {}

[Test]
public void Test()
{ Assert.That(new testclass(), Has.Attribute<SerializableAttribute>()); }

I have to admit, when I first saw the Has.Attribute constraint, I thought for sure one of my test base classes was going to get a bit lighter. One of the applications that I work on has a pluggable architecture and uses attributes to determine what should be exposed to the end user. I had to code up a method that used reflection to determine if the items that were supposed to be exposed did in fact contain the attribute that exposed them. I was somewhat disappointed to see that it only seemed to work with instances of objects and not Types or MemberInfos. This did get me thinking, however, that my approach of putting this in a base class may not have been as elegant as learning about extending NUnit. Perhaps that will be a post in the future. I should call out that this was in the NUnit blog as a feature added in Alpha 1, however, it is the only feature listed here that has not made it to the documentation yet. That could mean that it just has not been documented yet, or it could indicate that it is not yet ready for prime time.

Range Testing

Is.InRange allows you to use a more concise syntax for expressing assertions that required both a GreaterThan and LessThan for validating an item is within bounds.

[Test]
public void RangeTesting()
{
    // old way
    Assert.That(3, Is.GreaterThanOrEqualTo(1) & Is.LessThanOrEqualTo(5));
    // new way
    Assert.That(3, Is.InRange(1, 5));

    // NOTE: InRange is inclusive
    Assert.That(5, Is.InRange(1, 5));
    Assert.That(1, Is.InRange(1, 5));
}

New Assertions

In some tests you are able to determine early in the test that the expected behavior has been met. Rather than throwing in a return statement or creating triangular code, the new Assert.Pass method is available for early termination with passing results. There is also a new inconclusive result state that can be set by calling Assert.Inconclusive. I have no idea how this can be used; I cannot think of any test scenarios that I have had to implement where I really was looking for "Assert...oh...I don't know." It seems there is a failure in the system at this point, even if that failure point is not understanding the requirements. I would take this as an indicator to go clarify requirements with the client and write a better test.

Specialized Assertions

CollectionAssert has picked up a set of IsOrdered constraints. This constraint uses the IComparable interface to verify increasing or decreasing order of all of the items in a collection. This is another one of those features that I have implemented in a test utilities class. I am happy to remove those lines of code.

Previous versions of NUnit had a FileAssert class to verify that your application was generating an output file. DirectoryAssert has been added as a compliment to FileAssert. The DirectoryAssert.IsWithin method (or IsNotWithin) will crawl from the directory specified, including all of its subdirectories, to verify that the expected directory is present. You can also use IsEmpty to verify if a file was output. You are, of course, still responsible for determining that the correct output was written.

Generic Support

There are several type constraints in NUnit that take typeof(class) as a parameter. The following constraints have been updated to include a version that takes a generic parameter specifying the type.

Assert.IsInstanceOf<T>(object actual);
Assert.IsNotInstanceOf<T>(object actual);
Assert.IsAssignableFrom<T>(object actual);
Assert.IsNotAssignableFrom<T>(object actual);
Assert.Throws<T>(TypeSnippet code);

The TestFixtureAttribute also picked up a generic version. Use this to test a class that takes a generic type parameter. Specify this attribute multiple times on a class, each time specifying a different type for the generic parameter. For each type specification, a new test instance is created and executed. This is useful when you were trying to exercise a class under test with a couple of different types. It does seem like this is the wrong level of abstraction for a test, but I am sure it is a reaction to some level of demand.

Try it out

Every one has a different tolerance for using things that have not been marked as released, yet. Early releases of NUnit traditionally have been very solid. Much like GMail still being in beta, it seems the open source community drop releases much more often, but keeps them flagged with alpha and beta. I have already started using these features to produce cleaner, more readable test code.

Tags: programming | tdd

RowTests are a great tool for consolidating tests that have the same logic, but differ by the parameters and the expected values. I did a post a while back on using RowTest in MbUnit, and Michael Eaton recently posted on using RowTest with NUnit. Reading Eaton's post reminded me that I had just run into a new feature in the NUnit 2.5 drop. One of my favorite features of competitive open source tools is that they usually add at least one feature more than the other guy with each release. For the 2.5 release of NUnit, one of those features is the ValuesAttribute. Granted, MbUnit had this feature in concept with CombinationalTestAttribute, but the usage of the ValuesAttribute in NUnit seems much cleaner to me.

Tests with no [Values]

Looking at the previous posts on RowTests, you can see that passing in many values for a specific test is not that difficult. So why would we need anything else? In certain scenarios, the number of permutations that you would have to write Row attributes for could be pretty cumbersome, and if it is truly a permutation scenario, difficult to maintain. The test case for this post will be that the test should be called with all possible combinations of 1,2,3 for one parameter and 10,20,30 for another parameter. With this simple example you can see that we have already blown up to 9 rows, adding on more variant onto either of the parameters and we would have 12 rows to maintain.

Giving your tests [Values]

What the values attribute does is allow you to express all 9 of these combinations in a very concise syntax. This lines up better with the expression of your requirements; instead of 9 Row attributes, you would now have the following:

[Test]
public void Use_Value_Attributes([Values(1,2,3)] int param1,[Values(10,20,30)] int param2)
{
    r.DoStuffWithParms(param1, param2);
}

This nicely wraps up in one line what would have been expressed with 9 [Row] attributes in the past. Bring this up in the NUnit gui to verify that the tests you were looking for were in fact generated.

ValueAttributes

What do you use it for

Now that you have such a powerful tool in you toolset, start to think about how this would be useful. The real world case where I use this feature ensures that interaction with an external library is correct both in number and order of calls when the actual parameters from the system are applied. This test is purely focused on the behavior of the system not the expectation of a mathematical result.

What wouldn't you use it for

It would be very difficult to tie an expected value to the generated permutations. So if you are going after a specific output from an algorithm, a RowTest will work much better for you. Keep in mind that the RowTest feature has been around a while, so it is likely that you will use it more often.

Dennis Burton

View Dennis Burton's profile on LinkedIn
Follow me on twitter
Rate my presentations
Google Code repository

Community Events

Windows Azure Boot Camp Lansing GiveCamp