This post has been sitting in my draft folder for a while. At my client we use the Visual Studio Team Suite set of products, which means we also use MSTest for unit testing. I'm not going to get into MSTest vs. XUnit vs. NUnit vs. MbUnit, but I do have a comment about the implementation of the ExpectedException attribute from MSTest.
Let's say that I have some code that can throw a series of the same exception. A good example of this would guard clauses on a method. For example:
public void SomeMethod(FileStream inputFile, FileStream outputFile)
{
if (inputFile == null)
throw new ArgumentNullException("inputFile", "inputFile is null.");
if (outputFile == null)
throw new ArgumentNullException("outputFile", "outputFile is null.");
}
I'd like to test this method by passing a Null value for inputFile, then pass a Null value for outputFile. This tests both guard clauses. Since the guard clauses throw the SAME type of exception, the only thing that is going to be different is the message of the exception.
Taking a look at one of the constructors of the ExpectedException attribute we see the following:
public ExpectedExceptionAttribute(Type exceptionType, string message)
In the documentation the parameter "message" is defined as: "A message to be attached to the exception". The description of this constructor states: "Initializes a new instance of the ExpectedExceptionAttribute class with and expected exception type and a message that describes the exception." Originally, I read this to mean that what I put in the message parameter for this attribute would be compared to the message being thrown in the exception; however, that is not the case. This parameter just "names" the exception for some unknown purpose (if anyone can tell me why I'd appreciate it).
So doing the following doesn't work the way you think it should:
[TestMethod()]
[ExpectedException(typeof(ArgumentNullException), "inputFile is null.\r\nParameter name: inputFile")]
public void SomeMethod_InputFileIsNullThrowsException()
{
Bar target = new Bar();
FileStream inputFile = null;
FileStream outputFile = null;
target.SomeMethod(inputFile, outputFile);
}
[TestMethod]
[ExpectedException(typeof(ArgumentNullException), "outputFile is null.\r\nParameter name: outputFile")]
public void SomeMethod_OutputFileIsNullThrowsException()
{
Bar target = new Bar();
FileStream inputFile = new FileStream("C:\\temp\\foofile.txt", FileMode.Open);
FileStream outputFile = null;
target.SomeMethod(inputFile, outputFile);
}
If you run these both will pass, but that's actually a false sense of security in this case. Because if I changed the ExpectedException attribute message parameter to "a;sjfa;kjfa;fj" it will also pass.
This means to accurately test this I have to do something like this:
[TestMethod()]
public void SomeMethod_InputFileIsNullThrowsException()
{
Bar target = new Bar();
FileStream inputFile = null;
FileStream outputFile = null;
try
{
target.SomeMethod(inputFile, outputFile);
}
catch (Exception ex)
{
Assert.IsInstanceOfType(ex, typeof(ArgumentNullException));
Assert.IsTrue(ex.Message.Contains("inputFile"));
}
}
[TestMethod]
public void SomeMethod_OutputFileIsNullThrowsException()
{
Bar target = new Bar();
FileStream inputFile = new FileStream("C:\\temp\\foofile.txt", FileMode.Open);
FileStream outputFile = null;
try
{
target.SomeMethod(inputFile, outputFile);
}
catch (Exception ex)
{
Assert.IsInstanceOfType(ex, typeof(ArgumentNullException));
Assert.IsTrue(ex.Message.Contains("outputFile"));
}
}
Note, I've removed the ExpectedException attribute, tested for the exception type on my own and then added an Assert that let's me determine which of the argumentNullExceptions was thrown. I could have left the ExpectedException attribute and added the IsTrue assert adding a "throw" to let the ExpectedException attribute do it's work as well.
Let's compare this to NUnit:
[Test]
[ExpectedException(typeof(ArgumentNullException), ExpectedMessage="inputFile is null.\r\nParameter name: inputFile")]
public void SomeMethod_InputFileIsNullThrowsException()
{
Bar target = new Bar();
FileStream inputFile = null;
FileStream outputFile = null;
target.SomeMethod(inputFile, outputFile);
}
This works exactly as I expected it to. It checked the type and the message. Note that NUnit is more explicit regarding the name of the parameter in the attribute "ExpectedMessage". Perhaps coming from NUnit I was just expecting the message parameter on the MSTest version of ExpectedException to work the same way.
Note that there was feedback left on the Connect site regarding this issue by none other than Roy Osherove. I guess I'm not only one that thinks this is unexpected (no pun intended) behavior. This feedback was marked "Closed (Fixed)". It clearly wasn't. As you can see Roy entered this against VS 2005. The code above was executed in VS 2008.
I posted this as a .NET Nugget since it caught me off guard and I actually had several unit tests coded to work the way I thought they should and only discovered this because one of them should have been failing and wasn't.