4-6-04 Embedding JUnit tests

For the 4th edition of Thinking in Java, I've been experimenting with ways to take better control of JUnit testing. My primary interest is in automatically generating the JUnit code itself to minimize the effort of the programmer.

One approach is to use JUnitDoclet, which automatically creates frameworks of JUnit code for various classes. This is certainly a step in the right direction by eliminating all the redundant code generation, but:

1. The programmer must still switch over from the class code to the JUnit code framework and write the test code.

2. You have to worry about regeneration tags so that the JUnitDoclet can regenerate framework without losing work.

3. The essential test code is still mixed in with all the JUnit code, and requires extra effort to track down and maintain.

It would be nice if the programmer only needed to write the essence of the test code rather than write or even navigate around the JUnit code. In addition, it would be nice to be able to put the test code right next to the code that was being tested, so you could more easily keep track of and change the code. Finally, it would be helpful if the programmer rarely, if ever, has to look at all the JUnit code, but rather is able to just focus on the essence and assume that everything else will be automated.

I have been peripherally involved in the design of Walter Bright's "D" programming language (free at DigitalMars since it's inception, and one of the features I suggested was compiler support for unit testing. The resulting syntax produced what seems like an obvious Java solution for the same problem: embed the essence of the unit test code within inline comments in the code for the class to be tested, and then automatically generate the JUnit code - which can then be used without any examination or intervention on the part of the programmer. To make changes, the programmer only needs to change the commented source code, and run the JUnit generator again.

As an example, consider JUnitDemo.java from Thinking in Java 3rd edition. We'll embed the unit test code as comments. Each fragment of test code will be surrounded by a multiline comment beginning with /*~, to uniquely identify it and make it easy for the code generator to extract. Each block will always begin with the word "test," but the very first one will start with an uppercase 'T' to indicate that this is the test class name. This first block will also begin with information such as the package statement and any necessary imports. The remaining information in the first block can include constructors, setUp() and tearDown() methods (although these are normally not necessary), helper methods, and fields, such as list in the example below.

Each of the test method blocks also begins with /*~, but the word "test" begins with a lowercase 't' to indicate this is a test method. The lines of code that follow are simply wrapped into the test method of the given name.

//: c15:CountedListEmbedded.java
// Embedded comments to generate JUnit tests
import java.util.*;
import junit.framework.*;

public class CountedListEmbedded extends ArrayList {
  private static int counter = 0;
  private int id = counter++;
  public CountedListEmbedded() {
    System.out.println("CountedListEmbedded #" + id);
  }
  public int getId() { return id; }
  /*~ TestCountedListEmbedded
  // package statement goes here if necessary
  import java.util.*;  // Any imports go here
  private CountedListEmbedded list =
    new CountedListEmbedded();
  public TestCountedListEmbedded() {
    for(int i = 0; i < 3; i++)
      list.add("" + i);
  }
  protected void setUp() {
    System.out.println("Set up for " + list.getId());
  }
  public void tearDown() {
    System.out.println("Tearing down " + list.getId());
  }
  private void
  compare(CountedListEmbedded lst, String[] strs) {
    Object[] array = lst.toArray();
    assertTrue("Arrays not the same length",
      array.length == strs.length);
    for(int i = 0; i < array.length; i++)
      assertEquals(strs[i], (String)array[i]);
  }
  */
  // The methods to be tested are shown
  // redundantly here, as a demonstration:
  public void add(int index, Object element) {
    super.add(index, element);
  }
  /*~ testInsert
    System.out.println("Running testInsert()");
    assertEquals(list.size(), 3);
    list.add(1, "Insert");
    assertEquals(list.size(), 4);
    assertEquals(list.get(1), "Insert");
  */
  // A method to be tested:
  public Object set(int index, Object element) {
    return super.set(index, element);
  }
  /*~ testReplace
    System.out.println("Running testReplace()");
    assertEquals(list.size(), 3);
    list.set(1, "Replace");
    assertEquals(list.size(), 3);
    assertEquals(list.get(1), "Replace");
  */
  /*~ testOrder
    System.out.println("Running testOrder()");
    compare(list, new String[] { "0", "1", "2" });
  */
  // A method to be tested:
  public Object remove(int index) {
    return super.remove(index);
  }
  /*~ testRemove
    System.out.println("Running testRemove()");
    assertEquals(list.size(), 3);
    list.remove(1);
    assertEquals(list.size(), 2);
    compare(list, new String[] { "0", "2" });
  */
  // A method to be tested:
  public boolean addAll(Collection c) {
    return super.addAll(c);
  }
  /*~ testAddAll
    System.out.println("Running testAddAll()");
    list.addAll(Arrays.asList(new Object[] {
      "An", "African", "Swallow"}));
    assertEquals(list.size(), 6);
    compare(list, new String[] { "0", "1", "2",
       "An", "African", "Swallow" });
  */
} ///:~

This is a very simple, straightforward approach. With more effort, you could deduce the name of the class under test and prepend the word "Test" to it in order to generate the class name, and you could also automatically generate the test method names. For this first implementation of the idea, however, the approach above seems to achieve the desirable leverage.

The easiest and most appropriate approach to generating the JUnit test from the extracted code is a Python program, although it could be done using Java (with a great deal more effort - this exercise is left to the reader). This is JUnitCreate.py:

import os, re

pattern = re.compile("/\*~(.*?)\*/", re.DOTALL)

BeginFile = """\
import junit.framework.*;

public class %s extends TestCase {
"""

TestMethod = """\
  public void %s() {
%s
  }"""

EndFile = """\
  public static void main(String[] args) {
    // Invoke JUnit on the class:
    junit.textui.TestRunner.run(%s.class);
  }
}
"""

for path, dirs, files in os.walk('.'):
    for filepath in [ path + os.sep + f
            for f in files if f.endswith(".java")]:
        package = None
        imports = ''
        header = ''
        matches = pattern.findall(file(filepath).read())
        if matches:
            startBody = matches[0].split("\n")
            className = startBody[0].strip()
            assert className.startswith("Test"), \
                "className must start with 'Test'"
            junitName = path + os.sep + className + ".java"
            print "Creating", junitName
            tfile = file(junitName, "w")
            for line in startBody[1:]:
                if line.strip().startswith("package"):
                    package = line.strip()
                elif line.strip().startswith("import"):
                    imports += line.strip() + "\n"
                else:
                    if line.strip() != "":
                        header += line.rstrip() + "\n"
            if package:
                print >>tfile, package
            if imports:
                print >>tfile, imports,
            print >>tfile, BeginFile % className,
            print >>tfile, header,
            for m in matches[1:]:
                methodName, methodBody = m.split("\n",1)
                methodName = methodName.strip()
                assert methodName.startswith("test"), \
                    "methodName must start with 'test'"
                print >>tfile, TestMethod % (
                    methodName, methodBody.rstrip())
            print >>tfile, EndFile % className

This program uses the "re" regular-expression module to quickly extract all the fragments of unit-test code. The regular expression pattern is precompiled with the compile() statement. You'll see that regular expressions in most languages are fairly similar, although Python's do not require as much escaping as Java's do. The above expression captures the text within the starting '/*~' and the ending '*/', with the '*' escaped because they are regular expression characters. The main part of the expression, '(.*?)', captures any character any number of times with the '.*', but the question mark means that the expression is not "greedy" and will thus stop the first time a '*/' is encountered. DOTALL is a flag indicating that the '.' will capture everything including newlines. Because the main part of the expression is contained within parentheses, this will be saved separately when the expression is used.

BeginFile, TestMethod and EndFile are multiline strings, denoted by the triple-quotes. They also contain '%s' characters, which allow printf-style formatting.

The main operation of the code begins with the os.walk() expression that walks every file in the directory. The list comprehension produces the full path of each of the Java files. Next, the contents of each Java file is read and passed through the findall() regular expression, which produces a list of each of the portions of the string that matches the previously-compiled regular expression, but only the part that was inside the parentheses of that regular expression.

If findall() doesn't find anything the matches object will be the special value None and the if statement will be false, so the program will go on to the next file. If it finds a match, the first block will contain the basic information about the unit test file, so matches[0] is "split" at its newlines to produce startBody. Line zero of startBody contains the unit test class name; the strip() method removes leading and trailing white space. We want this to begin with the capitalized word "Test" and the assertion guarantees this.

With the test class name we can open the file to contain the text of the JUnit program; this is assigned to the variable tfile. Each of the rest of the lines in startBody are examined to see if they are package or import statements; if so they are stored so they can be inserted in the JUnit program in the appropriate place, otherwise the lines are added to the header section.

The remaining matches (denoted by Python's list-slicing syntax, so matches[1:] means "from index 1 until the end of the list) are dealt with similarly, except that each of their first lines indicate the test method name, and the rest of the lines are the method body. The first and remaining lines of each method block are separated with the m.split("\n", 1) call, which does a single split (the number of splits is determined by the second argument) at the first newline. The results come back as a two- element tuple, each of which is assigned in the expression:

methodName, methodBody = m.split("\n",1)

The print statements use a redirection syntax, for example:

print >>tfile, EndFile % className

The '>>tfile' sends the results to tfile rather than the console. The '%' causes className to be substituted for the %s tag in the EndFile variable; if there is more than one element to be substituted the elements must be place in a tuple, as in:

print >>tfile, TestMethod % (methodName, methodBody.rstrip())

When this program is run on CountedListEmbedded.java, the result is:

import java.util.*;  // Any imports go here
import junit.framework.*;

public class TestCountedListEmbedded extends TestCase {
  // package statement goes here if necessary
  private CountedListEmbedded list =
    new CountedListEmbedded();
  public TestCountedListEmbedded() {
    for(int i = 0; i < 3; i++)
      list.add("" + i);
  }
  protected void setUp() {
    System.out.println("Set up for " + list.getId());
  }
  public void tearDown() {
    System.out.println("Tearing down " + list.getId());
  }
  private void
  compare(CountedListEmbedded lst, String[] strs) {
    Object[] array = lst.toArray();
    assertTrue("Arrays not the same length",
      array.length == strs.length);
    for(int i = 0; i < array.length; i++)
      assertEquals(strs[i], (String)array[i]);
  }
  public void testInsert() {
    System.out.println("Running testInsert()");
    assertEquals(list.size(), 3);
    list.add(1, "Insert");
    assertEquals(list.size(), 4);
    assertEquals(list.get(1), "Insert");
  }
  public void testReplace() {
    System.out.println("Running testReplace()");
    assertEquals(list.size(), 3);
    list.set(1, "Replace");
    assertEquals(list.size(), 3);
    assertEquals(list.get(1), "Replace");
  }
  public void testOrder() {
    System.out.println("Running testOrder()");
    compare(list, new String[] { "0", "1", "2" });
  }
  public void testRemove() {
    System.out.println("Running testRemove()");
    assertEquals(list.size(), 3);
    list.remove(1);
    assertEquals(list.size(), 2);
    compare(list, new String[] { "0", "2" });
  }
  public void testAddAll() {
    System.out.println("Running testAddAll()");
    list.addAll(Arrays.asList(new Object[] {
      "An", "African", "Swallow"}));
    assertEquals(list.size(), 6);
    compare(list, new String[] { "0", "1", "2",
       "An", "African", "Swallow" });
  }
  public static void main(String[] args) {
    // Invoke JUnit on the class:
    junit.textui.TestRunner.run(TestCountedListEmbedded.class);
  }
}

With this system, you simply keep the test code up-to-date within the source file, and generate the surrounding JUnit code automatically, every time you do a build.

The build system for Thinking in Java, 4th edition, automatically runs JUnitCreate.py, and then the results are incorporated into the Ant build.xml script so that the unit tests are automatically executed during a build. However, another Python program removes the inline unit tests before the code is automatically incorporated back into the book.

Feedback