Whenever I have a problem that can be solved with some sort of framework I ask a colleague for advice on the best framework to use. The answer I get is always a strong opinion in favour of a particular framework, say framework 'X'

“'X' is the best option by far the colleague will say.”

and usually once you start using it another colleague will come along and tell you that 'X' is completely wrong and that you would have been much better off using 'Y'

“Yes 'Y' is the best thing to use, you should have used that and then you would not be having the problems you are having. The second person will say.”

This is particularly annoying as the statement is correct, there would not be the problems if you swapped one framework for another - it would be an entirely different set of problems.

I digress...

I recently needed to start using a mocking framework for unit tests (Now that I am no longer using Groovy and Grails which has good support of its own). My choices boiled down to JMock or Easymock. Initially I thought I would use whatever was already being used within the department but I found that both were in use, so I was left with the option of taking the advice of colleagues...

This time, however, I did not get the answer I expected instead I got something else – indifference. I got answers like 'There both OK, use which you like' and 'I prefer JMock but I think Easymock has got a lot better recently'.

So not having any strong advice (be it good or bad) I just picked Easymock at random and got on with learning it. Having been using it for a few weeks I have found it very good.

My one criticism is that the base framework is only designed to work with interfaces. While I believe that programming to interfaces is a good idea I don't think that having an interface to match every single class you create is really necessary. Obviously I am not alone in this opinion as there is an easymock extension package which works with concrete classes and its great. It works in exactly the same way as the base package does for interfaces but you can create mocks from classes.

Both packages use the same API, just with different packages structures. An EasyMock class can be found in the package org.easymock and is used for interfaces only, while an identical EasyMock class can be found in org.easymock.classextension and is used for classes.
It is highly likely that both classes will be needed in any tests written where there is a mixture of classes and interfaces to mock out. The clash in class name then becomes an issue. You have to import one set of classes and reference the other with its fully qualified name:

This does not work:

 1 interface MyInterface {}
 2 class MyClass {}
 3 
 4 // some test
 5 import org.easymock.EasyMock;
 6 
 7 public void setUp() {
 8 	MyInterface mockMyInterface = EasyMock.createMock(MyInterface.class); // works	
 9 	MyClass mockMyClass = EasyMock.createMock(MyClass.class); // fails MyClass is not an interface
10 }

---

So you have to do this:

 1 interface MyInterface {}
 2 class MyClass {}
 3 
 4 // some test
 5 import org.easymock.EasyMock;
 6 
 7 public void setUp() {
 8 	MyInterface mockMyInterface = EasyMock.createMock(MyInterface.class); // works
 9 	MyClass mockMyClass = org.easymock.classextension.EasyMock.createMock(MyClass.class); // works – correct class used
10 }

---

This looks simple but you end up with more than just a createMock state for each mock. You have expect/replay/verify statements as well and quickly the tests become a bit combersome.

Java 5 and later provides a simple option in the form of static imports. Thus you can statically import one set of classes and access the methods directly and import the other in the normal way:

 1 interface MyInterface {}
 2 class MyClass {}
 3 
 4 // some test
 5 import static org.easymock.classextension.EasyMock.*;
 6 import org.easymock.EasyMock;
 7 
 8 public void setUp() {
 9 	MyInterface mockMyInterface = EasyMock.createMock(MyInterface.class); // works
10 	MyClass mockMyClass = createMock(MyClass.class); // works – method statically imported.
11 }

---

Slightly easier but still a bit of a problem.

Cue all the comments from people telling me I should have used JMock.

syntax highlighted by Code2HTML, v. 0.9.1

I have written this mostly to help me understand what was going on with my tests. When I first discovered the problem I could not get my head around it but I now think I understand and hope that if other people can understand it they can avoid repeating the problem

I use Linux on my development machine in a team where all other team members seem to be big fans of Windows. From time to time this causes me problems as code written on Windows fails on my machine. Usually this is because people have hard coded file paths for Windows in a test case. This is easy to fix but sometimes other things cause problems. One problem recently took me ages to track down and once I had found it I was not sure what the root cause was. Was it simply differences in Operating Systems or something more sinister - test cases that are not as atomic and robust as they could be?

The problem focuses around two integration tests and I have recreated two tests to demonstrate this. The tests may be a bit contrived but the problem I encountered in the project's tests were caused by code written in exactly the same way. In particular these examples could very easily be moved to a unit test but the reason the original code was an Integration test was that the AuditTrail objects were persisted to a database. I have removed the persistance as its not required to demonstrate this problem and it makes them easier to follow.

The two tests are AuditServiceIntegrationTests and PageControllerIntegrationTests.

AuditServiceIntegrationTest.groovy

Here is the code for the first test and the first object under test. The AuditTrail object is trivial - it has just two attributes: description and eventDate so the code is not shown here.

 

 1 package net.darren
 2 class AuditServiceIntegrationTests extends GroovyTestCase {
 3 
 4     def auditService;
 5 
 6     void testAuditTrailEventTimeChanges() {
 7 
 8         AuditTrail trail = auditService.createAuditTrail("Feed Updated: a_feed")
 9         Thread.sleep(500)
10         AuditTrail trail1 =  auditService.createAuditTrail("Feed a_feed Failed: Null Pointer Exception")
11 
12         assertFalse "Time should be different", trail.eventDate.equals(trail1.eventDate);
13     }
14 }

---

 

 

 1 package net.darren
 2 class AuditService {
 3 
 4     static transactional = false
 5 
 6     public AuditTrail createAuditTrail(String auditDescription) {
 7         AuditTrail trail = new AuditTrail(eventDate: getDate(), description: auditDescription)
 8 	return trail
 9     }
10 
11 
12     def getDate = {
13         new Date()
14     }
15 }

---

 

This is a simple test that calls the createAuditTrail() method which returns a new AuditTrail object. It then waits for a small amount of time before calling it again to create a second instance of the AuditTrail class. The test is that each object has a different eventTime because they were created at different times. The code under test is also simple. The interesting part to note is the getdate() closure in AuditService this was probably done deliberately so that it could be stubbed or mocked during testing. This test on its own passes fine, the two dates are different because of the delay between each call to auditService.createAuditTrail()

Enter another test: PageControllerIntegrationTests

Now I introduce another test. This one requires the getDate() method to be stubbed out which is done in the last line of the setUp method. The stub version changes the functionality so that the time returned is always the same (Different to the real functionality which always returns the currently time which, in my universe at least, is constantly changing). There are no actual tests for this object as it does not matter for the purposes of problem - the damage is already done in the setUp() method.

 

 1 package net.darren
 2 class PageControllerIntegrationTests extends GroovyTestCase {
 3 
 4     PageController controller
 5     def auditService;
 6     Calendar constant = Calendar.getInstance()
 7 
 8     void setUp() {
 9         controller = new PageController(auditService:auditService);
10         Calendar now = Calendar.getInstance()
11         now.setTime(constant.getTime())
12         now.add(Calendar.YEAR, 1)
13         auditService.getDate = {
14 		println "stub created in PageControllerIntegrationTests has been called";
15 		return now.getTime()
16 	}
17     }
18 
19     void testSomething() {
20         /* no need to do anything damage is already done */
21     }
22 }

---

 

You may run this code and all tests will pass, for there is one last twist to this problem. The order the tests run matters. How you get them to run in a different order is dependent upon platform, On my system (Fedora 8 and Java 1.6.0_06 64 bit) I moved the test classes to different packages but other systems may require different re-factoring to see the behavior I describe. Windows XP seems to have a much more logical loading order - it does it alphabetically so AuditServiceIntegrationTests runs followed by the PageControllerIntegrationTests. So just rename PageControllerIntegrationTests to AardvarkPageControllerIntegrationTests to make it run before AuditServiceIntegrationTests

If on my system I keep both tests in the same package then AuditServiceIntegrationTests runs followed by the PageControllerIntegrationTests and everything is fine: both pass.

-------------------------------------------------------
Running 2 Integration Tests...
Running test net.darren.AuditServiceIntegrationTests...
testAuditTrailEventTimeChanges...SUCCESS
Running test net.darren.PageControllerIntegrationTests...
testSomething...SUCCESS
Integration Tests Completed in 1180ms
-------------------------------------------------------

If, however, I move PageControllerIntegrationTests from package net.darren.PageControllerIntegrationTests to net.PageControllerIntegrationTests they run PageControllerIntegrationTests followed by AuditServiceIntegrationTests and AuditServiceIntegrationTests will fail.

-------------------------------------------------------
Running 2 Integration Tests...
Running test net.PageControllerIntegrationTests...
testSomething...SUCCESS
Running test net.darren.AuditServiceIntegrationTests...
testAuditTrailEventTimeChanges...FAILURE
Integration Tests Completed in 1100ms
-------------------------------------------------------

The reason it fails in this case is that the call to getDate() in AuditService during the execution of AuditServiceIntegrationTests actually calls the stub version that was set in PageControllerIntegrationTests and since this always returns the same time both objects are created with the same time and the assertion fails.

Tests should run without interferring with each other and the order they run should not be an issue. So why is it a problem in this case? Note that nowhere in my code is the object auditService instantiated and there is not even a whiff of a NullPointerException. This is because the tests rely on Spring and its autowiring functionality (autowiring means to look for objects defined in Spring with the same name as your object property and automatically assign the object to the property) and as all objects in Spring are singletons by default, once you stub out a closure it stays stubbed out. The quick answer is to create a new version of the object to be stubbed in the setup() method that overwrites the one assigned by spring and stub that one. That way all other tests will continue to use the unmolested spring created version.

Really I was left not knowing what to blame, is it just unfortunate that my system loads and runs the tests in a different order to everyone else? Is it bad practice to rely on the running order of tests for them to work? Or should we be avioding stubs in Integration tests? Is it good practice to override the spring autowiring stuff by creating your own version of a property in the setUp method? Should we be refactoring some of our Integration tests into unit tests? Does everyone in the team understand spring autowiring and is aware that Integration tests use it? (I was not aware it was being used when I first started writing integration tests in Grails) Have I asked enough questions?

The source code as an IntelliJ project is available here.
TestingWithStubs.zip (357.31 kb)

syntax highlighted by Code2HTML, v. 0.9.1

I should be moving to a new job soon which will mean no more Groovy and Grails. Its been an interesting few months but I'll be glad to be writing Java again. Goovy is OK but I don't have the head for dynamic languages and never seem to make the most of there capabilities. Perhaps more experience would help and the new stuff my current team are about to start working on may have given me the experience I need. Unfortunately I will not get to find out because I want the challenge of a new position and extra responsibility, more money will be nice too...