I have written this mostly to help me understand what was going on with my tests. When I first discovered the problem I could not get my head around it but I now think I understand and hope that if other people can understand it they can avoid repeating the problem
I use Linux on my development machine in a team where all other team members seem to be big fans of Windows. From time to time this causes me problems as code written on Windows fails on my machine. Usually this is because people have hard coded file paths for Windows in a test case. This is easy to fix but sometimes other things cause problems. One problem recently took me ages to track down and once I had found it I was not sure what the root cause was. Was it simply differences in Operating Systems or something more sinister - test cases that are not as atomic and robust as they could be?
The problem focuses around two integration tests and I have recreated two tests to demonstrate this. The tests may be a bit contrived but the problem I encountered in the project's tests were caused by code written in exactly the same way. In particular these examples could very easily be moved to a unit test but the reason the original code was an Integration test was that the AuditTrail objects were persisted to a database. I have removed the persistance as its not required to demonstrate this problem and it makes them easier to follow.
The two tests are AuditServiceIntegrationTests and PageControllerIntegrationTests.
AuditServiceIntegrationTest.groovy
Here is the code for the first test and the first object under test. The AuditTrail object is trivial - it has just two attributes: description and eventDate so the code is not shown here.
1 package net.darren
2 class AuditServiceIntegrationTests extends GroovyTestCase {
3
4 def auditService;
5
6 void testAuditTrailEventTimeChanges() {
7
8 AuditTrail trail = auditService.createAuditTrail("Feed Updated: a_feed")
9 Thread.sleep(500)
10 AuditTrail trail1 = auditService.createAuditTrail("Feed a_feed Failed: Null Pointer Exception")
11
12 assertFalse "Time should be different", trail.eventDate.equals(trail1.eventDate);
13 }
14 }
1 package net.darren
2 class AuditService {
3
4 static transactional = false
5
6 public AuditTrail createAuditTrail(String auditDescription) {
7 AuditTrail trail = new AuditTrail(eventDate: getDate(), description: auditDescription)
8 return trail
9 }
10
11
12 def getDate = {
13 new Date()
14 }
15 }
This is a simple test that calls the createAuditTrail() method which returns a new AuditTrail object. It then waits for a small amount of time before calling it again to create a second instance of the AuditTrail class. The test is that each object has a different eventTime because they were created at different times. The code under test is also simple. The interesting part to note is the getdate() closure in AuditService this was probably done deliberately so that it could be stubbed or mocked during testing. This test on its own passes fine, the two dates are different because of the delay between each call to auditService.createAuditTrail()
Enter another test: PageControllerIntegrationTests
Now I introduce another test. This one requires the getDate() method to be stubbed out which is done in the last line of the setUp method. The stub version changes the functionality so that the time returned is always the same (Different to the real functionality which always returns the currently time which, in my universe at least, is constantly changing). There are no actual tests for this object as it does not matter for the purposes of problem - the damage is already done in the setUp() method.
1 package net.darren
2 class PageControllerIntegrationTests extends GroovyTestCase {
3
4 PageController controller
5 def auditService;
6 Calendar constant = Calendar.getInstance()
7
8 void setUp() {
9 controller = new PageController(auditService:auditService);
10 Calendar now = Calendar.getInstance()
11 now.setTime(constant.getTime())
12 now.add(Calendar.YEAR, 1)
13 auditService.getDate = {
14 println "stub created in PageControllerIntegrationTests has been called";
15 return now.getTime()
16 }
17 }
18
19 void testSomething() {
20 /* no need to do anything damage is already done */
21 }
22 }
You may run this code and all tests will pass, for there is one last twist to this problem. The order the tests run matters. How you get them to run in a different order is dependent upon platform, On my system (Fedora 8 and Java 1.6.0_06 64 bit) I moved the test classes to different packages but other systems may require different re-factoring to see the behavior I describe. Windows XP seems to have a much more logical loading order - it does it alphabetically so AuditServiceIntegrationTests runs followed by the PageControllerIntegrationTests. So just rename PageControllerIntegrationTests to AardvarkPageControllerIntegrationTests to make it run before AuditServiceIntegrationTests
If on my system I keep both tests in the same package then AuditServiceIntegrationTests runs followed by the PageControllerIntegrationTests and everything is fine: both pass.
-------------------------------------------------------
Running 2 Integration Tests...
Running test net.darren.AuditServiceIntegrationTests...
testAuditTrailEventTimeChanges...SUCCESS
Running test net.darren.PageControllerIntegrationTests...
testSomething...SUCCESS
Integration Tests Completed in 1180ms
-------------------------------------------------------
If, however, I move PageControllerIntegrationTests from package net.darren.PageControllerIntegrationTests to net.PageControllerIntegrationTests they run PageControllerIntegrationTests followed by AuditServiceIntegrationTests and AuditServiceIntegrationTests will fail.
-------------------------------------------------------
Running 2 Integration Tests...
Running test net.PageControllerIntegrationTests...
testSomething...SUCCESS
Running test net.darren.AuditServiceIntegrationTests...
testAuditTrailEventTimeChanges...FAILURE
Integration Tests Completed in 1100ms
-------------------------------------------------------
The reason it fails in this case is that the call to getDate() in AuditService during the execution of AuditServiceIntegrationTests actually calls the stub version that was set in PageControllerIntegrationTests and since this always returns the same time both objects are created with the same time and the assertion fails.
Tests should run without interferring with each other and the order they run should not be an issue. So why is it a problem in this case? Note that nowhere in my code is the object auditService instantiated and there is not even a whiff of a NullPointerException. This is because the tests rely on Spring and its autowiring functionality (autowiring means to look for objects defined in Spring with the same name as your object property and automatically assign the object to the property) and as all objects in Spring are singletons by default, once you stub out a closure it stays stubbed out. The quick answer is to create a new version of the object to be stubbed in the setup() method that overwrites the one assigned by spring and stub that one. That way all other tests will continue to use the unmolested spring created version.
Really I was left not knowing what to blame, is it just unfortunate that my system loads and runs the tests in a different order to everyone else? Is it bad practice to rely on the running order of tests for them to work? Or should we be avioding stubs in Integration tests? Is it good practice to override the spring autowiring stuff by creating your own version of a property in the setUp method? Should we be refactoring some of our Integration tests into unit tests? Does everyone in the team understand spring autowiring and is aware that Integration tests use it? (I was not aware it was being used when I first started writing integration tests in Grails) Have I asked enough questions?
The source code as an IntelliJ project is available here.
TestingWithStubs.zip (357.31 kb)
syntax highlighted by Code2HTML, v. 0.9.1