Debugging non-deterministic tests in PHPUnit

You have a test that fails when run as part of your test suite. You re-run the test on its own, and it passes. What gives? More than likely you have a non-deterministic test, and probably a test that lacks isolation.

Here’s a tip for debugging tests that are breaking isolation: a PHPUnit Listener can check which of your test cases is changing the environment and causing a failure of a later test case. For example, suppose your tests rely on the APP_ENV environment variable being set to test. The following PHPUnit listener will check this before every test:

class AppEnvIsTestListener extends PHPUnit_Framework_BaseTestListener
{

  public function startTest(PHPUnit_Framework_Test $test) {
    if (getenv("APP_ENV") !== "test") {
      echo "A prior test has changed APP_ENV!\n";
    }
  }
}

To use the Listener, you’ll need to add it to your phpunit.xml:

<phpunit>
  ...

  <listeners>
    <listener
      class="AppEnvIsTestListener"
      file="app/test/AppEnvIsTestListener.php"
    />
  </listeners>
</phpunit>

Combine this with the --testdox flag to phpunit and you’ll get a very basic glimpse into which of your tests is leaking environment changes, and causing a later test to fail:

$ phpunit --filter EnvTest --testdox
PHPUnit 5.7.27 by Sebastian Bergmann and contributors.

Acme\EnvTest
 [x] Tests a thing
 [x] Tests another thing
A prior test has changed APP_ENV!
 [x] Tests a thing that does not depend on APP_ENV
A prior test has changed APP_ENV!
 [ ] Tests a thing that depends on APP_ENV
A prior test has changed APP_ENV!
 [ ] Tests another thing that depends on APP_ENV

Above, we can now see that the Tests another thing test case is leaking a change to APP_ENV and is likely causing the last two tests to fail.

Automating Reminders with AppleScript

Occasionally, I need to provision a complete replica of our AWS production to get my work done. As you might imagine, this incurs a fairly hefty hourly charge and we try to avoid running this setup for any longer than necessary. Although I’ve tried to develop the habit of checking the AWS consoles before heading home for the day, I sometimes forget. We then run a completely unused production stack overnight, wasting a whole bunch of money and resources.

Automatically shutting down the replica every evening seemed like the obvious solution. But the trouble is that sometimes I do want to run the stack for a couple of extra hours or even overnight whilst a long-running test finishes.

And then it occurred to me: perhaps I should adapt my provisioning script to also create a task in Apple’s Reminders app.

Here’s the code:

# add_teardown_reminder.applescript

tell application "Reminders"

  # Calculate date time for midnight today
  set currentDay to (current date) - (time of (current date))
  # Calculate date time for 1700 today
  set theDate to currentDay + (17 * hours)

  # Select the relevant list in Reminders.app
  set myList to list "Work"

  tell myList
    # Create the reminder
    set newReminder to make new reminder
    set name of newReminder to "Teardown test servers"
    set remind me date of newReminder to theDate
  end tell
end tell

This script calculates a date time for today at midnight, adds 17 hours to get a due date of “5pm today” and then adds the task to the Work list in Reminders.app.

Running the script from a shell script is simple:

osascript add_teardown_reminder.applescript

I’d not dabbled with AppleScript very much before, but getting this done took around 10 minutes. The “Open Dictionary” tool in Apple’s Script Editor is an excellent way to explore the APIs provided by Reminders.app (and other applications), and I picked up a few tips from Federico Viticci.

Best of all, our AWS account won’t be wasting midnight oil anymore.