Found the Problem

At 10am we were down to 2 failures.

In looking at one of the acceptance tests I found the problem, by running it locally.

It failed when FF went through the same flow but then got a new overlay on one of the pages. Here's the deal... our acceptance tests are in two frameworks:
a) homegrown groovy/spock framework
b) GEB framework

We're actively trying to convert all acceptance tests to GEB. However, this has left older tests not yet converted in the lurch. In this case, the code we needed to bypass the new overlay, wasn't in the test.

Our decision was to convert the test to the new GEB framework.

So the developer on my sprint team: Vijay - he migrated the test over to GEB. I sat down with him and went through the flow of converting the test. It was pretty cool. Seems so much easier when a developer does it.

9 Failures

This morning I came in to work and checked in with Jenkins Continuous Integration/Deployment results.

We had 9 failures reported in Jenkins. I pulled in the latest code through Eclipse:
team > Update to Head

I ran these tests locally, against a QA environment:
Run As JUnit tests

I saw the problem. We have a new page that appears if a user goes to their Dashboard, after logging in. This new page, wasn't being accounted for in these tests... but other tests worked fine.

Turns out it's because of our migration to GEB. We have tests in two frameworks currently. The GEB framework and the older homegrown Groovy/Spock framework. The fix to pass this new page, was put into the older framework. However it wasn't put into the newer GEB tests.

So I was right about the problem.

However, my fix, which was a simple if statement - didn't work out so well. I pulled in a developer (Vijay) and he corrected my code.

We're considering a better solution, which is to create users who don't pull these features (like having the necessary data populated at the beginning of the test) so the feature wouldn't pop up anyway.

Resolving some Non-Determinism: Create Users

Originally we pulled existing users from our Database for our Acceptance Tests.

This worked for the most part, and then we started getting random errors. Tests failing one day, working the next.

It was discovered some of the test users randomly called from the database were modified users. Their data was bad: bad passwords, bad info, or just in weird states.

To resolve this, we started creating users (rather then pulling them from the db) by using an quick API. Users are created in a matter of seconds and they are starting from a clean slate, with known attributes.

This has significantly lowered the random failures we were getting.