Found the Problem

At 10am we were down to 2 failures.

In looking at one of the acceptance tests I found the problem, by running it locally.

It failed when FF went through the same flow but then got a new overlay on one of the pages. Here's the deal... our acceptance tests are in two frameworks:
a) homegrown groovy/spock framework
b) GEB framework

We're actively trying to convert all acceptance tests to GEB. However, this has left older tests not yet converted in the lurch. In this case, the code we needed to bypass the new overlay, wasn't in the test.

Our decision was to convert the test to the new GEB framework.

So the developer on my sprint team: Vijay - he migrated the test over to GEB. I sat down with him and went through the flow of converting the test. It was pretty cool. Seems so much easier when a developer does it.

Bye Green, Hello Red

5 new failures in the same trunk that was green yesterday.

Having an issue with the old "element not found in cache."

Googling it, I came up with:
http://code.google.com/p/selenium/issues/detail?id=1887

Comment 3 by leo.lath...@gmail.com, Jun 25, 2011

This happens for me a lot. I suppose that my page is too dynamic and it invalidates the references too often.
So I had to add a catch for the StaleElementReferenceException and retry the instruction for every selenium find. For me, that doesn't make much sense.

If the cached element is stale, shouldn't the cache miss trigger a new request to refresh the element?

Comment 4 by savag...@mail.ru, Jun 26, 2011

Thanks for your advice. I added a catch and everything works, but... it looks like a workaround or something like that. I thought it should work automatically.

http://www.echographia.com/blog/2011/01/13/element-not-found-in-the-cache/

Green!

Our tests are green again.

Jenkins - reports that the latest trunk build has no Acceptance Test failures.

That's awesome.

We do have some skipped tests - that's the next thing we're tackling... how to reduce these non-deterministic errors.

9 Failures

This morning I came in to work and checked in with Jenkins Continuous Integration/Deployment results.

We had 9 failures reported in Jenkins. I pulled in the latest code through Eclipse:
team > Update to Head

I ran these tests locally, against a QA environment:
Run As JUnit tests

I saw the problem. We have a new page that appears if a user goes to their Dashboard, after logging in. This new page, wasn't being accounted for in these tests... but other tests worked fine.

Turns out it's because of our migration to GEB. We have tests in two frameworks currently. The GEB framework and the older homegrown Groovy/Spock framework. The fix to pass this new page, was put into the older framework. However it wasn't put into the newer GEB tests.

So I was right about the problem.

However, my fix, which was a simple if statement - didn't work out so well. I pulled in a developer (Vijay) and he corrected my code.

We're considering a better solution, which is to create users who don't pull these features (like having the necessary data populated at the beginning of the test) so the feature wouldn't pop up anyway.

Resolving some Non-Determinism: Create Users

Originally we pulled existing users from our Database for our Acceptance Tests.

This worked for the most part, and then we started getting random errors. Tests failing one day, working the next.

It was discovered some of the test users randomly called from the database were modified users. Their data was bad: bad passwords, bad info, or just in weird states.

To resolve this, we started creating users (rather then pulling them from the db) by using an quick API. Users are created in a matter of seconds and they are starting from a clean slate, with known attributes.

This has significantly lowered the random failures we were getting.

Debug Mode

My friend and fellow teammate - Trung - has been spending time with me to help me understand how to debug failing tests and code.

He showed me how to put break points in Eclipse and then to run the test in debug mode. When it hits the breakpoint, the test can move line by line of the code - by hitting F6. F8, will run through the rest of it without break/stopping.

This has helped me zoom in closer to problems I was reporting. Such as a variable not being set before the page continues.

First Acceptance Test!

Pretty excited. I put together my first acceptance test today.

One problem we're running into is the lack of ID's on elements. We don't want to use x Paths to find elements on the page. So when I created this, I hit this problem... my dev's Trung and Victor were able to add ID's for me and I put together a communication test!

I login as a matched user, select a match, pull up their details and send an icebreaker. It's pretty cool!

So in case I've been remiss in describing these acceptance tests... The goal is this:
Each time a developer commits code, a build goes out and all the acceptance tests are run in a Continuous Integration environment.

These tests call a browser, and run through the automation of the browser to test the code. The more tests we have, the better and faster we'll catch a problem introduced through a commit.

How we automate this is in a Groovy/ Spock framework.

We have pages and tests. under the pages, we have each page being it's own Groovy Class. Then the tests themselves call each page, as an objects. This page object methodology makes it easier to understand whats going on, create tests and see where the failures are occurring.

For example:
If I wanted a test that would
Go to a Login page
Send valid login credentials
Get to a welcome page on the other side
Verify I'm at the welcome page
Logout
Verify I'm back at the login page

We set it up by making the page objects:
Loginpage.groovy
Welcomepage.groovy

The test:
logintest.groovy
would call the loginpage class.

Like
loginpage(username, password)
.logout()