REST Basics

There's a great blog about REST (what it is, explanation and examples) at:
http://rest.elkstein.org/

New Version of GEB

Thanks to some developer help, we now have a newer version of GEB running. This new version may help rule out some of our non deterministic tests.

The idea is that it now handles webdriver and selenium a lot better.

Also - the older GEB couldn't run screenshot compatibility. Now the latest and greatest GEB brings that back. So we run the Continuous Integration tests... check the results, in Hudson - and have linked .jpg's of each screen in the test run.

Jez Humble

Jez Humble was at our offices today, giving a talk on Continuous Integration... unfortunately I missed it - due to working on some issues with our Acceptance Tests, and filling in while my Manager is out with some other work.

Ah well, maybe next time!

Jez Humble runs: http://continuousdelivery.com/
He also is a published author: http://www.amazon.com/gp/product/0321601912?tag=contindelive-20

Found the Problem

At 10am we were down to 2 failures.

In looking at one of the acceptance tests I found the problem, by running it locally.

It failed when FF went through the same flow but then got a new overlay on one of the pages. Here's the deal... our acceptance tests are in two frameworks:
a) homegrown groovy/spock framework
b) GEB framework

We're actively trying to convert all acceptance tests to GEB. However, this has left older tests not yet converted in the lurch. In this case, the code we needed to bypass the new overlay, wasn't in the test.

Our decision was to convert the test to the new GEB framework.

So the developer on my sprint team: Vijay - he migrated the test over to GEB. I sat down with him and went through the flow of converting the test. It was pretty cool. Seems so much easier when a developer does it.

Bye Green, Hello Red

5 new failures in the same trunk that was green yesterday.

Having an issue with the old "element not found in cache."

Googling it, I came up with:
http://code.google.com/p/selenium/issues/detail?id=1887

Comment 3 by leo.lath...@gmail.com, Jun 25, 2011

This happens for me a lot. I suppose that my page is too dynamic and it invalidates the references too often.
So I had to add a catch for the StaleElementReferenceException and retry the instruction for every selenium find. For me, that doesn't make much sense.

If the cached element is stale, shouldn't the cache miss trigger a new request to refresh the element?

Comment 4 by savag...@mail.ru, Jun 26, 2011

Thanks for your advice. I added a catch and everything works, but... it looks like a workaround or something like that. I thought it should work automatically.

http://www.echographia.com/blog/2011/01/13/element-not-found-in-the-cache/

Green!

Our tests are green again.

Jenkins - reports that the latest trunk build has no Acceptance Test failures.

That's awesome.

We do have some skipped tests - that's the next thing we're tackling... how to reduce these non-deterministic errors.

9 Failures

This morning I came in to work and checked in with Jenkins Continuous Integration/Deployment results.

We had 9 failures reported in Jenkins. I pulled in the latest code through Eclipse:
team > Update to Head

I ran these tests locally, against a QA environment:
Run As JUnit tests

I saw the problem. We have a new page that appears if a user goes to their Dashboard, after logging in. This new page, wasn't being accounted for in these tests... but other tests worked fine.

Turns out it's because of our migration to GEB. We have tests in two frameworks currently. The GEB framework and the older homegrown Groovy/Spock framework. The fix to pass this new page, was put into the older framework. However it wasn't put into the newer GEB tests.

So I was right about the problem.

However, my fix, which was a simple if statement - didn't work out so well. I pulled in a developer (Vijay) and he corrected my code.

We're considering a better solution, which is to create users who don't pull these features (like having the necessary data populated at the beginning of the test) so the feature wouldn't pop up anyway.

Resolving some Non-Determinism: Create Users

Originally we pulled existing users from our Database for our Acceptance Tests.

This worked for the most part, and then we started getting random errors. Tests failing one day, working the next.

It was discovered some of the test users randomly called from the database were modified users. Their data was bad: bad passwords, bad info, or just in weird states.

To resolve this, we started creating users (rather then pulling them from the db) by using an quick API. Users are created in a matter of seconds and they are starting from a clean slate, with known attributes.

This has significantly lowered the random failures we were getting.

Debug Mode

My friend and fellow teammate - Trung - has been spending time with me to help me understand how to debug failing tests and code.

He showed me how to put break points in Eclipse and then to run the test in debug mode. When it hits the breakpoint, the test can move line by line of the code - by hitting F6. F8, will run through the rest of it without break/stopping.

This has helped me zoom in closer to problems I was reporting. Such as a variable not being set before the page continues.

First Acceptance Test!

Pretty excited. I put together my first acceptance test today.

One problem we're running into is the lack of ID's on elements. We don't want to use x Paths to find elements on the page. So when I created this, I hit this problem... my dev's Trung and Victor were able to add ID's for me and I put together a communication test!

I login as a matched user, select a match, pull up their details and send an icebreaker. It's pretty cool!

So in case I've been remiss in describing these acceptance tests... The goal is this:
Each time a developer commits code, a build goes out and all the acceptance tests are run in a Continuous Integration environment.

These tests call a browser, and run through the automation of the browser to test the code. The more tests we have, the better and faster we'll catch a problem introduced through a commit.

How we automate this is in a Groovy/ Spock framework.

We have pages and tests. under the pages, we have each page being it's own Groovy Class. Then the tests themselves call each page, as an objects. This page object methodology makes it easier to understand whats going on, create tests and see where the failures are occurring.

For example:
If I wanted a test that would
Go to a Login page
Send valid login credentials
Get to a welcome page on the other side
Verify I'm at the welcome page
Logout
Verify I'm back at the login page

We set it up by making the page objects:
Loginpage.groovy
Welcomepage.groovy

The test:
logintest.groovy
would call the loginpage class.

Like
loginpage(username, password)
.logout()

HTMLUnit

Victor and I set up forking logic to support HTMLUnit and FF. That way we'll have the ability to call a headless client test or a Firefox browser in the acceptance test run.

setup: properties files.

We're now using a properties file to define which db we are running acceptance tests against. Since we have several QA environments, we need to be able to call these different login scripts to each db.

This is pretty cool. Victor showed me how to set up the properties file to support our different environments

I work with some really cool Dev's... so on my Pilot team, they took the time out to help me with my Eclipse env issues from this weekend.

The DB errors were only specific to one specific test. My other tests were good (db connection wise) so i was on the right track. that feels good.

My acceptance tests weren't running because I was not extending them correctly! Victor, the dev I'm working with, showed me that I need to set up the Acceptance Tests and extend them correctly to the right class. This resolved the JUit issue - now I see "run as JUnit" avail for my tests.

Eclipse... grr....

This was a hair pulling weekend.

I got pretty pissed off with Eclipse. it somewhat balanced with my initial Java application success in the last post. Now I'm frustrated as I can't run any of the damn acceptance tests!

It's an eclipse environment problem. Eclipse is complaining it can't communicate with the database. I'm also not getting the JUnit option when running the classes i've made... grr.

My First Working App in JAVA!

OK - This weekend I got Eclipse set up. I've been working the past couple weeks in picking up Java. I found I couldn't really find resources to teach Groovy without first learning Java. Most Groovy resources are written to Java dev's. So I started with Java. I've been learning the JAVA basics so I can now pick up Groovy.

Backwards, I know, but I don't know how else to attempt this!

I've successfully written a few small applications in Java. I wrote a small application that uses selenium libraries to go to Google and pass a search term, then pull the results and report them back in Eclipse/IDE.

That was cool. My first working app in Java.

Leaving the Selenium IDE Crutches

When we started the pilot, I would write tests in Selenium IDE (just recording actions) then convert those scripts to Groovy and pass them to the developers. These developers would then write out the actual tests.

The Architect has made a request today: He wants QA to start writing the acceptance tests.

I'm a bit nervous - this is entirely new to me. New ground... but, damnit i'm going to do it. I'm going to get this done. I'm taking the work laptop home tonight and I have these goals:
a) get eclipse installed
b) download our acceptance tests repository locally
c) get maven and maven plugin for eclipse installed
d) install groovy
e) load in the acceptance tests correctly

So I've been on the Pilot Program to achieve a foothold into the continuous integration/deployment realm.

We have an architect leading our work. His name is Josh. Josh has worked with Thoughtworks and has some interesting ideas.

Our main goal is to get a framework of acceptance tests that extend out as we create new features in this team.

As we get a new project, we will create the tickets with a format of: Given, When, Then. All projects are consumer driven.

These tickets with the Given, When, Then breakdown are then translated into acceptance tests.

The acceptance test framework is this built with each new feature or enhancement.

An example is this:
Feature ABC:
As a user, I want to see XXXXXXXX when I enter my XXXXXXXX.

Given a logged in user

When they enter XXXXXXXXX

Then they will see XXXXXXXXX.

This is how the ticket would be formatted.

From that, the dev's are writing acceptance tests in Groovy - using Spock - to keep the same framework.

It's been decided that we'll be using Groovy and Spock for the Acceptance Tests.

First Post

Well I was informed I'll be on a new and exciting project.

Continuous Integration and Continuous Deployment is a big concept these days. A lot of people talk about it, and have various ways of implementing it.

Here at eHarmony - we'll be tackling this ourselves.

My background is QA. I'm a QA Eng. I haven't used an IDE and haven't developed in Java before.

I'm a bit timid of what to expect. This is a journal of my experience along the way of this process.