At 10am we were down to 2 failures.
In looking at one of the acceptance tests I found the problem, by running it locally.
It failed when FF went through the same flow but then got a new overlay on one of the pages. Here's the deal... our acceptance tests are in two frameworks:
a) homegrown groovy/spock framework
b) GEB framework
We're actively trying to convert all acceptance tests to GEB. However, this has left older tests not yet converted in the lurch. In this case, the code we needed to bypass the new overlay, wasn't in the test.
Our decision was to convert the test to the new GEB framework.
So the developer on my sprint team: Vijay - he migrated the test over to GEB. I sat down with him and went through the flow of converting the test. It was pretty cool. Seems so much easier when a developer does it.
Showing posts with label Acceptance Tests. Show all posts
Showing posts with label Acceptance Tests. Show all posts
29 July 2011
18 July 2011
9 Failures
This morning I came in to work and checked in with Jenkins Continuous Integration/Deployment results.
We had 9 failures reported in Jenkins. I pulled in the latest code through Eclipse:
team > Update to Head
I ran these tests locally, against a QA environment:
Run As JUnit tests
I saw the problem. We have a new page that appears if a user goes to their Dashboard, after logging in. This new page, wasn't being accounted for in these tests... but other tests worked fine.
Turns out it's because of our migration to GEB. We have tests in two frameworks currently. The GEB framework and the older homegrown Groovy/Spock framework. The fix to pass this new page, was put into the older framework. However it wasn't put into the newer GEB tests.
So I was right about the problem.
However, my fix, which was a simple if statement - didn't work out so well. I pulled in a developer (Vijay) and he corrected my code.
We're considering a better solution, which is to create users who don't pull these features (like having the necessary data populated at the beginning of the test) so the feature wouldn't pop up anyway.
We had 9 failures reported in Jenkins. I pulled in the latest code through Eclipse:
team > Update to Head
I ran these tests locally, against a QA environment:
Run As JUnit tests
I saw the problem. We have a new page that appears if a user goes to their Dashboard, after logging in. This new page, wasn't being accounted for in these tests... but other tests worked fine.
Turns out it's because of our migration to GEB. We have tests in two frameworks currently. The GEB framework and the older homegrown Groovy/Spock framework. The fix to pass this new page, was put into the older framework. However it wasn't put into the newer GEB tests.
So I was right about the problem.
However, my fix, which was a simple if statement - didn't work out so well. I pulled in a developer (Vijay) and he corrected my code.
We're considering a better solution, which is to create users who don't pull these features (like having the necessary data populated at the beginning of the test) so the feature wouldn't pop up anyway.
02 May 2011
Resolving some Non-Determinism: Create Users
Originally we pulled existing users from our Database for our Acceptance Tests.
This worked for the most part, and then we started getting random errors. Tests failing one day, working the next.
It was discovered some of the test users randomly called from the database were modified users. Their data was bad: bad passwords, bad info, or just in weird states.
To resolve this, we started creating users (rather then pulling them from the db) by using an quick API. Users are created in a matter of seconds and they are starting from a clean slate, with known attributes.
This has significantly lowered the random failures we were getting.
This worked for the most part, and then we started getting random errors. Tests failing one day, working the next.
It was discovered some of the test users randomly called from the database were modified users. Their data was bad: bad passwords, bad info, or just in weird states.
To resolve this, we started creating users (rather then pulling them from the db) by using an quick API. Users are created in a matter of seconds and they are starting from a clean slate, with known attributes.
This has significantly lowered the random failures we were getting.
03 March 2011
First Acceptance Test!
Pretty excited. I put together my first acceptance test today.
One problem we're running into is the lack of ID's on elements. We don't want to use x Paths to find elements on the page. So when I created this, I hit this problem... my dev's Trung and Victor were able to add ID's for me and I put together a communication test!
I login as a matched user, select a match, pull up their details and send an icebreaker. It's pretty cool!
So in case I've been remiss in describing these acceptance tests... The goal is this:
Each time a developer commits code, a build goes out and all the acceptance tests are run in a Continuous Integration environment.
These tests call a browser, and run through the automation of the browser to test the code. The more tests we have, the better and faster we'll catch a problem introduced through a commit.
How we automate this is in a Groovy/ Spock framework.
We have pages and tests. under the pages, we have each page being it's own Groovy Class. Then the tests themselves call each page, as an objects. This page object methodology makes it easier to understand whats going on, create tests and see where the failures are occurring.
For example:
If I wanted a test that would
Go to a Login page
Send valid login credentials
Get to a welcome page on the other side
Verify I'm at the welcome page
Logout
Verify I'm back at the login page
We set it up by making the page objects:
Loginpage.groovy
Welcomepage.groovy
The test:
logintest.groovy
would call the loginpage class.
Like
loginpage(username, password)
.logout()
One problem we're running into is the lack of ID's on elements. We don't want to use x Paths to find elements on the page. So when I created this, I hit this problem... my dev's Trung and Victor were able to add ID's for me and I put together a communication test!
I login as a matched user, select a match, pull up their details and send an icebreaker. It's pretty cool!
So in case I've been remiss in describing these acceptance tests... The goal is this:
Each time a developer commits code, a build goes out and all the acceptance tests are run in a Continuous Integration environment.
These tests call a browser, and run through the automation of the browser to test the code. The more tests we have, the better and faster we'll catch a problem introduced through a commit.
How we automate this is in a Groovy/ Spock framework.
We have pages and tests. under the pages, we have each page being it's own Groovy Class. Then the tests themselves call each page, as an objects. This page object methodology makes it easier to understand whats going on, create tests and see where the failures are occurring.
For example:
If I wanted a test that would
Go to a Login page
Send valid login credentials
Get to a welcome page on the other side
Verify I'm at the welcome page
Logout
Verify I'm back at the login page
We set it up by making the page objects:
Loginpage.groovy
Welcomepage.groovy
The test:
logintest.groovy
would call the loginpage class.
Like
loginpage(username, password)
.logout()
24 February 2011
Eclipse... grr....
This was a hair pulling weekend.
I got pretty pissed off with Eclipse. it somewhat balanced with my initial Java application success in the last post. Now I'm frustrated as I can't run any of the damn acceptance tests!
It's an eclipse environment problem. Eclipse is complaining it can't communicate with the database. I'm also not getting the JUnit option when running the classes i've made... grr.
I got pretty pissed off with Eclipse. it somewhat balanced with my initial Java application success in the last post. Now I'm frustrated as I can't run any of the damn acceptance tests!
It's an eclipse environment problem. Eclipse is complaining it can't communicate with the database. I'm also not getting the JUnit option when running the classes i've made... grr.
Subscribe to:
Posts (Atom)