Automation test frameworks should be able to meet the needs of a QA team.
After working with GEB for over a year now (almost two years), I've come to the opinion, that it's not a very good solution for QA. GEB is a very brittle and the Groovy language has some barriers to entry for people who are not professional developers (i.e. QA.)
There is some talk in some companies that any QA person should be technical enough to fill in and code for any developer... but it's an bizarre concept - that leads to QA automation strategies, such as GEB. If a QA person could fill the shoes of a developer, they'd switch titles and make 20% more money. This Free Market Economy establishes our roles and responsibilities. I've even heard some places where Project Managers are expected to be able to code and fill in for a developer. If they really could be the same or even similar quality, why wouldn't they just switch titles and make more money?
Initially I thought it was a great idea (bringing GEB into the field of view.) I thought it made sense. It utilized the skills of the entire development team (since they already knew Java) and gave us momentum. But as you'll see, my view of the fruit of this, almost two years later, isn't very positive.
What is GEB?
GEB is a automation framework built on three things:
Groovy
Spock
Webdriver
Brittle
I mentioned GEB is brittle. This is of course my opinion. Some of my friends and coworkers disagree on this point. But I've seen nothing great from this framework. Our tests are constantly broken. Granted one could say "well it's your tests!" Possibly, but there's so much problems with GEB. For starters GEB doesn't have a strong community. It gets few updates. The framework is so outdated, where QA is concerned. While Firefox releases a browser every couple weeks, GEB slips further and further behind.
GEB has old requirements. It needs specific old versions of Groovy (1.7 to be exact.) Groovy has already jumped two versions now (from 1.7, to 1.8 and now 2.0) But GEB won't work with any Groovy compiler above 1.7. The same is true with Spock. GEB only works with Spock 0.67.
This means, if you grab a IDE like STS, you need to go back about 15 versions to 2.7.1. Anything above that preinstalls Groovy 1.8 or higher. It also means you have to watch any auto updates.
What happens when Firefox comes up with a new version? you need to run through a gamut of trial and error to upgrade a simple selenium webdriver in the POM and then find out it's not compatible with the brittle framework. You wait for a solution. Get it, install it and Firefox is up another version!
What about webdriver... GEB doesn't like IE much. It can't seem to close the sessions. So we can't run this via Jenkins - as the IE sessions stay open and kill the server. GEB uses old versions of selenium webdrivers... it has to use versions like 2.23 for IE... and the current version as of writing this is 2.4!
You could spend your time 100% grossed in the framework, or get a framework that just works and spend your time writing tests. I prefer the latter.
Barrier To Entry
GEB uses the Groovy language. Groovy is very easy to get and understand, but all the resources for Groovy are written for experienced developers. I like to think of Groovy as shorthand for Java dev's. The problem here is, most QA teams are not made up of professional and experienced Java developers. If they were, they would switch titles in a heartbeat and make more money. Without a strong community pushing Groovy to those with little programming experience, Groovy becomes difficult to learn.
It's much like wanting to learn how to write Chinese and someone tells you there's a shorthand for Chinese, it's really simple... but unfortunately you have to learn Chinese first, in order to understand the shorthand!
GEB Setup
I work at a shop that first built it's own Groovy/Spock framework. It took months. Then we found GEB. 1 Architect and 2 dev's took 3 months to get the thing up and running.
I attempted the same thing at home. Despite all the git repo's with "geb sample project" - none of them worked for me. I couldn't get the damn thing working for months. I think I finally got it up and running in about 3 months.
Installing Ruby and Watir, and writing a test took me 10min.
What Other Options?
Where GEB can't seem to close a IE browser it opens in a test, Watir and Cucumber, just work. They pull up every browser I have. They can use the latest webdrivers... Where GEB needs old archaic versions of selenium webdrivers.
I wont begin to pretend I'm an expert in a variety of frameworks. But I will say that I installed Rails, and did a gem install watir and was able to write a test within 10min. This is impossible with GEB and it's brittleness - which took me months to get working and years of frustration of broken tests and environments.
After working with GEB for over a year now (almost two years), I've come to the opinion, that it's not a very good solution for QA. GEB is a very brittle and the Groovy language has some barriers to entry for people who are not professional developers (i.e. QA.)
There is some talk in some companies that any QA person should be technical enough to fill in and code for any developer... but it's an bizarre concept - that leads to QA automation strategies, such as GEB. If a QA person could fill the shoes of a developer, they'd switch titles and make 20% more money. This Free Market Economy establishes our roles and responsibilities. I've even heard some places where Project Managers are expected to be able to code and fill in for a developer. If they really could be the same or even similar quality, why wouldn't they just switch titles and make more money?
Initially I thought it was a great idea (bringing GEB into the field of view.) I thought it made sense. It utilized the skills of the entire development team (since they already knew Java) and gave us momentum. But as you'll see, my view of the fruit of this, almost two years later, isn't very positive.
What is GEB?
GEB is a automation framework built on three things:
Groovy
Spock
Webdriver
Brittle
I mentioned GEB is brittle. This is of course my opinion. Some of my friends and coworkers disagree on this point. But I've seen nothing great from this framework. Our tests are constantly broken. Granted one could say "well it's your tests!" Possibly, but there's so much problems with GEB. For starters GEB doesn't have a strong community. It gets few updates. The framework is so outdated, where QA is concerned. While Firefox releases a browser every couple weeks, GEB slips further and further behind.
GEB has old requirements. It needs specific old versions of Groovy (1.7 to be exact.) Groovy has already jumped two versions now (from 1.7, to 1.8 and now 2.0) But GEB won't work with any Groovy compiler above 1.7. The same is true with Spock. GEB only works with Spock 0.67.
This means, if you grab a IDE like STS, you need to go back about 15 versions to 2.7.1. Anything above that preinstalls Groovy 1.8 or higher. It also means you have to watch any auto updates.
What happens when Firefox comes up with a new version? you need to run through a gamut of trial and error to upgrade a simple selenium webdriver in the POM and then find out it's not compatible with the brittle framework. You wait for a solution. Get it, install it and Firefox is up another version!
What about webdriver... GEB doesn't like IE much. It can't seem to close the sessions. So we can't run this via Jenkins - as the IE sessions stay open and kill the server. GEB uses old versions of selenium webdrivers... it has to use versions like 2.23 for IE... and the current version as of writing this is 2.4!
You could spend your time 100% grossed in the framework, or get a framework that just works and spend your time writing tests. I prefer the latter.
Barrier To Entry
GEB uses the Groovy language. Groovy is very easy to get and understand, but all the resources for Groovy are written for experienced developers. I like to think of Groovy as shorthand for Java dev's. The problem here is, most QA teams are not made up of professional and experienced Java developers. If they were, they would switch titles in a heartbeat and make more money. Without a strong community pushing Groovy to those with little programming experience, Groovy becomes difficult to learn.
It's much like wanting to learn how to write Chinese and someone tells you there's a shorthand for Chinese, it's really simple... but unfortunately you have to learn Chinese first, in order to understand the shorthand!
GEB Setup
I work at a shop that first built it's own Groovy/Spock framework. It took months. Then we found GEB. 1 Architect and 2 dev's took 3 months to get the thing up and running.
I attempted the same thing at home. Despite all the git repo's with "geb sample project" - none of them worked for me. I couldn't get the damn thing working for months. I think I finally got it up and running in about 3 months.
Installing Ruby and Watir, and writing a test took me 10min.
What Other Options?
Where GEB can't seem to close a IE browser it opens in a test, Watir and Cucumber, just work. They pull up every browser I have. They can use the latest webdrivers... Where GEB needs old archaic versions of selenium webdrivers.
I wont begin to pretend I'm an expert in a variety of frameworks. But I will say that I installed Rails, and did a gem install watir and was able to write a test within 10min. This is impossible with GEB and it's brittleness - which took me months to get working and years of frustration of broken tests and environments.
Now I know, why there is no point of me breaking my head in GEB!!!
ReplyDeleteBut since you are a experienced QA on GEB, let me take this opportunity to ask a few questions.
My concern is with LoadUI where, i would like the performance of a logged in webpage, with flash content.
now currently I need a script , preferably a geb or groovy, (since loadui has a geb runner) where the output of the script can be directed to any other component and a report can be generated. how can i carry this out? I frakly cant learn html,css,jquery,spock,java,groovy and what not....
Hi Nevin, sorry for the delay in responding. I'm not familiar with LoadUI. For my load testing I used JMeter. The last time I used JMeter was to treat it as separate from my web ui automation framework.
DeleteWhat I was doing at that time, was I had Cucumber set up with WATIR to run all these web based automation tests. I was checking load times, by setting up a simple calculation in the page from button click, to the next action (i.e next page) being loaded. If that's all you need (i.e. to check and report out the load times of each page under the web automation run) then it's not hard. you create a date object, then another one when the actions ould be complete and find the difference.
If however, you want to run a full blown load test (with multiple users being simulated across your site) then for that I treated it as a separate test tool component.
I wrote the scripts in JMeter, and had the JMEter action output a report file... I used Jenkins and a plugin in Jenkins to pull in the JMeter report. So basically from the Jenkins UI, I clicked "build" for the load test job... which kicked of JMeter... when the job finished there was a report generated that was hooked into the Jenkins UI.