Jenkins and Cucumber Reports

I've provided details on this set up, but here's a visual walk through:

My Web Automation Jenkins looks like this the image below.  Each job listed (Announcements, Call Forwarding, etc.) is a specific feature file that Cucumber will run - so each entry runs a multitude of tests.  Each Test is sent to a Selenium Grid hub and is farmed to run on one of several VM's.  This simultaneous testing brings down the overall end to end duration significantly (from 45min to 6min) :

When you mouse over a job that has run, it will offer the reports view in the menu, like so:

Clicking on the Cucumber Reports link will first present you with a summary of results:

If you click the test link in the Feature Statistics table, it will break down more details on what failed and passed:

Cucumber Conditionals

I had an interesting experience with a GXT environment.  Since GXT generates the HTML code, I had some constraints to work around.

In this particular case, I had drop downs that were locked down to these specifications:
  - drop downs were not treated with the "select" tag
  - drop downs were read only

this meant, if I had a language drop down with values: English, Spanish, French

I had to find a way to select any of them, via a test.  Since the code in this case couldn't be modified to switch the drop downs to "selectors" I was stuck. Watir Webdriver expects drop downs to be accessed with the .select method.  In this case I had to use the .text_field method.

Since the fields were read only, I couldn't pass in values to select them.

My Solution
I ended up writing a conditional test in Cucumber itself.

The test would be a scenario outline defining the values in the drop down for the test, like this:

Scenario Outline: Testing each value in the language drop down
  Given a user at .....
  When they choose a specific language of <language>
  And Save .....
  Then the <language> will be set
  Examples:
  |language|
  |English|
  |Spanish|
  |French|

In the step definition, I wrote a conditional at the When and Thens.... the When would be like:
if arg == "English
   @browser.text_field(:id=>"the id on this field").send_keys :arrow_down, :return

elsif arg == "Spanish"
   @browser.text_field(:id=>"the id on this field").send_keys 2.times{:arrow_down)
   @browser.send_keys :return

and so forth.

For many choices I would use a case/when switch instead of a series of if's, but you get the idea.

Is it the best idea? No.

But it was optimal, as some would say. It got a solution out quickly, when I had a constraint that forced me into a specific path.

Cucumber Using Multiple Test Environments

I came across a need to run my tests across multiple environments.  For example:
Integration Environment, vs QA 1 vs QA 2.  Different versions or branches of the web app could be stood up in either location.

The method I used to achieve this was with the use of hooks.  But I hit a little snag, and found out my error.

What I did was this:

In my /support/env.rb file (within the Cucumber folder), I have this:
def browser_name
  (ENV['BROWSER'] ||= 'firefox').downcase.to_sym
end

def environment
  (ENV['ENVI'] ||= 'int').downcase.to_sym
end

Before do  |scenario|
  p "Starting #{scenario}"
  if environment == :int
    @browser = Watir::Browser.new browser_name
    @browser.goto "http://integration.atmysite.com:8080"
    @browser.send_keys :return
  elsif environment == :qa1
    @browser = Watir::Browser.new browser_name
    @browser.goto "http://qa1.atmysite.com:8080"
    @browser.send_keys :return
  else
    @browser = Watir::Browser.new browser_name
    @browser.goto "http://qa2.atmysite.com:8080"
    @browser.send_keys :return
  end
end

The first method defines the browser from the command line, this is useful for use with Jenkins or other batch file runs, that I can set the browser to be used in the test via the command line.  I learned this from an online resource someplace.

The second method there is similar - I based this on the previous. I'm saying take the value ENVI and it's value... then in the Before block I say:
If that environment method is :int, then run this login block... if it's :qa1, run this login block... else go to this other location.

I'll prob enhance that to where's instead of if statements at some point.

Where I initially errored on this, I was setting the equality to "int" rather then "int"... and I couldn't figure out why it wasn't working.  So I debugged it, by putting in a puts environment during the test run.  It was outputting :int.  So the parameter value was getting set as a symbol (i.e. :something.)

Once I updated the if statements to be :int, :qa1, :qa2 everything worked great.

Now I dont need the login helper I was using.  This before hook takes care of the login, based on the environment value I pass.

Automating SIP Testing with SIPp, FreeSwitch and Cucumber

SIP testing is pretty new to me.  In fact, 3 months ago, I didn't know what SIP was and I knew little of VOIP.  Today, I work at a company who's business is phone routing, virtual PBX, fax routing, VOIP, SIP, etc.

While I was able to set up the Front End web app automation, the development leads wanted me to also start building out a framework to test the SIP calls and the PBX software (FreeSwitch.)

Problem:

Create an automated framework that can be used to load test and functional test FreeSwitch.

Analysis:

I looked at a lot of opensource solutions, I was hoping for FS_Spec and some other ruby based solutions to work. But I had nothing but issues with them. In fact most software in this realm hasn't been updated since '09 or so.  In my research I did find one Open Source tool that was highly rated all around:
SIPp.

SIPp even comes pre-installed on the security distro of Linux (BackTrack.)  SIPp is pretty powerful, but it has a few drawbacks:

• It's not geared to functional tests. SIPp is really a load engine.  
• The reporting isn't great.  
• Can't run a suite of tests out of the box
• Doesn't have built in features to test simultaneous UAC and UAS (sender/receiver) - it expects a person to have multiple terminal sessions running, and one running a UAC and the other a UAS. 

 The above issues, broke down to too much human intervention.  What was desired, was something similar to the front end automation: you run a functional test, it either passes or fails, and the results are gathered for the test suite.

Solution:

I started with SIPp as the solution for doing the bulk of the tests, and then worked to build a framework around it to run the tests in an automated fashion.

What I came up with was Cucumber running SIPp functional and Load scripts.
The end result is, I just run:
cucumber
It runs all the tests (i.e. a few seconds later, it's driving traffic to my desk phone... Desk phone is ringing... yay.) Test fails/passes are tallied and the results are output to screen and file.

In the end, Cucumber is optional.  It's not a requirement here. For me, I'm only using Cucumber to tally the results.  I may opt to drop it all together as it has a high overhead.  The tests would be faster if just a collection of Ruby/Groovy or Scala scripts.

How it All Works:

To start it off, I had to see how the development servers work (i.e. FreeSwitch.)  So first things first, if you're going to test SIP, I recommend installing the Virtual PBX you are using in development (Asterix, FreeSwitch, etc.) on a local QA box.

Other tools you'll want installed on the same box are:
Wireshark
SIPp
SoftPhone for testing (I.e. X-Lite or "telephone")
Scripting Language (Ruby/Groovy/Python/Scala Scripts)
Optionally: Cucumber 
Optionally: Fully Functional Call Control with virtual numbers

I recommend only using OSX or a Linux distro for this set up.  Getting this to work on a PC is too difficult. 

You'll want to spend time with FreeSwitch and WireShark to see  how the packets are sent back and forth... and how to read the exchange.  Then you'll see how to write the tests.

Writing the SIP tests all boils down to two steps:

• The main test is going to be written in SIPp.  It's a command line SIP load tool.  But you can constrain it to only drive one test, for functional testing.  
• Use a Scripting Language (or Cucumber) to run the SIPp tests you wrote.

I detail each step below...

Details: Setting Up FreeSwitch

The easiest way to do this is on OSX or Linux.  Just do a git clone from the FreeSwitch repo... you can follow the instructions here: http://wiki.freeswitch.org/wiki/Installation_and_Tips_for_Mac_OS_X
Make sure you also do the step to make your audio files.

On Mac you'll prob need to bring in the libjpeg on my mac... to do this i used brew install libjpeg.

Configuration Setup:

This is where it gets a little tricky with FreeSwitch set up.

1. You'll need to know or change the default password... this is found in the vars.xml file  in /conf inside of the Freeswitch directory.  The instructions to do this are actually in the comments of the vars.xml file. 
2. Next, you'll want to change  /conf/sip_profiles/sip_profiles/internal.xml to update the inbounx-acl is localnet.auto 
3. As a tester, you're probably going to be  using an Internal Profile (meaning your machine's internal IP) for testing.  Developers will probably not use that set up. For testing though, you probably will.  When pointing to your Internal/local IP, it will bind that to a specific profile called "internal."  By default that folder is empty.  What I did, to make it quick and easy was this step:  I copied the files in the Default profile to the Internal Profile:  /conf/directory/default/* to /conf/sip_profiles/internal/* 
4. What you just copied over are actual user accounts and extensions.  They need to be modified.... Edit them individually and change the reference of "default" to "internal" and save them out.

Start Up FreeSwitch

Ok lets start it up, go to the freeswitch bin directory (i.e. /usr/local/freeswitch/bin) and type ./freeswitch

Once it's up, type: sofia status
If everything worked out, you should see 4 rows in a table display.  One will be an Alias, and it references the Internal Profile.  This Alias should have your local IP.   

You can tab for methods/commands... like this, type sofia, then hit TAB.  you'll see all the available options.  For example: sofia global siptrace on  is a very useful command.

Register with your local FreeSwitch

You can now launch a softphone (like X-Lite) and set it up to talk to login as a default user you copied over (i.e. 1000) and use the default password (what's in the vars.xml file) and then for domain set your local IP.

If everything is set up correctly, it should REGISTER on the FreeSwitch.

If you have a second softphone (i.e. "telephone"), you can also register as a different user (i.e. 1001) and then call the other user (i.e. 1000).

Once you make a call you should start seeing the FreeSwitch server display a lot of activity.  If you  have: sofia global siptrace on running, it will categorize the events.  You should see events like: INVITE, ACK, BYE.

Details: Using Wireshark

Wireshark Installation Overview

Wireshark is incredibly useful here.  If you are installing Wireshark on a MAC, you'll need to set up X11... on Mountain Lion, you'll need to use XQuartz.  Once installed, you can then bind Wireshark to XQuartz/X11 - I wont go into all the setup details here, but you'll need to restart your Mac for the changes to take effect (or logout/login.)

When Wireshark is up and running, you'll want to listen to the interface: lo0 for  your local traffic.

What can Wireshark show me?

What can't it show you.... it shows everything.  It not only records all the packets, this tool also has a whole Telephony menu.  From there you can build VOIP diagrams of the call flows. This is very useful in seeing what is sent and expected back for each test.  Wireshark can also copy the RTP stream!  That means it can hear the audio  you send, and verify the audio sent (say a recorded wav) is what's received!  

Using Wireshark to Generate a Flow Diagram

Go ahead and launch XQuartz, then Wireshark. 

Now, click on the interface lo0 and start recording.

Make sure FreeSwitch is up and running and that it is taking local traffic (i.e. sofia status shows alias and internal profiles up and running)

Make a call from one softphone to the other

Stop the Wireshark recording.

At first you'll see a dump of all packets captured on that interface.  

Now, go to the Telephony menu, and click "VOIP."  That will load only the VOIP related packets.  

Click the button, Select All.  

Now click Make Flow Diagram.

From that window, save it out.  In OSX the initial display usually doesn't render well. but once you save it out, it looks great.

This flow diagram will show you everything going on in a call, and removes all the uneeded data, making this more human readable. 

Using Wireshark to listen/verify the audio of a Call

Go ahead and launch XQuartz, then Wireshark. 

Now, click on the interface lo0 and start recording.

Make sure FreeSwitch is up and running and that it is taking local traffic (i.e. sofia status shows alias and internal profiles up and running)

Make a call from one softphone to the other

Stop the Wireshark recording.

Similar to before, click the Telephony menu.

Click the RTP sub menu and the "Show Streams" option.

You'll see packets here on each line item. click through on one, and click "Analyze."

You'll get the audio of the call/transmission. 

Using Wireshark Programmatically

My goal is to use it in automation testing.  To do this, I'll be calling the Wireshark utilities from the command line and getting back results in the command line interface.  

Details: SIPp

Now lets look at SIPp.  If you're new to SIP, you can think of SIPp as a command line version of JMeter.  It was designed to drive lots of traffic (hundreds or thousands of calls a min) to a PBX.

However, SIPp can be set up to run just one call through a functional scenario.

To install SIPp, just follow the instructions over at: http://sipp.sourceforge.net/

Pre-Installed Tests

SIPp has built in tests.  These include UAC, UAS, UAC_PCAP, REGEX, and more.  The whole list is off their main site.  If you don't know what UAC or UAS is.... you should probably read up on SIP.  But to just basically summarize it, SIP is a P2P system.  At any given time, one user is both the server and client.  They are sending data and receiving.  so UAC and UAS is a client server scenario.  

If you just want to see SIPp work, you can type:

./sipp uac [your local ip]

There's no Params, so it will run it's default load for the UAC test. 

If you want to see what the built in UAC test is actually doing, just do this:

./sipp -sd uac >> [filename]

Working with Parameters and SIPp

There's some sites out there with useful cheat sheets for SIPp... here's one such site:

http://tomeko.net/other/sipp/sipp_cheatsheet.php?lang=pl

That site also has some tests they wrote, which you can download and see how they work.  They also have tests that utilize CSV files which contain a set of users you want in your load.

For my tests, I use a lot of functional tests, so I don't want to slam 10,000 phone calls against the server. Instead I want to run one call, one time, to do something specific:

./sipp -s [phone number configured on call control to redirect to my desk phone] -m 1 -l 1 -recv_timeout 6000 -sf [path to my test xml file] [IP of our integration / uat freeswitch]

So here's what those parameters are saying:

Run SIPP against the QA Integration/UAT env. and dial that number configured in Call Control (-s means service, which could be a extension, phone number, etc.) -m tells it to run once -l tells it to run only one test at a time.  -recv_timeout 6000 says to run this test for only 6 seconds.  Finally -sf is the path to my test.... and of course i end it with the IP of the FS box.

Writing your Own SIPp tests

I wont go into too much detail here.  I'll save it for a future blog.  But to start with, you can take an existing built in test, and export it using the -sd option I mention above, or you can review some tests other people are writing out there and have put on github or elsewhere.

Automating All This

Now to automate this.

Initially I used straight Ruby for this... but then I decided having Cucumber run the tests, gives me the added benefit of compiling test results. I'm on the fence if that's necessary at this point or not.

Basically, I found it too hard to use an existing API with what we use here in house as our PBX.  I tried SIPr, FS_Switch and a variety of other tools. Most of these haven't been touched in many years and have known bugs flagged against them.  Rather then fight with that, I choose to use SIPp.

SIPp is an industry standard, but it doesn't have an open API.  To automate this, I use SIPp as is, and build the shell around it.

It's really very simple:

Create a script to drive all the SIPp tests.  

Assert via REGEX the output for expectations.

Mark pass / fail.

So I'm basically doing the core work in SIPp.  I write out my test, with the parameters to call it effectively.

Then I use Cucumber (you could use straight Ruby or Groovy or Scala, Python, etc.) to do this:

Given /^a SIP call is made to another SIP account$/ do

   @calling = %x{/Users/brainwarner/sipp-3.3/sipp -s 1000 -m 1 -l 1 -recv_timeout 6000 -sf /Users/brianwarner/sipp-3.3/calling.xml 10.98.1.1}

end

Then /^a SIP 180 ringing should return$/ do

   assert @calling.include?("180")

end

The above is a simple set of two steps in a step definition file.  Basically it's saying ok, I'm going to execute sipp with my sipp test, against the specific IP of my UAT server and I'm using those parameters to make sure it's just one test, run once and ends after 6 seconds.

Then I'm calling a Ruby method "include" which is like a REGEX, to search the output that's in that class variable @calling for the specific ID I expect for calling.  

Why not use REGEX? well every regex I tried, passed even when it failed. Initially i would search the output for a invalid value (i.e. 777iwin!), and it would always pass, no matter how I wrote the regex!  I ended up using this method because it actually works well.

Simple. But it works.

Future Steps In Automation

So this is the Framework.  

You could easily see how that simple test could be transformed to something more impressive:

%x{....} calling one UAS user session

%x{...} calling another session in UAC test mode... and then have them talk to each other.

Since we're calling 3rd party command line utilities, you can also run the command line wireshark! meaning at test time you could use Wireshark to sniff packets, or verify RTP audio.... now that's cool.

Getting Safari Webdriver To Work

This is going to take a good part of your day... so be prepared if you want to do this.

Where I work, I set up a Cucumber automation framework that basically runs: IE9, FF, Chrome and I was just given a Mac Mini to support Safari testing.

The ultimate goal is to have: IE8, IE9, FF, Chrome and Safari - running in parallel.

FF, Chrome and IE are pretty easy to set up.  HOWEVER, Safari, is a big pain.

First, you need to do everything here:
http://watirmelon.com/2012/04/17/using-watir-webdriver-with-safari-at-last/

That will pretty much take half your day.  If you have a fresh Mac, you'll need xCode and you'll need to get the command line package for xCode so you can use svn.  What Alister Scott on that link will tell you to do can be summed up as:

• Make a Apple Safari Dev account (free)
• Get a certificate
• Download the certificate
• Open Safari and install the certificate
• One thing not mentioned in the instructions, is that you can't install the cerficiate till Safari knows you're a developer, to do that you have to open the key chain on the mac and click system and drag/drop your certificate and they you must right-click the certificate, and set all actions as "TRUST" and finally you have to accept these changes.
• Then you build the extension (downloading via svn the code, and building the extension.

That's where Alister's tutorial ends... but you'll no doubt get a problem.  If you are like most, you'll get Safari hitting 127.0.0.1/wd for each test... not hitting your target site, and you'll get an error referencing "EOFError: end of file reached" One of the comments on Alister's post has the solution...

You now need to do a gem install selenium-webdriver (i couldn't do the -pre that he suggested, so i just did gem install selenium-webdriver) and I added this to my env.rb file as gem 'selenium webdriver'

After that, just rebuild: bundle update and you should be good.

Getting Cucumber working in the Amazon Cloud with Jenkins

This took a bit of work, and a variety of online resources.

First I used this tutorial to set up AWS Amazon cloud services for free, hooking it up to github, getting rvm/ruby installed, etc.:
http://watirmelon.com/2011/08/29/running-your-watir-webdriver-tests-in-the-cloud-for-free/

Second, I had to do a git clone of my repo to the AWS box and do a bundle install so that all dependancies were loaded

Third, I hit an issue with cucumber-rails... I used this resource to resolve it:
http://datacodescotch.blogspot.com/2011/11/warning-cucumber-rails-required-outside.html

Fourth, I had an issue with no JS executor on the ubuntu cloud box.. so I installed node.js to get past that:
https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager

Finally, the tests ran, but got webdriver errors... needed to install webdriver... :)

More detail on JSON validation in Cucumber

Here's more details on my last post about doing a JSON smoke test.  The idea here, is that a test is needed to check the schema of the JSON coming back - to verify that the JSON structure itself is intact with the correct categories and sub categories.

I used Rotten Tomato's API end point.  I'll pass in a fixed parameter to a specific movie and then validate the JSON coming back.

The end point URI that i'm hitting is:
http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=Toy+Story+3&page_limit=1

Let's review the JSON response we get with the above query - jsonlint.com shows that the top level categories are:

• total
• movies
• links
• link_template

Some of these categories have sub categories, and some do not.  For example, Total, does not have a sub category in the JSON, but movies has a bulk of sub categories.  In fact movies has:

• id
• title
• year
• mpaa_rating
• runtime
• critics_consensus
• release_dates
• ratings
• synopsis
• posters
• abridged_cast
• alternate_ids
• links

some of these of sub categories themselves.  But to just cover these categories and sub-categories here's how I wrote the code in the step definition file:
require 'open-uri'
require 'json'

Given /^A call to the Rotten Tomattoes API$/ do
  @mquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=Toy+Story+3&page_limit=1").read)
end
Then /^the response should have  the (.*) expected$/ do |category|
  @mquery["#{category}"].should be_true
end

Given /^A call is made to the Rotten Tomattoes API$/ do
  @mquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=Toy+Story+3&page_limit=1").read)
end
Then /^the response returned should have the movies (.*) expected$/ do |subcategory|
  @mquery['movies'][0]["#{subcategory}"].should be_true
end
Then /^the response returned should have the (.*) subcategory expected$/ do |release_dates_sub|
  @mquery['movies'][0]['release_dates']["#{release_dates_sub}"].should be_true
end
Then /^the ratings response returned should have the (.*) subcategory expected$/ do |ratings_sub|
  @mquery['movies'][0]['ratings']["#{ratings_sub}"].should be_true
end
Then /^the posters response returned should have the (.*) subcategory expected$/ do |posters_sub|
  @mquery['movies'][0]['posters']["#{posters_sub}"].should be_true
end
Then /^the abridged cast response returned should have the (.*) subcategory expected$/ do |abridged_cast_sub|
  @mquery['movies'][0]['abridged_cast'][0]["#{abridged_cast_sub}"].should be_true
end
Then /^the alternate id's response returned should have the (.*) subcategory expected$/ do |alternate_ids_sub|
  @mquery['movies'][0]['alternate_ids']["#{alternate_ids_sub}"].should be_true
end
Then /^the links response returned should have the (.*) subcategory expected$/ do |links_sub|
  @mquery['movies'][0]['links']["#{links_sub}"].should be_true
end


The Cucumber test I wrote uses a Cucumber Scenario Outline to pass in the expected categories or sub-categories into the code above.  Here's how I wrote the tests:

Feature: Smoke tests to ensure json validity by checking and verifying each JSON category and sub category exists
  #Note: this is not validating values, just that the JSON structure has the expected categories and sub categories

   Scenario Outline: Smoke test the JSON top level categories returned
     Given A call to the Rotten Tomattoes API
     Then the response should have  the <category> expected

     Examples:
     |category|
     |movies  |
     |links|
     |link_template|

  Scenario Outline: Smoke test the JSON movies sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the response returned should have the movies <subcategory> expected

  Examples:
    |subcategory|
    |id  |
    |title|
    |year|
    |mpaa_rating|
    |runtime    |
    |critics_consensus|
    |release_dates    |
    |ratings          |
    |synopsis         |
    |posters          |
    |abridged_cast    |
    |alternate_ids    |
    |links            |

  Scenario Outline: Smoke test the JSON release dates sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the response returned should have the <release dates_sub> subcategory expected

  Examples:
    |release dates_sub|
    |theater  |
    |dvd|

  Scenario Outline: Smoke test the JSON ratings sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the ratings response returned should have the <ratings_sub> subcategory expected

  Examples:
    |ratings_sub|
    |critics_rating  |
    |critics_score|
    |audience_rating|
    |audience_score |

  Scenario Outline: Smoke test the JSON posters sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the posters response returned should have the <posters_sub> subcategory expected

  Examples:
    |posters_sub|
    |thumbnail  |
    |profile|
    |detailed|
    |original |

  Scenario Outline: Smoke test the JSON abridged cast sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the abridged cast response returned should have the <abridged_cast_sub> subcategory expected

  Examples:
    |abridged_cast_sub|
    |name  |
    |id|
    |characters|

  Scenario Outline: Smoke test the JSON alternate ids sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the alternate id's response returned should have the <alternate_ids_sub> subcategory expected

  Examples:
    |alternate_ids_sub|
    |imdb  |

  Scenario Outline: Smoke test the JSON links sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the links response returned should have the <links_sub> subcategory expected

  Examples:
    |links_sub|
    |self  |
    |alternate|
    |cast     |
    |clips    |
    |reviews  |
    |similar  |

Testing API's with Cucumber

I put together several API tests.  Here's the project on Github.com:
https://github.com/wbwarnerb/movielookup

The project has three Cucumber features (under the features directory) that talk with a public API.

For the purpose of this test I'm using Rotten Tomatoes public API.  They have a nice JSON response to parse and utilize for these examples.

Example 1: /features/movie_json_smoke.feature

The first feature is a smoke test it basically validates the json itself.  The idea here is: We know what we expect to return in the JSON, we verify that the categories and sub categories are intact and no changes have occurred. This is useful, if a change to the service was made, but the overall structure of the JSON should remain the same.  Just kick off this test, and it validates the JSON structure.

Here's how I did it in cucumber:
https://github.com/wbwarnerb/movielookup/blob/master/features/movie_json_smoke.feature
from here we have the Cucumber test itself.  I build the test out as a Scenario Outline, this allows me to utilize the data tables.  When I get to the "Then", I pass in a parameter from the table, like this:

Then the response should have  the <category> expected

The category parameter matches the data table, which has a column header "category" - every element under that column in the table is passed into the Then statement.

The actual code is in the step definition:

https://github.com/wbwarnerb/movielookup/blob/master/features/step_definitions/movie_json_smoke_test.rb

it's here that I grab the parameters coming form the table / test and I pass them into the relevant spots.  

Here's an example:

Then /^the response should have the (.*) expected$/ do |category|

  @mquery["#{category}"].should be_true

end

First, notice the (.*), that is the placeholder for the variable in the test <category>. Second, the argument I'm grabbing |category| is coming from the test. It takes the Category value (from the data table in the test) and passes it through here.

The next line of code reads @mquery (that's the instance variable) ["#{category}"].should be_true

So this is the way ruby picks up the JSON objects ['category']['subcategory'] But since we're passing in a dynamic parameter here, we need double quotes. so ["#{Category}"] (the #{category} will be replaced with the value in the data table.

The .should be_true tripped me up. I kept trying == !nil and .should == !nil, nothing worked until I found it should be done like this .should be_true. This validates that the JSON element the method is attached to is present.

 At this point, I just iterate through each parameter in their JSON returned.  I go through each category and subcategory and make sure that it's all there.  

I figure that's a good smoke test.  If I worked at RT, and some dev committed some code to their service/api, I could just run this and quickly see that json categories are still present.  

Example 2: movielookup_data_validation.feature

https://github.com/wbwarnerb/movielookup/blob/master/features/movielookup_data_validation.feature

In this test, I'm treating a data table in the test as the source of truth.  This could be a database, but I'm using a data table with Cucumber's Scenario Outline.  This time, I've added several columns.  I'm verifying the mpaa_rating and the profile image stored at Rotten Tomatoes.   

The code for verifying this is similar in concept to what I did previously:

https://github.com/wbwarnerb/movielookup/blob/master/features/step_definitions/movielookup_data_validation_test.rb

I go through several movies... passed through, like this:

Given /^the API is queried for (.*)$/ do |title|

  @mquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=#{title}&page_limit=1").read)

end

The argument is passed through and is passed through to the API URI and grabs that movie's details which are then verified

Notice that the main difference then before is this code:

 @mquery['movies'][0]['mpaa_rating'].should == "#{mpaa_rating}"

The main difference, is that I'm passing the argument through to use as a validation check.  

Example 3: /features/movielookup_json_validation.feature

https://github.com/wbwarnerb/movielookup/blob/master/features/movielookup_json_validation.feature

These tests I separated out of the other two.  These tests are made to just test the JSON functionality itself.

Given /^A request for an unknown movie is sent to the API$/ do

  @bquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=bbxa&page_limit=1").read)

end



Then /^the response should be zero movies returned$/ do

  @bquery['total'].should == 0

end



Given /^A API call is made for more then one movie$/ do

  @mquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=Toy+Story&page_limit=1").read)

end



Then /^the results wil have a total greater than 1$/ do

  @mquery['total'].should > 1



end



Basically in these two simple tests I'm checking that if a query is sent for a movie not in the RT movie database, that we get back a proper response. 

The second test verifies that if a query is sent for a movie with more then one hit (like Toy Story) it will display a total value greater the 1.

Rotten Tomatoes

Video of Cucumber Test Running

Cucumber set up and sample test

Compared to the time I spent getting GEB up and running and getting tests working... Cucumber is so much easier.

Here's how I set up Cucumber
First, I edited the Gemfile for a project and added this group:

group :test, :development do
  gem 'cucumber-rails'
  gem 'database_cleaner'
  gem 'rspec'
  gem 'spork'
  gem 'watir-webdriver'
end

Once done, I run at the project root:

bundle install

It pulls down the gems needed - in this case, I'm mainly using cucumber-rails and watir-webdriver.

After the install, I run a rails generator call to set up cucumber from the project root:

rails generate cucumber:install

This will create a features folder... it's layout will be like this:

features

  |

  |--step_definitions

  |--support

At the root of the features folder, I put my BDD test.  This is where you define the goals... you can think of it as the high level "Given When Thens."  What's cool about Cucumber, is that the NO code goes in this file, it's simple.  In GEB, or groovy frameworks, simple code goes in the BDD tests.  But in Cucumber, NOTHING but the BDD goes there. 

The BDD Test

In my test, I wrote the following in a ruby file at the root here:

Feature: User searches for book on Amazon.com
  Scenario: User searches for the book Cucumber
    Given a user goes to Amazon.com
    When they input the title Cucumber in the Amazon search field
    And they click submit
    Then  they should see a resuls list with Cucumber in the title

So the directory looks like:

features

  |

  |--step_definitions

  |--support

  amazon_test.rb

If you were to run the command "cucumber" now, it would return back a statement that there are no step definitions.  Which is true, there aren't.  The step definitions is the code that makes this test work.  Right now, it just has the BDD language, but no code.

The Step Definitions
So lets create the amazon_test.rb ruby file... in there I use Watir-webdriver to power the browser... and I use Cucumber to write the Given When Thens'.  Here's my file's contents:
Given /^a user goes to Amazon.com$/ do
  @browser = Watir::Browser.new(:ff)
  @browser.goto "http://www.amazon.com"
  @browser.title
end

When /^they input the title Cucumber in the Amazon search field$/ do
  @browser.text_field(:name => "field-keywords").set "Cucumber"
end

And /^they click submit$/ do
  @browser.button(:class => "nav-submit-input").click
end

Then /^they should see a resuls list with Cucumber in the title$/ do
  @browser.h1(:id => "breadCrumb") == "Cucumber"
  @browser.close
end

Now if we run "cucumber" - it will execute each scenario in the test and each part of the test.  If it fails to get a value for any of the steps, it will return the error in the console.  If you have a ANSI color coded terminal you'll get results in a nice color coding.

Breaking the Code Down
First line defines the browser I'm using. In this case it's Firefox.

After that I tell it where to go, in this case www.amazon.com

Next I'm finding the main search field on Amazon ("field-keywords") - I'm searching for the name on the field.  It would be better to search by ID, but I'm going for a field name search.  In the same line, I'm passing the value of "cucumber" into the field.

Next I find the submit button and click it.

Then I find the breadcrumb on the page, and verify the value is "Cucumber"

Finally I close the browser to end the test.