Cucumber and ambiguous results

I had some solid tests that kept error out on me.  It stumped me for quite awhile. I was busy looking at the logic, and didn't notice the error was really referencing the Then statement and saying it was "ambiguous."  Since I had several similar tests, I realized that the error was really about the Then statement in my Cucumber test.

All I had to do was change the language of the Then statement to be more unique and the tests began to pass.

Rails application - Palindrome Validation

I wrote a ruby/rails application to validate a palindrome.  Here's the published palindrome app on Heroku:
http://glacial-river-2425.herokuapp.com/

I came up with the code in my head on the way home from work.  I came across the method .reverse in rails.  This makes checking for a palindrome pretty easy in rails.

After I came home from work I sat down, and ran IRB then input this code:
x = "racecar"
if x == x.reverse
puts "this is a palindrome"
else "this is not a palindrome"
end

It worked.  So now I just needed to put this into a published Rails application. 

I created a Rails project
Generated a Controller and a some views

in the Index view I put this code on the page:
<%=form_tag(:action =>'results') do %>

<%=text_field_tag(:word)  %>
<%= submit_tag("Submit") %>
<% end %>

Then in the controller I put this code:
  def results
    @word = params[:word]
    if @word == @word.reverse
      @palindrome = "This is a palindrome"
    else
      @palindrome = "This is NOT a palindrome"
    end
  end

Finally on the results page:
<%= @palindrome %>

So basically that simple line of code, is used to verify that a word input into the form is a palindrome or not and the results are put to a results page.

More detail on JSON validation in Cucumber

Here's more details on my last post about doing a JSON smoke test.  The idea here, is that a test is needed to check the schema of the JSON coming back - to verify that the JSON structure itself is intact with the correct categories and sub categories.

I used Rotten Tomato's API end point.  I'll pass in a fixed parameter to a specific movie and then validate the JSON coming back.

The end point URI that i'm hitting is:
http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=Toy+Story+3&page_limit=1

Let's review the JSON response we get with the above query - jsonlint.com shows that the top level categories are:

• total
• movies
• links
• link_template

Some of these categories have sub categories, and some do not.  For example, Total, does not have a sub category in the JSON, but movies has a bulk of sub categories.  In fact movies has:

• id
• title
• year
• mpaa_rating
• runtime
• critics_consensus
• release_dates
• ratings
• synopsis
• posters
• abridged_cast
• alternate_ids
• links

some of these of sub categories themselves.  But to just cover these categories and sub-categories here's how I wrote the code in the step definition file:
require 'open-uri'
require 'json'

Given /^A call to the Rotten Tomattoes API$/ do
  @mquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=Toy+Story+3&page_limit=1").read)
end
Then /^the response should have  the (.*) expected$/ do |category|
  @mquery["#{category}"].should be_true
end

Given /^A call is made to the Rotten Tomattoes API$/ do
  @mquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=Toy+Story+3&page_limit=1").read)
end
Then /^the response returned should have the movies (.*) expected$/ do |subcategory|
  @mquery['movies'][0]["#{subcategory}"].should be_true
end
Then /^the response returned should have the (.*) subcategory expected$/ do |release_dates_sub|
  @mquery['movies'][0]['release_dates']["#{release_dates_sub}"].should be_true
end
Then /^the ratings response returned should have the (.*) subcategory expected$/ do |ratings_sub|
  @mquery['movies'][0]['ratings']["#{ratings_sub}"].should be_true
end
Then /^the posters response returned should have the (.*) subcategory expected$/ do |posters_sub|
  @mquery['movies'][0]['posters']["#{posters_sub}"].should be_true
end
Then /^the abridged cast response returned should have the (.*) subcategory expected$/ do |abridged_cast_sub|
  @mquery['movies'][0]['abridged_cast'][0]["#{abridged_cast_sub}"].should be_true
end
Then /^the alternate id's response returned should have the (.*) subcategory expected$/ do |alternate_ids_sub|
  @mquery['movies'][0]['alternate_ids']["#{alternate_ids_sub}"].should be_true
end
Then /^the links response returned should have the (.*) subcategory expected$/ do |links_sub|
  @mquery['movies'][0]['links']["#{links_sub}"].should be_true
end


The Cucumber test I wrote uses a Cucumber Scenario Outline to pass in the expected categories or sub-categories into the code above.  Here's how I wrote the tests:

Feature: Smoke tests to ensure json validity by checking and verifying each JSON category and sub category exists
  #Note: this is not validating values, just that the JSON structure has the expected categories and sub categories

   Scenario Outline: Smoke test the JSON top level categories returned
     Given A call to the Rotten Tomattoes API
     Then the response should have  the <category> expected

     Examples:
     |category|
     |movies  |
     |links|
     |link_template|

  Scenario Outline: Smoke test the JSON movies sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the response returned should have the movies <subcategory> expected

  Examples:
    |subcategory|
    |id  |
    |title|
    |year|
    |mpaa_rating|
    |runtime    |
    |critics_consensus|
    |release_dates    |
    |ratings          |
    |synopsis         |
    |posters          |
    |abridged_cast    |
    |alternate_ids    |
    |links            |

  Scenario Outline: Smoke test the JSON release dates sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the response returned should have the <release dates_sub> subcategory expected

  Examples:
    |release dates_sub|
    |theater  |
    |dvd|

  Scenario Outline: Smoke test the JSON ratings sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the ratings response returned should have the <ratings_sub> subcategory expected

  Examples:
    |ratings_sub|
    |critics_rating  |
    |critics_score|
    |audience_rating|
    |audience_score |

  Scenario Outline: Smoke test the JSON posters sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the posters response returned should have the <posters_sub> subcategory expected

  Examples:
    |posters_sub|
    |thumbnail  |
    |profile|
    |detailed|
    |original |

  Scenario Outline: Smoke test the JSON abridged cast sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the abridged cast response returned should have the <abridged_cast_sub> subcategory expected

  Examples:
    |abridged_cast_sub|
    |name  |
    |id|
    |characters|

  Scenario Outline: Smoke test the JSON alternate ids sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the alternate id's response returned should have the <alternate_ids_sub> subcategory expected

  Examples:
    |alternate_ids_sub|
    |imdb  |

  Scenario Outline: Smoke test the JSON links sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the links response returned should have the <links_sub> subcategory expected

  Examples:
    |links_sub|
    |self  |
    |alternate|
    |cast     |
    |clips    |
    |reviews  |
    |similar  |

Testing API's with Cucumber

I put together several API tests.  Here's the project on Github.com:
https://github.com/wbwarnerb/movielookup

The project has three Cucumber features (under the features directory) that talk with a public API.

For the purpose of this test I'm using Rotten Tomatoes public API.  They have a nice JSON response to parse and utilize for these examples.

Example 1: /features/movie_json_smoke.feature

The first feature is a smoke test it basically validates the json itself.  The idea here is: We know what we expect to return in the JSON, we verify that the categories and sub categories are intact and no changes have occurred. This is useful, if a change to the service was made, but the overall structure of the JSON should remain the same.  Just kick off this test, and it validates the JSON structure.

Here's how I did it in cucumber:
https://github.com/wbwarnerb/movielookup/blob/master/features/movie_json_smoke.feature
from here we have the Cucumber test itself.  I build the test out as a Scenario Outline, this allows me to utilize the data tables.  When I get to the "Then", I pass in a parameter from the table, like this:

Then the response should have  the <category> expected

The category parameter matches the data table, which has a column header "category" - every element under that column in the table is passed into the Then statement.

The actual code is in the step definition:

https://github.com/wbwarnerb/movielookup/blob/master/features/step_definitions/movie_json_smoke_test.rb

it's here that I grab the parameters coming form the table / test and I pass them into the relevant spots.  

Here's an example:

Then /^the response should have the (.*) expected$/ do |category|

  @mquery["#{category}"].should be_true

end

First, notice the (.*), that is the placeholder for the variable in the test <category>. Second, the argument I'm grabbing |category| is coming from the test. It takes the Category value (from the data table in the test) and passes it through here.

The next line of code reads @mquery (that's the instance variable) ["#{category}"].should be_true

So this is the way ruby picks up the JSON objects ['category']['subcategory'] But since we're passing in a dynamic parameter here, we need double quotes. so ["#{Category}"] (the #{category} will be replaced with the value in the data table.

The .should be_true tripped me up. I kept trying == !nil and .should == !nil, nothing worked until I found it should be done like this .should be_true. This validates that the JSON element the method is attached to is present.

 At this point, I just iterate through each parameter in their JSON returned.  I go through each category and subcategory and make sure that it's all there.  

I figure that's a good smoke test.  If I worked at RT, and some dev committed some code to their service/api, I could just run this and quickly see that json categories are still present.  

Example 2: movielookup_data_validation.feature

https://github.com/wbwarnerb/movielookup/blob/master/features/movielookup_data_validation.feature

In this test, I'm treating a data table in the test as the source of truth.  This could be a database, but I'm using a data table with Cucumber's Scenario Outline.  This time, I've added several columns.  I'm verifying the mpaa_rating and the profile image stored at Rotten Tomatoes.   

The code for verifying this is similar in concept to what I did previously:

https://github.com/wbwarnerb/movielookup/blob/master/features/step_definitions/movielookup_data_validation_test.rb

I go through several movies... passed through, like this:

Given /^the API is queried for (.*)$/ do |title|

  @mquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=#{title}&page_limit=1").read)

end

The argument is passed through and is passed through to the API URI and grabs that movie's details which are then verified

Notice that the main difference then before is this code:

 @mquery['movies'][0]['mpaa_rating'].should == "#{mpaa_rating}"

The main difference, is that I'm passing the argument through to use as a validation check.  

Example 3: /features/movielookup_json_validation.feature

https://github.com/wbwarnerb/movielookup/blob/master/features/movielookup_json_validation.feature

These tests I separated out of the other two.  These tests are made to just test the JSON functionality itself.

Given /^A request for an unknown movie is sent to the API$/ do

  @bquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=bbxa&page_limit=1").read)

end



Then /^the response should be zero movies returned$/ do

  @bquery['total'].should == 0

end



Given /^A API call is made for more then one movie$/ do

  @mquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=Toy+Story&page_limit=1").read)

end



Then /^the results wil have a total greater than 1$/ do

  @mquery['total'].should > 1



end



Basically in these two simple tests I'm checking that if a query is sent for a movie not in the RT movie database, that we get back a proper response. 

The second test verifies that if a query is sent for a movie with more then one hit (like Toy Story) it will display a total value greater the 1.

Rotten Tomatoes

Got IE working with WATIR/Cucumber and AJAX

My code for this, is up on github.com:
https://github.com/wbwarnerb/cucumber-demo

I had some issues getting IE working with WATIR/Cucumber, but they were resolved recently.  Really it didn't take a lot to resolve it.  What was happening, were two problems, causing one error.

The error read from Cucumber something about Selenium not finding the window/browser.

I did some googleing and found out that Cucumber needs to have IE to have the same security settings in each zone.  So I did that.  But I still got the error....

That's when I simply added a wait for an element to load.  Here's an example:
  @browser = Watir::Browser.new(:ie)
  @browser.goto "http://www.amazon.com"
  @browser.text_field(:name => "field-keywords").wait_until_present(10)

While Firefox doesn't need that "wait_until_present", IE did require it.  Once I passed that in and waited, it allowed IE to run the test with Amazon.com, finding the first element. 

Out of Process vs. In Process

In Cucumber/Ruby when testing a Restful API, you can test it either In Process or Out of Process.

The main difference here is the stack used.  In a "In Process" scenario, you are talking to the API within the process itself.  Meaning you don't need a webserver, http calls, or Web Framework at all.  This is faster since there's less overhead. 

In process works by using a gem called Rack-Test.  This is a library allowing the user to test the application.  It's like mocking out the webserver. 

"In Process" to me, seems the same as Unit Tests.

The downside here is that while the code is being directly tested - it is not going through a real world scenario.  This might be fine for Unit Testing, but it doesn't handle a real browser.

Out  Of Process handles a real world scenario.  In this case, you need to have the webserver up and running.  Then make calls to the REST service as you would in the production environment.  You would pass a HTTP request through the HTTP client, to the HTTP Server, etc. 

While this is slower, this is typically how QA Automation works.  QA would use a tool like WATIR or WATIR-Webdriver to drive multiple browsers against a site, restful service, api, etc. and get a response.

But you don't always need to do that.  Sometimes you just need to make a call, pass some parameters and get a result fast, so you can move on with a test.

An example would be, needing to create a user quickly to run a subscription scenario on your website.  You aren't testing user creation, so why go through the front end to create a user?  If you can create the user "in process" by simply passing some parameters to set up a user in a specific state... it's much better. 

The trick is knowing when to use In Process vs. Out of Process.  For QA, I think any point you are testing a flow the user would hit for the BDD's specified, you should be using Out of Process testing.  For any set up to get to a BDD's When state (i.e. the Given state), then you would use In Process. 

Watir Wait_Until_Present

In getting IE9 working with my cucumber framework, I figured out the problem.  IE9 was throwing errors of "can't find window" or "can't find driver" each time I ran the test.  After kicking it around a few times I tried a solution to wait for the element.  That seemed to be the solution.

We did this in GEB, where I work - using waitFor's.

 In Watir, it's like this:
@browser.text_field(:name => "field-keywords").wait_until_present(10)

basically it's saying wait for the text field named "field-keywords" for up to 10 seconds.

Right now I'm doing with Watir, and Cucumber what GEB hasn't got working in 2 years at the office - simply being cross browser compatible.  GEB has a hard time even closing the IE sessions... Watir and Cucumber make it so easy.

 

Error Handling in Rails

Rails makes error handling pretty easy.

In my WoW Quest lookup application, I pass a user inputted numeric to a API call.  I would get back a JSON result and then post the result to the results.html.erb page that the user would be routed to. 

It was fine, except that if a user inputs a numeric that doesn't map to a valid quest, the user would get a broken page. 

So here's what I did.  I found that if you send an invalid Quest ID to WoW's API, it returns this JSON:

{"status":"nok", "reason": "unable to get quest information."}

So in the controller, I made a modification on the Results action.  I added something called "rescue."

Rescue handles errors in Rails. 

So now, my results action looks like this, with the added bit in Red text below: 

  def results

   @output = JSON.parse(open("http://us.battle.net/api/wow/quest/"+params[:qid]).read)
  rescue
    render :template => 'search/error', :status => "nok"
  end 

Basically it's saying this:

If the JSON comes back with a element called "status" and it's equal to the string "nok", then redirect to "view/search/error.html.erb"

That's it.  Now if a user inputs a invalid quest id, like 667, it will go to my error.html.erb page... which has some HTML in it to basically say you hit an error, and put a link to go back.