API Testing

API testing doesn't involve a GUI but the testing does come down to the same concepts of inputs and outputs.  You provide a specific input and you get the expected output.   Just like with any type of testing, the methodology is the same.  You would verify the Happy Path, and try and break the API (passing invalid parameters, too many parameters, etc. and verify errors are thrown gracefully, that databases are not updated with empty rows, or duplicate data... and son on.)

So what is an API?

An API is a interface you can use in development, it stands for Application Programming Interface.  Think of it as a building block of code. Rather then writing that code each time, you have a block of code that can take parameters and use those parameters to do something.  You could pass in parameters to create a user in a db, or perform a query and return data as JSON, or handle authentication (such as Facebook login api's.)  API's can be public or private. 

Verifying An API

Some examples to verify in API testing could be:

1. What does the API return? If an API is designed to return a result (like data), then you would want to know that a) there is something returned and b) it's what's expected and c) it fits the boundaries of what's expected d) errors are handled correctly
2. Event Listener Testing: If the API is making an event, you would have to have access to an event listener or event log and verify that the event is a) captured correctly b) it's in the expected format and c) has the correct payload d) errors are handled correctly
3. Modifying Databases: If the API is making a DB insert or update, you would want to verify that a) the db update/insert/delete occurs b) the data that is changed is correct c) errors are handled correctly.

Early on at eHarmony I had the chance to write a simple API for testing. We were designing a new Automation framework and I needed some code to subscribe users.  This way I could pass in a userid, and it would subscribe this user for me programmaticly.  This was not a production API, but just something used in testing. We had test credit card numbers, that are not valid in production. But it lets us exercise the subscription flow.  So instead of calling the UI on the front end each time to run a subscription (which might take 2 min each test, just to sub a user via the GUI), I would subscribe a user using this simple API. 

What this API was doing was taking the userid as a parameter and passing it into a SQL insert to our test db.

Testing it was amounting to:

1. The Happy Path: Does the data insert/update for a userid reach the db. New users would be inserted to the db, existing users would be updated. 
2. Validate the data inserted: are all columns expected to be updated, updated?  Is the user flagged as a subscriber?
3. Error Cases: What happens if an empty string is passed in? it should error gracefully, does it?  What happens if the user is already a subscriber. That would mean the insert would fail for new users.  Updates would pass.
4. Events: In this case we didn't want to trigger any events. This should bypass the events, so just double check that no events are recorded in the event service logs.

How To Get Into API Testing

There's a lot of ways a person can pick up API testing on their own.  If you can write code in Rails/Ruby, Groovy, JAVA, etc, you could build your API and then test it!

But... there are easier ways.  You could find a open/public API and work with it.  Some examples of API's you could start with would be:

• Rotten Tomatoes - http://developer.rottentomatoes.com/ This is a great API to work with. It's very easy to pass parameters to and get back JSON data.  
• Facebook - https://developers.facebook.com/docs/reference/apis/
• Google Maps - https://developers.google.com/maps/

In fact you can find a lot of online API's that take data and return some sort of result to you.  I even found one for the card game, Magic the Gathering:
http://daccg.com/ajax_ccgsearch.php?cardname=jace%20beleren
You can simply set up a form to pass in a parmeter here (URL encoded of course) and get back some JSON with the result!

The above list is of course online API's, many of them restful service with API's you can hit with a URI endpoint.  But it is easy to get started right now!

More detail on JSON validation in Cucumber

Here's more details on my last post about doing a JSON smoke test.  The idea here, is that a test is needed to check the schema of the JSON coming back - to verify that the JSON structure itself is intact with the correct categories and sub categories.

I used Rotten Tomato's API end point.  I'll pass in a fixed parameter to a specific movie and then validate the JSON coming back.

The end point URI that i'm hitting is:
http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=Toy+Story+3&page_limit=1

Let's review the JSON response we get with the above query - jsonlint.com shows that the top level categories are:

• total
• movies
• links
• link_template

Some of these categories have sub categories, and some do not.  For example, Total, does not have a sub category in the JSON, but movies has a bulk of sub categories.  In fact movies has:

• id
• title
• year
• mpaa_rating
• runtime
• critics_consensus
• release_dates
• ratings
• synopsis
• posters
• abridged_cast
• alternate_ids
• links

some of these of sub categories themselves.  But to just cover these categories and sub-categories here's how I wrote the code in the step definition file:
require 'open-uri'
require 'json'

Given /^A call to the Rotten Tomattoes API$/ do
  @mquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=Toy+Story+3&page_limit=1").read)
end
Then /^the response should have  the (.*) expected$/ do |category|
  @mquery["#{category}"].should be_true
end

Given /^A call is made to the Rotten Tomattoes API$/ do
  @mquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=Toy+Story+3&page_limit=1").read)
end
Then /^the response returned should have the movies (.*) expected$/ do |subcategory|
  @mquery['movies'][0]["#{subcategory}"].should be_true
end
Then /^the response returned should have the (.*) subcategory expected$/ do |release_dates_sub|
  @mquery['movies'][0]['release_dates']["#{release_dates_sub}"].should be_true
end
Then /^the ratings response returned should have the (.*) subcategory expected$/ do |ratings_sub|
  @mquery['movies'][0]['ratings']["#{ratings_sub}"].should be_true
end
Then /^the posters response returned should have the (.*) subcategory expected$/ do |posters_sub|
  @mquery['movies'][0]['posters']["#{posters_sub}"].should be_true
end
Then /^the abridged cast response returned should have the (.*) subcategory expected$/ do |abridged_cast_sub|
  @mquery['movies'][0]['abridged_cast'][0]["#{abridged_cast_sub}"].should be_true
end
Then /^the alternate id's response returned should have the (.*) subcategory expected$/ do |alternate_ids_sub|
  @mquery['movies'][0]['alternate_ids']["#{alternate_ids_sub}"].should be_true
end
Then /^the links response returned should have the (.*) subcategory expected$/ do |links_sub|
  @mquery['movies'][0]['links']["#{links_sub}"].should be_true
end


The Cucumber test I wrote uses a Cucumber Scenario Outline to pass in the expected categories or sub-categories into the code above.  Here's how I wrote the tests:

Feature: Smoke tests to ensure json validity by checking and verifying each JSON category and sub category exists
  #Note: this is not validating values, just that the JSON structure has the expected categories and sub categories

   Scenario Outline: Smoke test the JSON top level categories returned
     Given A call to the Rotten Tomattoes API
     Then the response should have  the <category> expected

     Examples:
     |category|
     |movies  |
     |links|
     |link_template|

  Scenario Outline: Smoke test the JSON movies sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the response returned should have the movies <subcategory> expected

  Examples:
    |subcategory|
    |id  |
    |title|
    |year|
    |mpaa_rating|
    |runtime    |
    |critics_consensus|
    |release_dates    |
    |ratings          |
    |synopsis         |
    |posters          |
    |abridged_cast    |
    |alternate_ids    |
    |links            |

  Scenario Outline: Smoke test the JSON release dates sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the response returned should have the <release dates_sub> subcategory expected

  Examples:
    |release dates_sub|
    |theater  |
    |dvd|

  Scenario Outline: Smoke test the JSON ratings sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the ratings response returned should have the <ratings_sub> subcategory expected

  Examples:
    |ratings_sub|
    |critics_rating  |
    |critics_score|
    |audience_rating|
    |audience_score |

  Scenario Outline: Smoke test the JSON posters sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the posters response returned should have the <posters_sub> subcategory expected

  Examples:
    |posters_sub|
    |thumbnail  |
    |profile|
    |detailed|
    |original |

  Scenario Outline: Smoke test the JSON abridged cast sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the abridged cast response returned should have the <abridged_cast_sub> subcategory expected

  Examples:
    |abridged_cast_sub|
    |name  |
    |id|
    |characters|

  Scenario Outline: Smoke test the JSON alternate ids sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the alternate id's response returned should have the <alternate_ids_sub> subcategory expected

  Examples:
    |alternate_ids_sub|
    |imdb  |

  Scenario Outline: Smoke test the JSON links sub categories returned
    Given A call is made to the Rotten Tomattoes API
    Then the links response returned should have the <links_sub> subcategory expected

  Examples:
    |links_sub|
    |self  |
    |alternate|
    |cast     |
    |clips    |
    |reviews  |
    |similar  |

Testing API's with Cucumber

I put together several API tests.  Here's the project on Github.com:
https://github.com/wbwarnerb/movielookup

The project has three Cucumber features (under the features directory) that talk with a public API.

For the purpose of this test I'm using Rotten Tomatoes public API.  They have a nice JSON response to parse and utilize for these examples.

Example 1: /features/movie_json_smoke.feature

The first feature is a smoke test it basically validates the json itself.  The idea here is: We know what we expect to return in the JSON, we verify that the categories and sub categories are intact and no changes have occurred. This is useful, if a change to the service was made, but the overall structure of the JSON should remain the same.  Just kick off this test, and it validates the JSON structure.

Here's how I did it in cucumber:
https://github.com/wbwarnerb/movielookup/blob/master/features/movie_json_smoke.feature
from here we have the Cucumber test itself.  I build the test out as a Scenario Outline, this allows me to utilize the data tables.  When I get to the "Then", I pass in a parameter from the table, like this:

Then the response should have  the <category> expected

The category parameter matches the data table, which has a column header "category" - every element under that column in the table is passed into the Then statement.

The actual code is in the step definition:

https://github.com/wbwarnerb/movielookup/blob/master/features/step_definitions/movie_json_smoke_test.rb

it's here that I grab the parameters coming form the table / test and I pass them into the relevant spots.  

Here's an example:

Then /^the response should have the (.*) expected$/ do |category|

  @mquery["#{category}"].should be_true

end

First, notice the (.*), that is the placeholder for the variable in the test <category>. Second, the argument I'm grabbing |category| is coming from the test. It takes the Category value (from the data table in the test) and passes it through here.

The next line of code reads @mquery (that's the instance variable) ["#{category}"].should be_true

So this is the way ruby picks up the JSON objects ['category']['subcategory'] But since we're passing in a dynamic parameter here, we need double quotes. so ["#{Category}"] (the #{category} will be replaced with the value in the data table.

The .should be_true tripped me up. I kept trying == !nil and .should == !nil, nothing worked until I found it should be done like this .should be_true. This validates that the JSON element the method is attached to is present.

 At this point, I just iterate through each parameter in their JSON returned.  I go through each category and subcategory and make sure that it's all there.  

I figure that's a good smoke test.  If I worked at RT, and some dev committed some code to their service/api, I could just run this and quickly see that json categories are still present.  

Example 2: movielookup_data_validation.feature

https://github.com/wbwarnerb/movielookup/blob/master/features/movielookup_data_validation.feature

In this test, I'm treating a data table in the test as the source of truth.  This could be a database, but I'm using a data table with Cucumber's Scenario Outline.  This time, I've added several columns.  I'm verifying the mpaa_rating and the profile image stored at Rotten Tomatoes.   

The code for verifying this is similar in concept to what I did previously:

https://github.com/wbwarnerb/movielookup/blob/master/features/step_definitions/movielookup_data_validation_test.rb

I go through several movies... passed through, like this:

Given /^the API is queried for (.*)$/ do |title|

  @mquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=#{title}&page_limit=1").read)

end

The argument is passed through and is passed through to the API URI and grabs that movie's details which are then verified

Notice that the main difference then before is this code:

 @mquery['movies'][0]['mpaa_rating'].should == "#{mpaa_rating}"

The main difference, is that I'm passing the argument through to use as a validation check.  

Example 3: /features/movielookup_json_validation.feature

https://github.com/wbwarnerb/movielookup/blob/master/features/movielookup_json_validation.feature

These tests I separated out of the other two.  These tests are made to just test the JSON functionality itself.

Given /^A request for an unknown movie is sent to the API$/ do

  @bquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=bbxa&page_limit=1").read)

end



Then /^the response should be zero movies returned$/ do

  @bquery['total'].should == 0

end



Given /^A API call is made for more then one movie$/ do

  @mquery = JSON.parse(open("http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=jc7eaxjfpemb2uz7qsrudnsq&q=Toy+Story&page_limit=1").read)

end



Then /^the results wil have a total greater than 1$/ do

  @mquery['total'].should > 1



end



Basically in these two simple tests I'm checking that if a query is sent for a movie not in the RT movie database, that we get back a proper response. 

The second test verifies that if a query is sent for a movie with more then one hit (like Toy Story) it will display a total value greater the 1.

Rotten Tomatoes