Getting Safari Webdriver To Work

This is going to take a good part of your day... so be prepared if you want to do this.

Where I work, I set up a Cucumber automation framework that basically runs: IE9, FF, Chrome and I was just given a Mac Mini to support Safari testing.

The ultimate goal is to have: IE8, IE9, FF, Chrome and Safari - running in parallel.

FF, Chrome and IE are pretty easy to set up.  HOWEVER, Safari, is a big pain.

First, you need to do everything here:
http://watirmelon.com/2012/04/17/using-watir-webdriver-with-safari-at-last/

That will pretty much take half your day.  If you have a fresh Mac, you'll need xCode and you'll need to get the command line package for xCode so you can use svn.  What Alister Scott on that link will tell you to do can be summed up as:

• Make a Apple Safari Dev account (free)
• Get a certificate
• Download the certificate
• Open Safari and install the certificate
• One thing not mentioned in the instructions, is that you can't install the cerficiate till Safari knows you're a developer, to do that you have to open the key chain on the mac and click system and drag/drop your certificate and they you must right-click the certificate, and set all actions as "TRUST" and finally you have to accept these changes.
• Then you build the extension (downloading via svn the code, and building the extension.

That's where Alister's tutorial ends... but you'll no doubt get a problem.  If you are like most, you'll get Safari hitting 127.0.0.1/wd for each test... not hitting your target site, and you'll get an error referencing "EOFError: end of file reached" One of the comments on Alister's post has the solution...

You now need to do a gem install selenium-webdriver (i couldn't do the -pre that he suggested, so i just did gem install selenium-webdriver) and I added this to my env.rb file as gem 'selenium webdriver'

After that, just rebuild: bundle update and you should be good.

Restful Web Service in Scala and Lift

I just put up the skeleton of a rest service, I hacked together in Scala and Lift... the code is up on my github: https://github.com/wbwarnerb/Scala-RestfulService

I can't take full credit, I used a variety of tutorials (cited in the readme) and came up with that code as my solution.

Currently it has hard coded data for a city (Burbank) and it sets up a restful url of:
.../api/city/
when Burbank is caught, it returns XML data.

I'd like this to get hooked into a database (the mysql "world" sample database) and I'd like to get CRUD actions running on the service, so I can post, delete and get various values.

But I got the skeleton done.

Getting Intellij Set up with Lift and Scala

I didn't find getting Scala set up too hard with Intellij... it was just the plugin. But getting Lift or other frameworks like Play set up with Intellij was a real beast.

I found a variety of information online and put it together as my steps to getting this working on Windows, with Intellij.  Without doing this, Intellij will not understand your imports like import net.liftweb._  and will complain about "Unknown symbol" or something like that.

What you will need installed is:

• Scala of course
• Download Lift
• Install Maven
• Get SBT (http://www.scala-sbt.org/release/docs/Getting-Started/Setup.html#manual-installation)

When you have all of this installed and on your windows env path, open a new console (note if your console was open while you were installing you'll need to restart it to pick up the path changes) and do the following:

1. Generating the Maven Archetype:
go into a folder of your choice and do:
mvn archetype:generate \
  -DarchetypeGroupId=net.liftweb \
  -DarchetypeArtifactId=lift-archetype-basic_2.9.1 \
  -DarchetypeVersion=2.4-M5 \
  -DarchetypeRepository=http://scala-tools.org/repo-snapshots \
  -DremoteRepositories=http://scala-tools.org/repo-snapshots \
  -DgroupId=com.company \
  -DartifactId=lift_test \
  -Dversion=1.0

On Windows, replace "\" with "^" for multi-line commands

2. Once done, run SBT in this folder with simply typing sbt and hitting ENTER.

3. In Intellij create a project and do an import project from external model (Maven) and pull this project in.


You can now write your code and save etc.  I didn't have to do anything special to get Scala to run... it was all set up from my previous work getting the scala plugin installed.

When you decide it's time to try the lift app out, you simply go to the lift project on the command line and type:
sbt update ~jetty-run

Scala Object Update - Formatting the map results

Working on the same Scala Script I pretty much finished it tonight.

I got the output to finally adhere to what I wanted:
title,category
title,category
...

Initially I was getting the entire map.  like the output was map("something")
Then when I previously combined maps I got output like
Title
Category
Title
Category

which isn't what I wanted.

So I updated the code to create a new method called jsonOutput, like this:
import scala.util.parsing.json._

object Parser {
  def main(args:Array[String]){
    for (i <- 1 until 1000) {
      try {
        val json = JSON.parseFull(scala.io.Source.fromURL("http://us.battle.net/api/wow/quest/" + i).getLines().mkString("\n"))
        val jsonType = json.asInstanceOf[Option[Map[String, String]]]
        val jsonTitle = jsonType.map(_("title")).get  
        val jsonLoc = jsonType.map(_("category")).get
        def jsonOutput() = jsonTitle+","+jsonLoc+"\n"

        jsonOutput() foreach {
          print
        }
      }catch {
        case e: Exception =>
      }
    }
  }
}

The updates are in Red above.  The first thing I did was update the foreach to point to my new method (ditching my map ++ combining idea.)  I constructed the format in the method:
def jsonOutput() = jsonTitle+","+jsonLoc+"\n"

So I didn't need a println anymore, I opted for print.

This returned output like:
Some(Kanrethad's Quest),Some(Designer Island)

I really wanted the values in the Some... so I saw on StackOverflow they used .get method to retrieve the result... http://stackoverflow.com/questions/6385383/problem-with-outputting-map-values-in-scala

So I appended .get to the maps values I'm taking... and I got the anticipated output of:
Kanrethad's Quest,Designer Island
Sharptalon's Claw,Ashenvale
Riverpaw Gnoll Bounty,Elwynn Forest
Give Gerard a Drink,Elwynn Forest
...

Finally, I wanted to also output some values that were not Strings... but Integers... like the level of each quest.  To do that, I added a new map:
val jsonInt = json.asInstanceOf[Option[Map[String, Any]]]

Then, like before, called it with
val jsonLev = jsonInt.map(_("level")).get

And finally updated the jsonOutput method to be:
def jsonOutput() = jsonTitle+","+jsonTitle+","+jsonLev+"\n"

Which would give the output of:
Kanrethad's Quest,Designer Island,80.0
Sharptalon's Claw,Ashenvale,23.0
Riverpaw Gnoll Bounty,Elwynn Forest,10.0
Give Gerard a Drink,Elwynn Forest,1.0
Ursangous' Paw,Ashenvale,24.0
Shadumbra's Head,Ashenvale,24.0
Simmer Down Now,Ashenvale,25.0
Further Concerns,Elwynn Forest,10.0

Refactoring

I just realized I didn't need to have a duplicate MAP for an integer... so I reused the same map jsonType and just set it to String, Any.

The final code is:
import scala.util.parsing.json._

object Parser {
  def main(args:Array[String]){
    for (i <- 1 until 1000) {
      try {
        val json = JSON.parseFull(scala.io.Source.fromURL("http://us.battle.net/api/wow/quest/" + i).getLines().mkString("\n"))
        val jsonType = json.asInstanceOf[Option[Map[String, Any]]]
        val jsonTitle = jsonType.map(_("title")).get
        val jsonLoc = jsonType.map(_("category")).get
        val jsonLev = jsonType.map(_("level")).get
        def jsonOutput() = jsonTitle+","+jsonLoc+","+jsonLev+"\n"

        jsonOutput() foreach {
          print
        }
      }catch {
        case e: Exception =>
      }
    }
  }
}

Changing Types in Scala

More on that Scala object....

I changed my Scala object a bit, to break out the items in the map it was getting.  What it was doing was going to an API, getting some data, and then getting more data, discarding any errors.  However, the data it was getting was coming back as a map.

Since JSON has both integers and strings in it, it returned as Map Type Any.

I couldn't understand why I just couldn't print out the values from the map.  But I would get an error saying that I couldn't apply println or other methods to Type Any.

What it's saying is, that I need to transform the Type from Any to String.



import scala.util.parsing.json._
object Parser {
  def main(args:Array[String]){
    for (i <- 1 until 1000) {
      try {
        val json = JSON.parseFull(scala.io.Source.fromURL("http://us.battle.net/api/wow/quest/" + i).getLines().mkString("\n"))
        val jsonType = json.asInstanceOf[Option[Map[String, String]]]
        val jsonTitle = jsonType.map(_("title"))
        val jsonLoc = jsonType.map(_("category"))
        val jsonComplete = jsonTitle ++ jsonLoc

        jsonComplete foreach {println}
      }catch {
        case e: Exception =>
      }
    }
  }
}

I did it by doing the above in Red.  I thought that the mkString appended to the val json would have done this, but it didn't.  In fact I had to make a new variable (val jsonType) that takes the json variable and does a asInstanceOf[Option[Map[String, String]]  This was how it worked for the JSON I was getting back.

Then to pull out a element from the map, I'm using a variable to jsonType.map(_("title")) or whatever value I want.

My current frustration though, is that by doing it this way, I have to combine the maps to get all the variables I want to print out.  Like if I want: title, category to print line by line, it's tough. I add the two maps together with the ++ operator and then print out that new variable.  But it prints like this:
title
category
title
category

It's not the formatting I want. ugh!  So that's my next hurdle.

API Scraper written in Scala

I redid the Ruby script I made, in Scala.  This is the scala version... it seems to work better with less duplicate data:

import scala.util.parsing.json._
object Parser {
  def main(args:Array[String]){
    for (i <- 1 until 1000) {
      try {
        val json = JSON.parseFull(scala.io.Source.fromURL("http://us.battle.net/api/wow/quest/"+i).getLines().mkString("\n"))
        json foreach {println}
      } catch {
        case e: Exception =>
      }
    }
  }
}

It works like this.... it's pretty simple.  I'm using the scala.util.parsing.json library to parse some json i'll be getting back from a public API.

I start with a for loop and say i = 1, lets increment this till we reach a value (I picked 1000.)
Within the loop I do a try, because the try will let me catch exceptions... I already know that not all values passed into the api will work. some id's will be a 404.  I dont want the app to stop because it fails on a id.

I set a immutable variable (val) to the JSON.parseFull.... which ultimately goes to WoW's quest API and I append a: +i.  The i, being the value here that's incrementing in the for loop.

For each result, I print it... using foreach.

Since I'm using "try" i'm also using "catch" and making sure I catch exceptions.... and move on.

Ruby Script to Scrape an API

I wrote a basic Ruby script last weekend to scrape a public API for data.

The public API is World of Warcraft's Quest Data API.  It takes a quest ID, and returns information on the quest (like quest title, start location, level, etc.)  The problem is, I don't know Quests by ID.  I want to query by Level, or start location.  So I thought about pulling down their data, and sorting it locally.

So I wrote this little script:
 
require 'open-uri'
require 'json'

  def Rubyresults(y)
      $a=0
      begin
          @output = JSON.parse(open("http://us.battle.net/api/wow/quest/"+"#{y}").read)
          puts @output['title']+", "+ @output['category']+","+@output['level'].to_s
          y=y+$a
          $a +=1
      end  until y > 9269

      rescue
        #puts "invalid quest id"+","+@output['id'].to_s
        y=y+1
        retry
  end

  puts Rubyresults(1)

Script Breakdown
I start off by importing/requiring two libraries (open-uri and json), these allow me to call a URL/URI and to be able to parse the JSON results returned.

I built the script to take a param (in my case "y"), which is the quest ID I want to start at.  So I do a:
def Rubyresults(y)

Next, I assign a variable $a to 0, this will be my counter.

Now I start the loop.  I use a Began loop and go until a certain value is reached.
I start off in the loop with a instance variable of @output and assign it to hit the Wold of Warcraft quest API.  I leave the ID a variable: "#{y}"  This will be the value being passed in and incremented.

The next line outputs the results in JSON.  since I previously used JSON.parse on that URL, I can now use @output['title'] (or any category returned in the JSON) to return the value.  I set this output to be comma separated so that I could sort this as a CSV doc later.  so I have @output['title'] + ","+ @output['category'] and so forth.  So the output will be title,category,level per line.

Now I increment the value of Y my adding $a and $a is incremented by 1 each time. So if I start this at quest id 1, it will turn $a to 2, and 3 etc. and add it to y each time.

The end of the loop is set to stop at 9269. 

Now for the Rescue.  I added the rescue, because the World of Warcraft API doesn't have all ID's.  For example it has a quest for id = 1 and 2, but not 3.  If you try and query for quest id 3, it will return a 404.   If that happens, the whole script errors and stops.  So Rescue lets me continue on.  I initially added a puts to output the invalid id, but later commented it out. 

In the rescue routine, I add 1 to y and retry... this continues the script.

At the very end is the puts Rubyresults(1), this sets the quest id value to start with, which is being passed in as value y.

The output of this script looks like (I need to work on not getting duplicate entries):

Kanrethad's Quest, Designer Island,80
Sharptalon's Claw, Ashenvale,23
Give Gerard a Drink, Elwynn Forest,1
Ursangous' Paw, Ashenvale,24
Ursangous' Paw, Ashenvale,24
Shadumbra's Head, Ashenvale,24
A Fishy Peril, Elwynn Forest,10
A Fishy Peril, Elwynn Forest,10
Discover Rolf's Fate, Elwynn Forest,10
Discover Rolf's Fate, Elwynn Forest,10
Bounty on Murlocs, Elwynn Forest,10
Protect the Frontier, Elwynn Forest,10
Protect the Frontier, Elwynn Forest,10
Cloth and Leather Armor, Elwynn Forest,10
Cloth and Leather Armor, Elwynn Forest,10
The Fargodeep Mine, Elwynn Forest,7
The Fargodeep Mine, Elwynn Forest,7
Report to Thomas, Elwynn Forest,10
Report to Thomas, Elwynn Forest,10
The Jasperlode Mine, Elwynn Forest,10
The Jasperlode Mine, Elwynn Forest,10
Fine Linen Goods, Elwynn Forest,10
Fine Linen Goods, Elwynn Forest,10

Setting up Scala with Intelij

I had a few problems getting this combo set up... so I followed a few tutorials and got this set up on my windows laptop.

Steps:

1. Make sure you have Java 1.6 or higher installed
2. Download Scala for windows (get the zip!!) the Windows binary installer of scala doesn't install with everything you need (like src folder!)  but the zip has it all
3. Uncompress Scala someplace
4. Download Intelij
5. Install Intelij
6. Within Intelij open the plugin manager: File / Settings / Plugins
7. in the plugin manager click browse
8. Go to Scala and double click it.  accept to install it
9. After it installs restart intelij
10. Make a new Java Module Project
11. On the next page of project creation check "Scala"
12. Now you prob got a path to Scala and it says "not valid home" - browse to the uncompressed Scala uncompressed folder (root of it) - it should find all the src library and compilers it needs

That should do it!

Drying the Cucumber - Calling Steps from Within Steps

After adding quite a few Cucumber tests to my current automation framework, I wanted to get it a bit more clean.  There's a lot of code, some of which could be reused.

First I looked at this as a Ruby problem and tried to create classes within the Cucumber Step Definition files and do some inheritance.

However, I found that Cucumber has it's own way of handling this.  You can call a step from within a step.  Similar to a method call from a different class, you can do something like this:

Given /^a user who is at the Profile Edit screen$/ do
    step "user logs in"
    .....
end

That code:
step "user logs in"
will grab the code driving that step. In my case it could be a login.rb file that has
Given /^user logs in$/ do
    @browser = Watir::Browser.new browser_type
    @browser.goto "http://www......"
    ....
end

Doing this helps dry up tests and keep the code clean.

Automating File Upload prompts

I really love using Watir with Cucumber.  It has some great built in methods.

There's this awesome method in Watir called file_field.  It basically references any input of type = "file" and follows the prompts to push the file you give through the submit process.

Here's an example of how I'm doing it:

  @browser.span(:text=>"Upload New").click
  @browser.file_field(:class=>"UploadSoundfile").set("C:\\Users\\Public\\Music\\Sample Music\\Kalimba.mp3")
  sleep(2)
  @browser.div(:text=>"Submit").click
  sleep(2)

For windows to work, I did have to escape the \ with double \\.

But it works like  a charm!

0102/085939:ERROR:ipc_sync_channel.cc

I noticed I've been getting these errors via Chrome when webdriver closes the session:

0102/085939:ERROR:ipc_sync_channel.cc(378)
 
I saw some other people also were getting these. I checked with Google and it appears this is a benign error.  It doesn't seem to affect the results.