Changing Types in Scala

More on that Scala object....

I changed my Scala object a bit, to break out the items in the map it was getting.  What it was doing was going to an API, getting some data, and then getting more data, discarding any errors.  However, the data it was getting was coming back as a map.

Since JSON has both integers and strings in it, it returned as Map Type Any.

I couldn't understand why I just couldn't print out the values from the map.  But I would get an error saying that I couldn't apply println or other methods to Type Any.

What it's saying is, that I need to transform the Type from Any to String.



import scala.util.parsing.json._
object Parser {
  def main(args:Array[String]){
    for (i <- 1 until 1000) {
      try {
        val json = JSON.parseFull(scala.io.Source.fromURL("http://us.battle.net/api/wow/quest/" + i).getLines().mkString("\n"))
        val jsonType = json.asInstanceOf[Option[Map[String, String]]]
        val jsonTitle = jsonType.map(_("title"))
        val jsonLoc = jsonType.map(_("category"))
        val jsonComplete = jsonTitle ++ jsonLoc

        jsonComplete foreach {println}
      }catch {
        case e: Exception =>
      }
    }
  }
}

I did it by doing the above in Red.  I thought that the mkString appended to the val json would have done this, but it didn't.  In fact I had to make a new variable (val jsonType) that takes the json variable and does a asInstanceOf[Option[Map[String, String]]  This was how it worked for the JSON I was getting back.

Then to pull out a element from the map, I'm using a variable to jsonType.map(_("title")) or whatever value I want.

My current frustration though, is that by doing it this way, I have to combine the maps to get all the variables I want to print out.  Like if I want: title, category to print line by line, it's tough. I add the two maps together with the ++ operator and then print out that new variable.  But it prints like this:
title
category
title
category

It's not the formatting I want. ugh!  So that's my next hurdle.

API Scraper written in Scala

I redid the Ruby script I made, in Scala.  This is the scala version... it seems to work better with less duplicate data:

import scala.util.parsing.json._
object Parser {
  def main(args:Array[String]){
    for (i <- 1 until 1000) {
      try {
        val json = JSON.parseFull(scala.io.Source.fromURL("http://us.battle.net/api/wow/quest/"+i).getLines().mkString("\n"))
        json foreach {println}
      } catch {
        case e: Exception =>
      }
    }
  }
}

It works like this.... it's pretty simple.  I'm using the scala.util.parsing.json library to parse some json i'll be getting back from a public API.

I start with a for loop and say i = 1, lets increment this till we reach a value (I picked 1000.)
Within the loop I do a try, because the try will let me catch exceptions... I already know that not all values passed into the api will work. some id's will be a 404.  I dont want the app to stop because it fails on a id.

I set a immutable variable (val) to the JSON.parseFull.... which ultimately goes to WoW's quest API and I append a: +i.  The i, being the value here that's incrementing in the for loop.

For each result, I print it... using foreach.

Since I'm using "try" i'm also using "catch" and making sure I catch exceptions.... and move on.

Ruby Script to Scrape an API

I wrote a basic Ruby script last weekend to scrape a public API for data.

The public API is World of Warcraft's Quest Data API.  It takes a quest ID, and returns information on the quest (like quest title, start location, level, etc.)  The problem is, I don't know Quests by ID.  I want to query by Level, or start location.  So I thought about pulling down their data, and sorting it locally.

So I wrote this little script:
 
require 'open-uri'
require 'json'

  def Rubyresults(y)
      $a=0
      begin
          @output = JSON.parse(open("http://us.battle.net/api/wow/quest/"+"#{y}").read)
          puts @output['title']+", "+ @output['category']+","+@output['level'].to_s
          y=y+$a
          $a +=1
      end  until y > 9269

      rescue
        #puts "invalid quest id"+","+@output['id'].to_s
        y=y+1
        retry
  end

  puts Rubyresults(1)

Script Breakdown
I start off by importing/requiring two libraries (open-uri and json), these allow me to call a URL/URI and to be able to parse the JSON results returned.

I built the script to take a param (in my case "y"), which is the quest ID I want to start at.  So I do a:
def Rubyresults(y)

Next, I assign a variable $a to 0, this will be my counter.

Now I start the loop.  I use a Began loop and go until a certain value is reached.
I start off in the loop with a instance variable of @output and assign it to hit the Wold of Warcraft quest API.  I leave the ID a variable: "#{y}"  This will be the value being passed in and incremented.

The next line outputs the results in JSON.  since I previously used JSON.parse on that URL, I can now use @output['title'] (or any category returned in the JSON) to return the value.  I set this output to be comma separated so that I could sort this as a CSV doc later.  so I have @output['title'] + ","+ @output['category'] and so forth.  So the output will be title,category,level per line.

Now I increment the value of Y my adding $a and $a is incremented by 1 each time. So if I start this at quest id 1, it will turn $a to 2, and 3 etc. and add it to y each time.

The end of the loop is set to stop at 9269. 

Now for the Rescue.  I added the rescue, because the World of Warcraft API doesn't have all ID's.  For example it has a quest for id = 1 and 2, but not 3.  If you try and query for quest id 3, it will return a 404.   If that happens, the whole script errors and stops.  So Rescue lets me continue on.  I initially added a puts to output the invalid id, but later commented it out. 

In the rescue routine, I add 1 to y and retry... this continues the script.

At the very end is the puts Rubyresults(1), this sets the quest id value to start with, which is being passed in as value y.

The output of this script looks like (I need to work on not getting duplicate entries):

Kanrethad's Quest, Designer Island,80
Sharptalon's Claw, Ashenvale,23
Give Gerard a Drink, Elwynn Forest,1
Ursangous' Paw, Ashenvale,24
Ursangous' Paw, Ashenvale,24
Shadumbra's Head, Ashenvale,24
A Fishy Peril, Elwynn Forest,10
A Fishy Peril, Elwynn Forest,10
Discover Rolf's Fate, Elwynn Forest,10
Discover Rolf's Fate, Elwynn Forest,10
Bounty on Murlocs, Elwynn Forest,10
Protect the Frontier, Elwynn Forest,10
Protect the Frontier, Elwynn Forest,10
Cloth and Leather Armor, Elwynn Forest,10
Cloth and Leather Armor, Elwynn Forest,10
The Fargodeep Mine, Elwynn Forest,7
The Fargodeep Mine, Elwynn Forest,7
Report to Thomas, Elwynn Forest,10
Report to Thomas, Elwynn Forest,10
The Jasperlode Mine, Elwynn Forest,10
The Jasperlode Mine, Elwynn Forest,10
Fine Linen Goods, Elwynn Forest,10
Fine Linen Goods, Elwynn Forest,10

Setting up Scala with Intelij

I had a few problems getting this combo set up... so I followed a few tutorials and got this set up on my windows laptop.

Steps:

1. Make sure you have Java 1.6 or higher installed
2. Download Scala for windows (get the zip!!) the Windows binary installer of scala doesn't install with everything you need (like src folder!)  but the zip has it all
3. Uncompress Scala someplace
4. Download Intelij
5. Install Intelij
6. Within Intelij open the plugin manager: File / Settings / Plugins
7. in the plugin manager click browse
8. Go to Scala and double click it.  accept to install it
9. After it installs restart intelij
10. Make a new Java Module Project
11. On the next page of project creation check "Scala"
12. Now you prob got a path to Scala and it says "not valid home" - browse to the uncompressed Scala uncompressed folder (root of it) - it should find all the src library and compilers it needs

That should do it!

Drying the Cucumber - Calling Steps from Within Steps

After adding quite a few Cucumber tests to my current automation framework, I wanted to get it a bit more clean.  There's a lot of code, some of which could be reused.

First I looked at this as a Ruby problem and tried to create classes within the Cucumber Step Definition files and do some inheritance.

However, I found that Cucumber has it's own way of handling this.  You can call a step from within a step.  Similar to a method call from a different class, you can do something like this:

Given /^a user who is at the Profile Edit screen$/ do
    step "user logs in"
    .....
end

That code:
step "user logs in"
will grab the code driving that step. In my case it could be a login.rb file that has
Given /^user logs in$/ do
    @browser = Watir::Browser.new browser_type
    @browser.goto "http://www......"
    ....
end

Doing this helps dry up tests and keep the code clean.

Automating File Upload prompts

I really love using Watir with Cucumber.  It has some great built in methods.

There's this awesome method in Watir called file_field.  It basically references any input of type = "file" and follows the prompts to push the file you give through the submit process.

Here's an example of how I'm doing it:

  @browser.span(:text=>"Upload New").click
  @browser.file_field(:class=>"UploadSoundfile").set("C:\\Users\\Public\\Music\\Sample Music\\Kalimba.mp3")
  sleep(2)
  @browser.div(:text=>"Submit").click
  sleep(2)

For windows to work, I did have to escape the \ with double \\.

But it works like  a charm!

0102/085939:ERROR:ipc_sync_channel.cc

I noticed I've been getting these errors via Chrome when webdriver closes the session:

0102/085939:ERROR:ipc_sync_channel.cc(378)
 
I saw some other people also were getting these. I checked with Google and it appears this is a benign error.  It doesn't seem to affect the results.