23 July 2013

Grails - Creating an Auto Dialer

I had a need for an internal tool where I work.

The premise is that our company buys and stores various numbers that they lease out later.  However, if those numbers do not have activity in a certain amount of time, they can become deactivated.

In the past, the company had individuals who would (every few months) get a list of current numbers in the risk zone for deactivation (hundreds of numbers) and dial each one to make sure it still went to our menu. 

So the Provisioning Team reached out to me to ask me if this could be automated....

They're requirements were that:
    a) they have lists of phone numbers (DNIS) that need to be called. this list could be a CSV
    b) they would like to upload it to a tool which would dial each number validating if it was active or not
    c) they would be notified of the results somehow.

I put together this tool and here's how I did it...

Basically I built an auto dialer with Grails and SIPCLI.  I could have used SIPP, but since I foresaw a future need to do text to speech, I opted for sipcli.  The downside to sipcli is that it is a windows tool... so you're tied to the Windows Env in this case.  But you could easily repurpose the methodology here to work with Linux using SIPP instead.

Using Grails, I created an application with:
grails create-app dialer
then I cd'd into the dialer directory and created the main controller:
grails create-controller csvImport
Once done I opened up my IDE and imported the Grails project (I use Grails Tools Suite.)

View

In the View I did this to capture the user's CSV file:
        <div id="status" role="complementary">
            <h1>Upload your CSV file of phone numbers</h1>
            <g:form controller="CsvImport" method="post" action="save"
                enctype ="multipart/form-data">

                   E-mail Results to a @mydomain.com address:<br>
                <g:textField type="field" name="emailAdd" value="brian@mydomain.com" style='width: 500px;' required=""/><p/>
                <input type="file" name="myfile" required/>
                <g:actionSubmit value="Start Process" action="save"/>
            </g:form>
        </div>


The controller looks like this:
   def save() {
        def emailTo = params.emailAdd
           if (emailTo =~ /@mydomain.com/) {
        def csv = request.getFile('myfile').inputStream.text
       
        def data = parseCsv(csv)
        runAsync{
            for(line in data) {
            println "Trying... $line.Phone"
            def dialNum = "sipcli/sipcli.exe $line.Phone -d **.**.*.*** [masked IP of our proxy] -o 4 -t \"This is a test. this is a test. this is a test. this is a test\"-l 3".execute()
            def outFile = new File("grails-app/test.txt")
            if (dialNum.text =~ /success/){               
                outFile << ("PASS on number $line.Phone\r\n")

            } else {
                System.getProperty("line.separator")
                outFile.append("FAIL on number $line.Phone\r\n")

            }
            }
 
            sendMail {
                multipart true
                to "${emailTo}"
                from "SOMEONE@ADDRESS.COM"
                subject "Provisioning Report"
                body 'Please find the attached Provisioning Report...'
                attachBytes 'grails-app/test.txt','text/csv', new File('grails-app/test.txt').readBytes()
            }

} else {
   render "Error: Email does not conform to @mydomain.com" }
            }
           
         }


To explain how it works...
The view is pretty easy to understand, it just takes a file and passes it to the controller.

Controller

The controller isn't doing any special validation, since this is a internal tool.  It takes the CSV and expects to have a column header called "Phone."  Phone will have a list of DNIS (or numbers) that need to be called.


CSV Parsing

To parse the CSV I import:
@Grab('com.xlson.groovycsv:groovycsv:1.0')
import static com.xlson.groovycsv.CsvParser.parseCsv

Then use this call def csv = request.getFile('myfile').inputStream.text to grab the input and finally pass it to the parser:
def data = parseCsv(csv)

Running Asynchronously

I don't want the  user to wait 30min for 500 phone numbers to be dialed... so instead, I send them immediately to a  page... this is handled with runAsync... this is a plugin called executor.  The Grails Executor plugin will run a closure asynchronously so you can do other stuff while the longer method runs.

Inside the runAsync closure is this for loop and if statement:
            for(line in data) {
            println "Trying... $line.Phone"
            def dialNum = "sipcli/sipcli.exe $line.Phone -d **.**.*.*** [masked IP of our proxy] -o 4 -t \"This is a test. this is a test. this is a test. this is a test\"-l 3".execute()
            def outFile = new File("grails-app/test.txt")
            if (dialNum.text =~ /success/){               
                outFile << ("PASS on number $line.Phone\r\n")
               
               
            } else {
                System.getProperty("line.separator")
                outFile.append("FAIL on number $line.Phone\r\n")

            }


Basically it's doing this:

For each line in the csv file, it prints out the phone number it's trying, and then I've defined an action to run the sipcli sip client.... to call that same number.  SipCli is a awesome command line sip tool for windows that can be used to find sip problems and issues.  It's very light and easy to use.  In this case I have it set to a 4 second timeout and i'm telling it to read the text, "This is a test..." when it makes the phone connection.

Assertions

The assertion of whether or not the phone call is valid is via the regex i'm doing in the if statement... If dialNum.text has success then we output to a file "Pass on number [DNIS]"
However, if the number fails to connect, we append to the same file "FAIL on number [DNIS]"

E-Mail

Finally, at the end, outside the runAsync closure, I do a call to email the results (that flat file) to a recipient, using the Grails mail plugin.  I'm passing the To value from the form... and as you may have seen, I'm forcing the tool to only work if an internal email is passed through.  The domain "mydomain.com" is a filler for the real domain I check. Since the tool uses an internal SMTP server, we can't send to outside emails... so I dont want the user to violate the SMTP capabilities.  Their email address entered (if from the right domain) is added to the To line below:
             sendMail {
                multipart true
                to "${emailTo}"
                from "SOMEONE@ADDRESS.COM"
                subject "Provisioning Report"
                body 'Please find the attached Provisioning Report...'
                attachBytes 'grails-app/test.txt','text/csv', new File('grails-app/test.txt').readBytes()
            }


That's it.  

19 June 2013

Grails Project: Using Grails as a UI Wrapper to a CLI like SIPP



For many people this project is going to be Easy.  It's very simple.

The idea for this came from a situation at work.

I am a sole user of a command line SIP/VOIP tool called SIPP.  The tool is incredibly complex. It took me many weeks to get it up and running, but I know it well now and use it often. 

However, other members of the team could also benefit form the use of the tool, but some members of the office may not have a computer background, or SSH access to run the command line tool itself.

You may have command line tools you'd like to offload to others, but don't want them to make mistakes... so maybe using a GUI to wrap up the commands and constrain the options is useful for you as well.

To that end, I came up with an idea to play with a bit of Grails and get a Web Application stood up, that would power the command line tool itself.

The Setup (I wont cover these aspects):

First, in my case the tool (SIPP) needs to be installed
Second, Grails needs to be installed on the same environment

What I'll cover:

Building the Grails app
Making the calls to run the command line tool
Making the calls run in the background using Executor
Daemonize the app so it can run/stop and start on restart.

Pulling in server stats via Cacti will be covered in a different post.

Building the Grails app

This is pretty simple. There's not much difficulty in this task. 
So first things first... we need to know the command line tool parameters.  In my case I'm using SIPP.  The most common parameters I use are:
sipp -s [DNIS] [PROXY] -r [CALLS PER SECOND] -m [MAX CALLS] -sf [SCENARIO FILE] -d [DELAY OF THE CALL]

So I'll need to make a application that has a form with user selects for those items.  The PROXY in my case is going to be hard coded, as I don't want people accidentally sending traffic to the wrong IP/Host. 

In my case I only foresaw the need for one controller and a couple Views.  If you're not familiar with MVC, you might want to read up on it a bit.  Basically the Controller will handle logic, the View will be the display/rendering aspect of the app and the Model is the data handling.

So I'm going to make a super easy app that has one controller, and some views.

So lets start:
  1. Make the grails app.  Go to a folder of your choice, and type grails create-app  At the prompt for a name, give it a name and hit enter.
  2. It will now make a folder with the name of the Grails App that you just created.  Go ahead and enter the folder - cd [APP NAME]
  3. In the app folder create a controller with the command grails create-controller At the prompt for a name, give it a name and hit enter.
Let's launch an IDE to edit this... I  use Spring's Grails Tool Suite.  I love Intellij, but the community (free) edition doesn't currently support Grails.  Only the paid ($500+) version does.

Import the project you just made.  In Grails Tool Suite it's File / Import / Project - Grails.

Once imported, lets edit the controller.


In the Project Tree in your IDE you'll see [web-app name]/controllers/[project-name]/[your-controller]

Go ahead and open that up in the IDE.

You'll have some code with a index action like
class SippcallController {
     def index() {

                    }
  }

I made a action that was more useful for what I'm doing.  So I have:
def callLogic() {
    def sipp = "sipp -s 18008888888 10.98.1.1 -r 1 -m 1 -sf uac.xml -d 1200".execute()
  }

Initially I just put in the actual command to dial a number via SIPP.  Then I tested it... it worked... so at that point I replaced all the parameters with values that will be coming from a user Form.  So the action became:

def callLogic() {
    def sipp = "sipp -s ${params.phoneNumber} 10.98.1.1 -r $(params.Rate} -m ${params.Max} -sf ${params.scenario} -d ${params.delay}".execute()

[view_darta:sipp.text]
  }

That last bit, [view_data:sipp} is a model I can access in a View or web page.  In a webpage of this app, I can just have ${view_data} and it will reference the output of the SIPP sip call itself.


Creating the Web Form for User Input


Now lets go ahead and edit the index.gsp file.  It's listed in the Project Explorer as: Project Name / Views /

Once that's open, you'll notice the basic sample page that Grails creates with each new application.  I removed all the Grails code within the Body Tags and replaced it with a form.... using g tags.

My form is like this ( the Grails tags are highlighted in green):
        <g:form name="callForm" controller="makeCall" action="callLogic">
        <p><h3>Phone number:</h3></p>
        <g:select name="phoneNumber" from = "${['12132830920', '12132830912', '12132830591', '13237549121'] }" value="12132830920" noSelection="['':'-Choose the phone number -']"/>
        <p><h3>Calls Per Second:</h3></p>
        <g:select name="cps" from="${1..30}" value="1"/>
        <p><h3>Max Calls:</h3></p>
        <g:select name="maxCount" from="${['1', '10', '50', '100', '500', '1000', '2000']}" value="1"/>
        <p><h3>How Long to Hold Audio Open (ms):</h3></p>
        <g:select name="delay" from="${['1200', '5000', '10000', '20000', '60000', '120000']}" value="1000"/>
          <p><h3>Scenario:</h3> </p>
          <g:select name="scenario" from ="${['uac.xml', 'codec_speex.xml', 'codec_g729.xml', 'carrier_sprint.xml'] }"
            value = "uac.xml" />

        <g:actionSubmit value="Place Call" action="callLogic"/>
        </g:form>

It's pretty easily readable. A few things to note:
  • To handle multiple selections in a dropdown, you can use a function or just list them out like I did here.  A function is superior, but this was quick and easy to set up a test.  you make the g tag a select and you add from with "${['value1', 'value2']}" and so forth.  
  • Like a regular form, you can set the default value with value="value1"  
  • Make sure the form references your controller and action correctly.
 If done right, this should send the field data to your Controller and Action correctly... which would run the command line with the parameters supplied via this form. 

Creating an Output View

 

In my case I decided to make another page that would handle the output form the controller.  It's the same name as the controller.  So in the [project]/views/[subfolder]/callLogic.gsp page I will put some data from the SIPP tool....

I could add ${view_data}, and it would display the output from the SIPP test into the page itself.

In my case, as you can see in the screenshot, I'm pulling in server stats via Cacti.  It's too much to go into in this blog post, but I set up Cacti on the linux VM here that I'm using, to pull snmp data from the box I'm driving the SIPP load to.

When I first did this, the page loaded fine, but it would wait for SIPP to finish, then output the data to the view.

I now send the call to a background job using the Executor plugin for grails, and using the runAsync closure around the SIPP call in the controller.  This way once a user submits the form they load on the results page.  The data is still being collected and will need to be updated later onto the page (I haven't worked that part out yet.)

Installing Executor (to run background jobs)

In case you need this, here's how it works... you do a grails install-plugin executor
After it installs, you can then see the plugin listed in your IDE's project tree like so: project/plugins.  You can now use the methods the plugin gives access to.

In my case, I just wanted the runAsync method.  so back in my controller I added it like this:

def callLogic() {
    def sipp = "sipp -s ${params.phoneNumber} 10.98.1.1 -r $(params.Rate} -m ${params.Max} -sf ${params.scenario} -d ${params.delay}".execute()

runAsync{
[view_darta:sipp.text]
}
  }

This allows the output to be collected in the background. 

The part I haven't worked out yet, is getting a push or polling mechanism set up to get that data once it's ready.

Daemonizing It
I wanted to be able to start and stop this thing, so I set up a job in the /etc/init.d/ folder and called it sippgui  in that file is a series of commands, such as:

start() {
        cd /sippcall

        /sippcall/grails run-app -Dserver.port=8090 &
        sudo touch /var/lock/subsys/sippgui
        echo

}


You'll want to do something similar to stop it.

At that point it can be added to cron, or run manually with /etc/init.d/sippgui start|stop|restart, etc.

Btw. the -Dserver.port=8090 is how i'm setting the port it's using. 

Opening the Port

That's it... Oh WAIT one more thing...

If you can't get this App to load, you may need to open your firewall to allow traffic to this port (i.e. 8090.)  In my case I couldn't  use 8080, so I used a non standard port. I had to modify the IPTABLES to allow this. 

CENTOS uses IPTABLES so that's my solution. I won't go into details on it, just know that you'll need to open any non standard port and may need to google how to do that on whatever OS you're using. 


02 June 2013

Setting up Raspberry Pi

This is a departure from the main crux of this blog.  But I wanted to gather some notes on my pitfalls I encountered with setting up a Raspberry Pi device.

Hardware and SD Card Set Up


I had the version 2 B 512 version of the Pi.
I used a PNY 16GB class 6 SD card
I used a solar/electric battery to power it
I used a usb keyboard
and a standard hdmi cable

I started this all off by downloading the official Raspberry "wheezy" distro (2013-05-25-wheezy-raspbian) from the direct download link.  However, I couldn't unzip it on windows... although the download was the correct size, windows' default unarcher gave me a "can't read" this file error.  However, 7zip would read it and uncompress it... I figured this was just a problem with Windows.

Butut when I used the img file that it produced to drop on the SD card... it corrupted the SD card.

Again - this was the direct download official link of the "wheezy" distro.  

At this point he SD card would read "18mb left of 56mb" yet this was a 16Gig card.  I freaked out a bit thinking the card was now damaged.  However I read that it only needs to be reformatted....

I tried to reformat using Windows' reformat tool (right click / format)... but the card would always read "56mb avail."

I in fact had to now download the official SD formatter... ugh.  Ok, Now that got me back to 16gigs.

I tried multiple downloads from that direct download link - all with the same problem!

Then I tried the torrent.

They recommend the torrent, but I hate using torrents.  I always end up with spyware from the torrent clients... anyway, I installed bit torrent and downloaded the distro via the torrent, and then uninstalled bit torrent afterwards.

I did get the torrent to download, it was about 10x faster then the direct link.  I also was able to unzip it fine without the error's encountered from the direct link.

I put the SD card back in the Pi, and plugged it back in... It booted up great.

Configuration

When it boots up successfully you'll see something resembling the old PC Bios menu.  This is how you'll configure your Raspberry Pi.

At the configuration screen, you'll want to:
1. set the thing to use the entire space on the SD card, it's the first option in this boot menu.
2. after that, if you want to boot to a desktop, go to that option, if you use a console only, skip it.
3. To turn on SSH (so you can remote into the box via SSH) you need to go to advanced settings and then choose SSH and then Enable.
4. If this is a public facing device, you might want to change the password as well - it's a option on the boot menu.
5. I also changed the hostname (it defaults to 'raspberrypi')

After that I chose to finish and reboot.

From here on out, you'll log in as your user "pi" with the password you picked... or use the default (raspberry.)

SSH

To SSH to this box, first get it's IP by running ifconfig from the command line.  You'll see "eth0" and in there a line inet addr:.  take that address.

From another box, using either terminal on mac, or Putty/supper putty on PC, you can just ssh over to:
ssh [ip]
login as user pi and input your pass.

Installing Software from the Command Line

If you're used to installers in Linux (like Centos' YUM) then you'll get the same thing here, but it's apt-get.  You'll sudo each command like this.  So: sudo apt-get iw tshark would install iw and tshark (for packet sniffing.)

Find the tools you want and they should give you the proper apt-get install commands.

Shutting Down the Raspberry Pi


Also another important thing.  The official site for Raspberry Pi, says to shut it down you just "unplug it" from what I've read this could damage the SD card.  So... you must run an official shutdown:

sudo shutdown -h now
From the UI, you click the red icon and choose shutdown.  Wait for the proper screen statement "system halted," and then remove the power cable.

16 May 2013

Simulating Real World Latency during Automation

So why  not use Jmeter to run performance tests?

The main problem with Jmeter is that it doesn't give a good assessment of front end performance.  It can send load and give back response times in data retrieval, URI end point response times, page load, etc.

But... it's harder to see how long it takes a AJAX menu to pop up after login, or for a "one page site" that loads in all the data asynchronously, to determine the performance hit per browser of loading dynamic HTML 5 elements.  

For this reason, I came up with a process of simulating and reporting the latency during browser automation tests.

I approached this by:
  •     Recording the time between a submit/save/delete and the next rendered screen or alert
  •     Setting up a Latency Generator
  •     Configuring the Automation Framework with Jenkins to dynamically kick off latency, launch the automation, capture the results to a report.

Recording Latency In The Tests

First I approached the idea of how to capture the time it takes to perform an action.  I had hoped there was a Cucumber gem to do this... Those I found really didn't help me much.  They handled a different set of problems. I wanted to know how long it would take from the click of a button, to the load of the next screen.

I realized I needed to write my own wrapper.

It's really very simple.  I define a class variable to be the current time, like:
@start = Time.now

This is run right before the action... for example:
@start = Time.now
@browser.div(:id=>"submit").click

Then I add in a "wait until" in Watir (Wait For in Groovy/geb) like this:
Watir::Wait.until { @browser.alert.exists? }

In the above example it's waiting for an alert window to appear.  If your next screen had a field or button instead, you'd do the wait until, for that element to be loaded.

Just after the Wait Until, I add the end of the timer:
@end = Time.now - @start

with a output to the display:
p "It took #{end} seconds to load XYZ screen."

While it may not be perfect down to the ms, it is quite useful in judging latencies and changes with different bandwidth connections.

I can see that I get a general time of 0.83 seconds on some submit action.  Then change the bandwidth, and rerun and see I am now at 2 seconds on average.

I have since modified this to now output the performance results to a CSV file.  More details on that in a later post.

Setting Up Netem

Second, I added in some throttling of bandwidth and making use of latency and packet loss.  This is handled via a linux tool called Netem.  Netem allows me to generate latency, packet loss and bandwidth throttling.  For example, if you wanted to discover how long it took to go from clicking "submit" to the rendered "dashboard" on a DSL line Or from a visitor from Europe.  Or how long it would take from clicking save, to getting the save confirmation for a user in Asia.

Netem requires the use of Linux. So I choose to set up a special Linux VM that would run Netem.... and set up a Proxy on that VM, so that the browser could connect to it for bandwidth simulation.

Setting Up the VM

I had to set up a Linux VM to be a proxy. I used Squid to be the proxy server.  Squid defaults to using port 3128.  You can follow many tutorials online on setting up squid (like http://www.cyberciti.biz/tips/howto-rhel-centos-fedora-squid-installation-configuration.html)

Then I modified iptables (i.e. sudo vi /etc/sysconfig/iptables) and added a INPUT for port 3128.

Afterwards I restarted iptables.

To test it, I configured the browser's proxy manually to my Proxy and port.  Then ramped up the latency to crazy amounts on the VM and verified that the browser performance degraded.

Netem Commands

I took various target markets for the company I work with (Asia, Europe and US) and gathered some latency reports from the IT dept.

I also took some target customer bandwidth profiles (10Mbit, 768k, 128k, etc.)

Last, I grabed some concept of what we might see for packet loss in the real world.

Here's some examples:

US traffic simulation:

   tc qdisc add dev eth0 root netem delay 80ms 10ms

Asia traffic simulation:

   tc qdisc add dev eth0 root netem delay 160ms 70ms

Bandwidth throttling:

    tc qdisc add dev eth1 root handle 1:0 tbf rate 200kbit buffer 1600 limit 3000
    tc qdisc add dev eth1 root handle 1: cbq avpkt 1000 bandwidth 10Mbit

    Simulate 768k down and 128k up:

    tc qdisc replace dev eth0 root handle 1:0 tbf rate 768kbit burst 2048 latency 100ms
    tc qdisc replace dev eth1 root handle 2:0 tbf rate 128kbit burst 2048 latency 100ms

Kill Netem:

To end any Netem protocols running, I use:
tc qdisc del dev eth0 root

Automating Netem as part of Cucumber Automation

First, I made a change in the env.rb file within the features folder.  In that file, in the begin block, I added the highlighted part:

def environment
  (ENV['ENVI'] ||= 'proxy').downcase.to_sym
end

Before do  |scenario|
  p "Starting #{scenario}"
  if environment == :int
    @browser = Watir::Browser.new(:remote, :url=>"http://[my qa selenium grid server]:4444/wd/hub", :desired_capabilities=> browser_name)
    @browser.goto "http://[my integration test env]:8080"
  elsif environment == :local
    @browser = Watir::Browser.new browser_name
    @browser.goto "http://[my integration test env]:8080"
  elsif environment == :proxy
    profile = Selenium::WebDriver::Firefox::Profile.new
    proxy = Selenium::WebDriver::Proxy.new(:http => "[my centos VM running netem and squid goes here]:3128")
    profile.proxy = proxy
    driver = Selenium::WebDriver.for :firefox, :profile => profile
    @browser = Watir::Browser.new(driver)
    @browser.goto "http://[my integration test env]:8080"

  else
    @browser = Watir::Browser.new browser_name
    @browser.goto "http://[alternate test env]:8080"
  end
end

So now, when I send the command: Cucumber features/performance_test_Asia.feature ENVI=proxy
it will kick off cucumber, launch Firefox - which will use the proxy we set up on the Linux VM.  that Linux VM, if using Netem to trigger expected latency from Asia - will simulate real world response times.

Configuring Jenkins

Now that the automation works, we want to turn Netem latency on before a test, and off after a test.  The bet way to do this, that I've found is to use Jenkins.

Jenkins configuration on the Netem VM

In my case I have that Linux VM with the proxy and netem. I put Jenkins on it.  I created jobs pertaining to netem.  Jobs like:
1. Start Netem with Latency for Asia (70ms-140ms)
2. Start Netem with Latency for US-Michigan (10ms-80ms)
3. Start Netem with Latency for Europe (70ms - 100ms)
4. Start Netem with 0.3 percent packet loss
5. Stop Netem

Each job is just running a shell script on that linux VM.  I.e. a stand alone job to simply run
sudo tc qdisc add dev eth0 root netem delay 100ms 70ms
for example.

The stop Netem job, simply runs:
 sudo tc qdisc del dev eth0 root


In my case, I needed to use the ssh plugin for Jenkins. This allows me to sudo commands. So although i'm not ssh'ing anywhere, the SSH plugin allows me to authenticate on the same box with the account that has sudo privledges.  I have more on setting that up in a separate blog.

At any rate, here's what you do next.... you go to a job, like "Start Netem with Latency for Asia..." in the job details, right click the build link and copy out the URL.  Paste that build URL for each job.

These URL's will look something like: 
http://[Your Jenkins]:8080/view/[Your Project]/job/Stop%20Netem/build?delay=0sec

Jenkins configuration on the Automation Environment

Step 1

Back at the main automation suite (hopefully you have jenkins running jobs there - if not, you'll need to set Jenkins up), create a job.  This will be a free standing job. All it will do is call a script to hit those URL's you copied.

How do you do that? Well you could curl it, if your automation jenkins is on a linux env.  Or you could do wget... or... you could script it in ruby/python/groovy... or in my case, I use Jmeter.  I simply have a Jmeter script that hits that URL.

So to recap:
In the automation environment, I have a Jenkins job that is a parent job. It starts Netem, by calling a Jmeter script to hit the appropriate Netem URL on the Linux VM.  As long as this environment can talk to the linux proxy server, this will work fine.

Step 2

Next I create another job in the automation Jenkins. This job will execute the Cucumber script.  It is a simple stand alone job that runs the command line:
cucumber ENVI=proxy features/my_asia_performance.feature

Step 3

After creating that Cucumber job, I link it as a child to the job we made in step 1.  In this way, Step 1, sends a command to turn on Netem on the Linux VM.  Then that job is configured (in the configure of the job) to launch a project when it's finished. That project will be the automation job we made in step 2.

Step 4

After setting the cucumber automation as a child project of the Netem job in step 1, we will now make the last job.  This will again be a stand alone job, it will simply stop Netem.  It will be a child of the Cucumber job/project.

So create a new Jenkins job, and have it curl/wget or use a script to hit the URL to the linux Jenkins that will launch the Netem kill job.

Step 5.

Edit the Cucumber job, so that after it finishes, it will launch a project - this being the Netem stop/kill job you created in Step 4.

All done.

To run it, simply run the first job in step 1.

That job will turn Netem on, setting the appropriate latency - then when that job finishes it will automatically launch the browser automation job (which is recording the response times) - then after that job finishes, it will launch the job to turn off the Netem latency.

Pipping Out the Output to CSV

Most people don't want to read log data to find response times.  I got a request to keep this data I was gathering in a common flat file.  So I decided to go with CSV.

 What I did was use the native CSV class in Ruby to handle this:
  CSV.open("C:\\#{locale}_{$TestTime}.csv", "ab") do |csv|
    csv << ["Save Action","#{@CF_save_end}", Time.now]
  end

I'll go into more detail on this in my next blog post.

22 April 2013

Automated Verification of VOIP Audio


I've created a Presentation that goes over these points as well:
http://prezi.com/-29ebxieb4ek/copy-of-rtp-re-assembly/

*UPDATE*
I found this awesome work, using google's translate api, to transcode the audio to text:
http://cheateinstein.com/category-shell/using-google-voice-api-to-transcribe-audio/

I've now used this at the final end of the process, to verify the text heard is what is expected!!

I've been working on this VOIP/SIP automation framework for a few months now. I started with a Cucumber framework, and then added on with some VOIP/SIP specific tools like SIPP and SIPCLI.

I got to where the test harness' I built with these tools, would use Jenkins to push button (or on a schedule or build commit) drive traffic to a phone number... verify it reached it by acknowledgements sent back.  But what if the phone number was going to the wrong destination, and sent back acknowledgements?

At that point I used TollFreeForwarding.com's technology to set a email alert as an endpoint on a phone number. For example, you call: 888-888-8888 and you get an IVR. you press 1, and are sent to a voicemail - you pass in audio and hang up. Then TollFreeForwarding.com emails the configured email on the account, the recording.

It was better, but it required a voicemail to email application at every end point. It also doesn't verify that audio actually occurred on the call. What if no audio played back? Or there was significant jitter to not understand it?

To further this testing, I started thinking of recording the call and using some sort of analysis of the recording to verify it's what was expected.  

This is my first draft at answering that need.  It can be improved.  But it's a step in the right direction.

What I'm doing

  1. This automation dials a number, with a known IVR or greeting.  
  2. It does a packet capture during the recording
  3. It filters out the RTP channels from the packet capture and then creates a wav out of the pcap file.
  4. Once there is a wav file, it runs diagnostics on it... generating some visual graphs like the image on this blog... but more importantly (and more useful) it generates audio information that I use as a footprint for the audio playback.
  5. This audio is also sent to google who transcribes it and sends me back the text which is compared to the expected string.

Tools used

  1. sipp to drive an automated command line sip call
  2. tshark (command line version of wireshark)
  3. jenkins (for the GUI to drive and schedule these tests)
  4. sox (linux based audio conversion and analysis tool)
  5. some shell scripting

How it Works

The test has a parent job, that kicks off two sub jobs.  These sub jobs run simultaneously.  One does a phone call to a phone number with a recording Greeting/IVR.  The other job runs a shell script that maintains the test itself.  The second job uses tshark to record the packets and filter the rtp, then uses sox to convert the raw audio to a wav and do some analysis on the wav.

The Shell Script

First I set tshark to record for a specific duration, that I think will encompass the call:
tshark -a duration:20 -w /jenkins/userContent/sip_1call.pcap

I assign a variable to a tsark task to scan the RTP packets and find the hex value for the RTP packets (I learned these three parts from a online tutorial, but lost the bookmark):
ssrc=$(tshark -n -r /jenkins/userContent/sip_1call.pcap -R rtp -T fields -e rtp.ssrc -Eseparator=, | sort -u | awk 'FNR ==1 {print}')

The above would return a hex value like:
0x344292302

Which is followed by:
sudo tshark -n -r /jenkins/userContent/sip_1call.pcap -R rtp -R "rtp.ssrc == $ssrc" -T fields -e rtp.payload | tee payloads

The above looks for that Hex value captured previously, and holds that as a variable, payload.

Finally, we have a for statement in the shell script to convert the payload value from above, to a raw audio file:
for payload in `cat payloads`; do IFS=:; for byte in $payload; do printf "\\x$byte" >> /jenkins/userContent/sip_1call.raw; done; done

At this point I had a raw audio file. I found a linux tool called sox that was  a good fit for this conversion... so I installed it and added these lines into my script...
Sox is then invoked to convert the raw audio to a wav:
sox -t raw -r 8000 -v 4 -c 1 -U /jenkins/userContent/sip_1call.raw /jenkins/userContent/sip_1call.wav


Then I run a couple more Sox commands:
This one creates stats, which Jenkins captures in the log file of the test run:
sox /var/lib/jenkins/userContent/sip_audio_1call.wav -n stat

The stats generated will look like this:
Samples read:             15680
Length (seconds):      1.960000
Scaled by:         2147483647.0
Maximum amplitude:     0.425659
Minimum amplitude:    -0.285034
Midline amplitude:     0.070313
Mean    norm:          0.043354
Mean    amplitude:    -0.000055
RMS     amplitude:     0.070984
Maximum delta:         0.243896
Minimum delta:         0.000000
Mean    delta:         0.019919
RMS     delta:         0.034190
Rough   frequency:          613
Volume adjustment:        2.349
 
The two highlighted values seem to be consistent with the same audio.  At this point, that's what the test assertion is based on.  I have a better plan in the works for a future upgrade to this test.  But for now, I'm using the rough frequency and max amplitude to determine the pass / fail criteria.

Is it perfect? No. It's potential for false negatives. The rough frequency *could* change, but so far it hasn't for the same audio I expect.

Spectograms
If your into spectrogram's (and who isn't?), then sox will also output one if you like, I end the shell script with this:

sox /jenkins/userContent/sip_1call.wav -n spectrogram -y 2 -l -o /jenkins/userContent/sip_1call.png

If anyone has any other tools that can pull out more data, please let me know.

The Upshot?

One shell script, called by Jenkins, running 3 tools gets this job done.

Verify Audio via Speech To Text

A few people approached me and mentioned rough frequency may not remain constant as the test call goes through different hops.  So I began to investigate this some more... I found this guy:
http://cheateinstein.com/category-shell/using-google-voice-api-to-transcribe-audio/

he had created a way to use a shell script to send audio files to google for transcription.

I modified his script a little to work for my needs, and added a text assertion.  If the text fails comparison then I exit the script with a error code, which forces jenkins to regard this as a total failure.

Here's the part I added to the bottom of my previous script:
echo "1 - Translate with SOX - Convert WAV to FLAC with 16000"
sox /jenkins/userContent/sip_audio_1call.wav input.flac rate 16k
echo "2 - Submit to Google Voice API"
wget -q -U "Mozilla/5.0" --post-file input.flac --header="Content-Type: audio/x-flac; rate=16000" -O - "http://www.google.com/speech-api/v1/recognize?lang=en-us&client=chromium" > output.ret
echo "3 - Extract recognized text"

cat output.ret | sed 's/.*utterance":"//' | sed 's/","confidence.*//' > output.txt
echo "4 - Display text"
a=`cat output.txt`
echo $a
b="tollfreeforwarding.com"
if [ "$a" = "tollfreeforwarding.com" ];
then
        echo "Verified audio is tollfreeforwarding.com"
else
        echo "FAIL audio is not tollfreeforwarding.com"
        exit 666


fi;


In my scenario, I've seeded the phone greeting on the number that is called to be an announcement audio that says, "Toll Free Forwarding Dot Com"  which google turns correctly to "tollfreeforwarding.com" and I validate against that.

19 April 2013

Converting RDP in pcap to Audio Wav files

After following a lot of different tutorials (some of which worked some of which didn't), I came up with a shell script using a couple tools to scrape a packet capture file, pull out the rdp packets, and then convert them back into audio.

For me this will be useful in automated testing.  I currently drive automated SIP calls via SIPCLI and ruby for a variety of tests at work.  But how do I know I get the right end point?  In the past, I'd have the phone number I dial, record voice mail and send me an email, and the sipcli client would send over text to speech audio.

But I dont always have the luxury of being able to configure the phone number to voice mail. 

I've been wanting to do a packet capture during the test and convert it back to audio afterwards, then do a wav comparison on the expected audio vs. the captured audio.

Tools used:

tshark
sox

These are both linux tools. 
tshark is a command line version of wireshark.  It's installed on centos boxes using yum install wireshark-gnome.
sox via yum install sox

Sox is a audio analysis tool that is run from the command line. 

Test Script:

After looking at some examples online of different tools, I pieced this together from other people's examples, with a few modifications. It seems to work for me:

Contents of pcap_to_wav.sh:


ssrc=$(sudo tshark -n -r capture.pcap -R rtp -T fields -e rtp.ssrc -Eseparator=, | sort -u)

echo $ssrc

sudo tshark -n -r capture.pcap -R rtp -R "rtp.ssrc == $ssrc" -T fields -e rtp.payload | tee payloads

for payload in `cat payloads`; do IFS=:; for byte in $payload; do printf "\\x$byte" >> sound.raw; done; done

echo 'sox has converted pcap to wav file'
sudo sox -t raw -r 8000 -c 1 -U sound.raw capture3d.wav

That's it!

basically if you have sudo access, you can run this and it will take the pcap and find the rdp packets, then make that a raw audio file... sox is then used to convert the raw file to a wav file.

At this point, you can further use sox to compare one wav to another wav.

08 April 2013

Selenium Grid Up and Running with Cucumber Automation

Setting up Selenium Grid with Cucumber


This is more of an advanced topic I think. I did a lot of research to come up with the solution I use here at TollFreeForwarding.com

I have tests that would take hours to run sequentially.  To bring this back to normalcy, I use Selenium Grid to farm the jobs to multiple VM's simultaneously.  This way the total time for all tests to complete is the longest test I have (5min.)

To get to this point, the Cucumber tests have to be modular.  They have to be broken up. You can't have one giant cucumber test.  Selenium Grid can't take one giant test and farm out each Scenario.  Instead you have to have multiple features.

For example:
If you  have a UI you automate and you have coverage for areas like -
  • Account Creation
  • Profile CRUD actions
  • Forum Post CRUD actions
  • Calendar Schedules
  • Call Customer Support (WebPhone)
  • Admin: Create Menus for Customers
  • Admin: Make new announcements for Customers
  • Admin: CRUD actions on gallery uploads/edits/deletes
Then I'd make each of these it's own feature. 

Imagine each of the eight features above took 5 min to complete.  That's a total of 40min if they are run sequentially.  Meaning you'd start the tests and come back in 40min.  What if you could get that down to 5 min?

You can!

Run them all at the same time, across multiple VMs.  That way they all run in parallel and finish in 5min.

This is where Selenium Grid comes in.

Install the Grid

On the Main VM that will run this job, you will install the Selenium Grid Hub.  This is rather simple to do.  You basically have to have java installed, and you run a command like:
[path to your selenium server standalone jar]/selenium-server-standalone-2.31.0.jar -role hub

I found our VM's might sometimes restart (IT restarts and what not) so I added a batch file on this Windows VM and use the Windows Scheduler to assign a new task of "on restart run this batch file" the batch file has this content:
@echo off
"C:\Program Files (x86)\Java\jdk1.7.0_17\bin\java.exe" -jar "C:\Selenium-grid\selenium-server-standalone-2.31.0.jar" -role hub

Which tells the server "run Java to execute the Selenium server stand alone with the parameter role of "hub."

Node VM's must be configured

On each VM that will be used to get jobs from the Grid hub (referred to as 'nodes'), you will need to set them up.  These boxes/VM's again need to have JAVA installed and need to have the same selenium-server-standalone jar.  But it will be run with the role 'node.'

Here's an example:
java -jar selenium-server-standalone-2.31.0.jar" -role node -hub http://qa1.ifn.com:4444/grid/register -browser browserName=chrome,maxInstances=5

Similar to the Hub, I also created a batch file and used the windows scheduler to make sure that on a restart the batch file executes... it has this code:

C:\Program Files (x86)\Java\jre7\bin\java.exe" -jar "C:\Selenium-grid\selenium-server-standalone-2.31.0.jar" -role node -hub http://mydomain.com:4444/grid/register -browser browserName=chrome,maxInstances=5

Note that http://mydomain.com:4444 is the domain that the hub is running on.  This command is registering the browser Chrome and saying it has 5 max instances (this means it will run up to 5 chrome tests simultaneously on this box.)  you can of course change that.


Setting Up Cucumber

Since I use Jenkins to run all the Cucumber jobs, I want to be able to specify parameters from a command line that will:
  • Run the Cucumber Features
  • Specify the Browser for the test
  • Specify if the grid will be used or not
To do that, I use the env.rb file in Cucumber's /Features/support  directory.

Here's an example of what I did to create these parameters to be used from the command line....
First I added some code:

def browser_name
  (ENV['BROWSER'] ||= 'firefox').downcase.to_sym
end

def environment
  (ENV['ENVI'] ||= 'int').downcase.to_sym
end

This is setting the parameter "browser" and "envi" (for environment) to be called on the command line.  It is also giving a default value for each.  By default the browser is Firefox and the environment is int (for integration.)

Below that code, I wrote this before statement:
Before do  |scenario|
  p "Starting #{scenario}"
  if environment == :int
    @browser = Watir::Browser.new(:remote, :url=>"http://mydomain.com:4444/wd/hub", :desired_capabilities=> browser_name)
    @browser.goto "http://integration.env.com:8080"
  elsif environment == :local
    @browser = Watir::Browser.new browser_name
    @browser.goto "http://integration.env.com:8080"
  end
end

This block above says that before we run the Cucumber scenarios, we must define a few things.  First if the environment is passed in as 'int' (our default) we will define @browser to be equal to Watir::Browser.new(:remote, :url=>"http://mydomain.com:4444/wd/hub", :desired_capabilities=>browser_name)

So if environment is int, we're defining the @browser to be going to selenium grid to send all the traffic.  If a browser is passed in, we're also sending that along. 

Test this out by sending a command from the project you have like:
cucumber envi=int browser=firefox

You can monitor the jobs get picked up by different vm's from the Grid console.

Jenkins Configuration

Once you have Selenium Grid set up, go to Jenkins and make a new basic job. 

This job will just do a Windows Shell command.

You'll want to use the same command line parameter that was working to test your test above. For example, if you have a feature called "Account Creation" you would do something like this:
cucumber features/account_creationl.feature BROWSER=firefox ENVI=int

If you did that for each job, you'd have all your jobs going to the grid.  You could make a parent job that runs all features simultaneously.

To do that you create a project that has only one function, to kick off downstream projects... each feature would be it's own project/job.  So the parent project would launch downstream jobs of: account creation, profile crud actions, forum crud actions, etc.  So they all run in parallel and are sent to the grid.

Gotcha's

There's a gotcha.  IE webdriver doesn't like to be used remotely.  For this one outlier, I use IE in a serial fashion.

So in my Jenkins I have my jobs on tabs.  The first time is Functional Tests (these run in Firefox), the second tab is Chrome Tests, the third is IE tests.

On the first tab I have the jobs configured to run with FF only.  On the second to run with Chrome only. The third tab runs in IE only AND does NOT SEND the jobs to the grid. 

This way, I get the Firefox functional tests finished in 5min, Chrome finished in 5 min and IE takes the normal time of 40min.  It's still better then 40min a pop.