19 June 2013

Grails Project: Using Grails as a UI Wrapper to a CLI like SIPP



For many people this project is going to be Easy.  It's very simple.

The idea for this came from a situation at work.

I am a sole user of a command line SIP/VOIP tool called SIPP.  The tool is incredibly complex. It took me many weeks to get it up and running, but I know it well now and use it often. 

However, other members of the team could also benefit form the use of the tool, but some members of the office may not have a computer background, or SSH access to run the command line tool itself.

You may have command line tools you'd like to offload to others, but don't want them to make mistakes... so maybe using a GUI to wrap up the commands and constrain the options is useful for you as well.

To that end, I came up with an idea to play with a bit of Grails and get a Web Application stood up, that would power the command line tool itself.

The Setup (I wont cover these aspects):

First, in my case the tool (SIPP) needs to be installed
Second, Grails needs to be installed on the same environment

What I'll cover:

Building the Grails app
Making the calls to run the command line tool
Making the calls run in the background using Executor
Daemonize the app so it can run/stop and start on restart.

Pulling in server stats via Cacti will be covered in a different post.

Building the Grails app

This is pretty simple. There's not much difficulty in this task. 
So first things first... we need to know the command line tool parameters.  In my case I'm using SIPP.  The most common parameters I use are:
sipp -s [DNIS] [PROXY] -r [CALLS PER SECOND] -m [MAX CALLS] -sf [SCENARIO FILE] -d [DELAY OF THE CALL]

So I'll need to make a application that has a form with user selects for those items.  The PROXY in my case is going to be hard coded, as I don't want people accidentally sending traffic to the wrong IP/Host. 

In my case I only foresaw the need for one controller and a couple Views.  If you're not familiar with MVC, you might want to read up on it a bit.  Basically the Controller will handle logic, the View will be the display/rendering aspect of the app and the Model is the data handling.

So I'm going to make a super easy app that has one controller, and some views.

So lets start:
  1. Make the grails app.  Go to a folder of your choice, and type grails create-app  At the prompt for a name, give it a name and hit enter.
  2. It will now make a folder with the name of the Grails App that you just created.  Go ahead and enter the folder - cd [APP NAME]
  3. In the app folder create a controller with the command grails create-controller At the prompt for a name, give it a name and hit enter.
Let's launch an IDE to edit this... I  use Spring's Grails Tool Suite.  I love Intellij, but the community (free) edition doesn't currently support Grails.  Only the paid ($500+) version does.

Import the project you just made.  In Grails Tool Suite it's File / Import / Project - Grails.

Once imported, lets edit the controller.


In the Project Tree in your IDE you'll see [web-app name]/controllers/[project-name]/[your-controller]

Go ahead and open that up in the IDE.

You'll have some code with a index action like
class SippcallController {
     def index() {

                    }
  }

I made a action that was more useful for what I'm doing.  So I have:
def callLogic() {
    def sipp = "sipp -s 18008888888 10.98.1.1 -r 1 -m 1 -sf uac.xml -d 1200".execute()
  }

Initially I just put in the actual command to dial a number via SIPP.  Then I tested it... it worked... so at that point I replaced all the parameters with values that will be coming from a user Form.  So the action became:

def callLogic() {
    def sipp = "sipp -s ${params.phoneNumber} 10.98.1.1 -r $(params.Rate} -m ${params.Max} -sf ${params.scenario} -d ${params.delay}".execute()

[view_darta:sipp.text]
  }

That last bit, [view_data:sipp} is a model I can access in a View or web page.  In a webpage of this app, I can just have ${view_data} and it will reference the output of the SIPP sip call itself.


Creating the Web Form for User Input


Now lets go ahead and edit the index.gsp file.  It's listed in the Project Explorer as: Project Name / Views /

Once that's open, you'll notice the basic sample page that Grails creates with each new application.  I removed all the Grails code within the Body Tags and replaced it with a form.... using g tags.

My form is like this ( the Grails tags are highlighted in green):
        <g:form name="callForm" controller="makeCall" action="callLogic">
        <p><h3>Phone number:</h3></p>
        <g:select name="phoneNumber" from = "${['12132830920', '12132830912', '12132830591', '13237549121'] }" value="12132830920" noSelection="['':'-Choose the phone number -']"/>
        <p><h3>Calls Per Second:</h3></p>
        <g:select name="cps" from="${1..30}" value="1"/>
        <p><h3>Max Calls:</h3></p>
        <g:select name="maxCount" from="${['1', '10', '50', '100', '500', '1000', '2000']}" value="1"/>
        <p><h3>How Long to Hold Audio Open (ms):</h3></p>
        <g:select name="delay" from="${['1200', '5000', '10000', '20000', '60000', '120000']}" value="1000"/>
          <p><h3>Scenario:</h3> </p>
          <g:select name="scenario" from ="${['uac.xml', 'codec_speex.xml', 'codec_g729.xml', 'carrier_sprint.xml'] }"
            value = "uac.xml" />

        <g:actionSubmit value="Place Call" action="callLogic"/>
        </g:form>

It's pretty easily readable. A few things to note:
  • To handle multiple selections in a dropdown, you can use a function or just list them out like I did here.  A function is superior, but this was quick and easy to set up a test.  you make the g tag a select and you add from with "${['value1', 'value2']}" and so forth.  
  • Like a regular form, you can set the default value with value="value1"  
  • Make sure the form references your controller and action correctly.
 If done right, this should send the field data to your Controller and Action correctly... which would run the command line with the parameters supplied via this form. 

Creating an Output View

 

In my case I decided to make another page that would handle the output form the controller.  It's the same name as the controller.  So in the [project]/views/[subfolder]/callLogic.gsp page I will put some data from the SIPP tool....

I could add ${view_data}, and it would display the output from the SIPP test into the page itself.

In my case, as you can see in the screenshot, I'm pulling in server stats via Cacti.  It's too much to go into in this blog post, but I set up Cacti on the linux VM here that I'm using, to pull snmp data from the box I'm driving the SIPP load to.

When I first did this, the page loaded fine, but it would wait for SIPP to finish, then output the data to the view.

I now send the call to a background job using the Executor plugin for grails, and using the runAsync closure around the SIPP call in the controller.  This way once a user submits the form they load on the results page.  The data is still being collected and will need to be updated later onto the page (I haven't worked that part out yet.)

Installing Executor (to run background jobs)

In case you need this, here's how it works... you do a grails install-plugin executor
After it installs, you can then see the plugin listed in your IDE's project tree like so: project/plugins.  You can now use the methods the plugin gives access to.

In my case, I just wanted the runAsync method.  so back in my controller I added it like this:

def callLogic() {
    def sipp = "sipp -s ${params.phoneNumber} 10.98.1.1 -r $(params.Rate} -m ${params.Max} -sf ${params.scenario} -d ${params.delay}".execute()

runAsync{
[view_darta:sipp.text]
}
  }

This allows the output to be collected in the background. 

The part I haven't worked out yet, is getting a push or polling mechanism set up to get that data once it's ready.

Daemonizing It
I wanted to be able to start and stop this thing, so I set up a job in the /etc/init.d/ folder and called it sippgui  in that file is a series of commands, such as:

start() {
        cd /sippcall

        /sippcall/grails run-app -Dserver.port=8090 &
        sudo touch /var/lock/subsys/sippgui
        echo

}


You'll want to do something similar to stop it.

At that point it can be added to cron, or run manually with /etc/init.d/sippgui start|stop|restart, etc.

Btw. the -Dserver.port=8090 is how i'm setting the port it's using. 

Opening the Port

That's it... Oh WAIT one more thing...

If you can't get this App to load, you may need to open your firewall to allow traffic to this port (i.e. 8090.)  In my case I couldn't  use 8080, so I used a non standard port. I had to modify the IPTABLES to allow this. 

CENTOS uses IPTABLES so that's my solution. I won't go into details on it, just know that you'll need to open any non standard port and may need to google how to do that on whatever OS you're using. 


02 June 2013

Setting up Raspberry Pi

This is a departure from the main crux of this blog.  But I wanted to gather some notes on my pitfalls I encountered with setting up a Raspberry Pi device.

Hardware and SD Card Set Up


I had the version 2 B 512 version of the Pi.
I used a PNY 16GB class 6 SD card
I used a solar/electric battery to power it
I used a usb keyboard
and a standard hdmi cable

I started this all off by downloading the official Raspberry "wheezy" distro (2013-05-25-wheezy-raspbian) from the direct download link.  However, I couldn't unzip it on windows... although the download was the correct size, windows' default unarcher gave me a "can't read" this file error.  However, 7zip would read it and uncompress it... I figured this was just a problem with Windows.

Butut when I used the img file that it produced to drop on the SD card... it corrupted the SD card.

Again - this was the direct download official link of the "wheezy" distro.  

At this point he SD card would read "18mb left of 56mb" yet this was a 16Gig card.  I freaked out a bit thinking the card was now damaged.  However I read that it only needs to be reformatted....

I tried to reformat using Windows' reformat tool (right click / format)... but the card would always read "56mb avail."

I in fact had to now download the official SD formatter... ugh.  Ok, Now that got me back to 16gigs.

I tried multiple downloads from that direct download link - all with the same problem!

Then I tried the torrent.

They recommend the torrent, but I hate using torrents.  I always end up with spyware from the torrent clients... anyway, I installed bit torrent and downloaded the distro via the torrent, and then uninstalled bit torrent afterwards.

I did get the torrent to download, it was about 10x faster then the direct link.  I also was able to unzip it fine without the error's encountered from the direct link.

I put the SD card back in the Pi, and plugged it back in... It booted up great.

Configuration

When it boots up successfully you'll see something resembling the old PC Bios menu.  This is how you'll configure your Raspberry Pi.

At the configuration screen, you'll want to:
1. set the thing to use the entire space on the SD card, it's the first option in this boot menu.
2. after that, if you want to boot to a desktop, go to that option, if you use a console only, skip it.
3. To turn on SSH (so you can remote into the box via SSH) you need to go to advanced settings and then choose SSH and then Enable.
4. If this is a public facing device, you might want to change the password as well - it's a option on the boot menu.
5. I also changed the hostname (it defaults to 'raspberrypi')

After that I chose to finish and reboot.

From here on out, you'll log in as your user "pi" with the password you picked... or use the default (raspberry.)

SSH

To SSH to this box, first get it's IP by running ifconfig from the command line.  You'll see "eth0" and in there a line inet addr:.  take that address.

From another box, using either terminal on mac, or Putty/supper putty on PC, you can just ssh over to:
ssh [ip]
login as user pi and input your pass.

Installing Software from the Command Line

If you're used to installers in Linux (like Centos' YUM) then you'll get the same thing here, but it's apt-get.  You'll sudo each command like this.  So: sudo apt-get iw tshark would install iw and tshark (for packet sniffing.)

Find the tools you want and they should give you the proper apt-get install commands.

Shutting Down the Raspberry Pi


Also another important thing.  The official site for Raspberry Pi, says to shut it down you just "unplug it" from what I've read this could damage the SD card.  So... you must run an official shutdown:

sudo shutdown -h now
From the UI, you click the red icon and choose shutdown.  Wait for the proper screen statement "system halted," and then remove the power cable.

16 May 2013

Simulating Real World Latency during Automation

So why  not use Jmeter to run performance tests?

The main problem with Jmeter is that it doesn't give a good assessment of front end performance.  It can send load and give back response times in data retrieval, URI end point response times, page load, etc.

But... it's harder to see how long it takes a AJAX menu to pop up after login, or for a "one page site" that loads in all the data asynchronously, to determine the performance hit per browser of loading dynamic HTML 5 elements.  

For this reason, I came up with a process of simulating and reporting the latency during browser automation tests.

I approached this by:
  •     Recording the time between a submit/save/delete and the next rendered screen or alert
  •     Setting up a Latency Generator
  •     Configuring the Automation Framework with Jenkins to dynamically kick off latency, launch the automation, capture the results to a report.

Recording Latency In The Tests

First I approached the idea of how to capture the time it takes to perform an action.  I had hoped there was a Cucumber gem to do this... Those I found really didn't help me much.  They handled a different set of problems. I wanted to know how long it would take from the click of a button, to the load of the next screen.

I realized I needed to write my own wrapper.

It's really very simple.  I define a class variable to be the current time, like:
@start = Time.now

This is run right before the action... for example:
@start = Time.now
@browser.div(:id=>"submit").click

Then I add in a "wait until" in Watir (Wait For in Groovy/geb) like this:
Watir::Wait.until { @browser.alert.exists? }

In the above example it's waiting for an alert window to appear.  If your next screen had a field or button instead, you'd do the wait until, for that element to be loaded.

Just after the Wait Until, I add the end of the timer:
@end = Time.now - @start

with a output to the display:
p "It took #{end} seconds to load XYZ screen."

While it may not be perfect down to the ms, it is quite useful in judging latencies and changes with different bandwidth connections.

I can see that I get a general time of 0.83 seconds on some submit action.  Then change the bandwidth, and rerun and see I am now at 2 seconds on average.

I have since modified this to now output the performance results to a CSV file.  More details on that in a later post.

Setting Up Netem

Second, I added in some throttling of bandwidth and making use of latency and packet loss.  This is handled via a linux tool called Netem.  Netem allows me to generate latency, packet loss and bandwidth throttling.  For example, if you wanted to discover how long it took to go from clicking "submit" to the rendered "dashboard" on a DSL line Or from a visitor from Europe.  Or how long it would take from clicking save, to getting the save confirmation for a user in Asia.

Netem requires the use of Linux. So I choose to set up a special Linux VM that would run Netem.... and set up a Proxy on that VM, so that the browser could connect to it for bandwidth simulation.

Setting Up the VM

I had to set up a Linux VM to be a proxy. I used Squid to be the proxy server.  Squid defaults to using port 3128.  You can follow many tutorials online on setting up squid (like http://www.cyberciti.biz/tips/howto-rhel-centos-fedora-squid-installation-configuration.html)

Then I modified iptables (i.e. sudo vi /etc/sysconfig/iptables) and added a INPUT for port 3128.

Afterwards I restarted iptables.

To test it, I configured the browser's proxy manually to my Proxy and port.  Then ramped up the latency to crazy amounts on the VM and verified that the browser performance degraded.

Netem Commands

I took various target markets for the company I work with (Asia, Europe and US) and gathered some latency reports from the IT dept.

I also took some target customer bandwidth profiles (10Mbit, 768k, 128k, etc.)

Last, I grabed some concept of what we might see for packet loss in the real world.

Here's some examples:

US traffic simulation:

   tc qdisc add dev eth0 root netem delay 80ms 10ms

Asia traffic simulation:

   tc qdisc add dev eth0 root netem delay 160ms 70ms

Bandwidth throttling:

    tc qdisc add dev eth1 root handle 1:0 tbf rate 200kbit buffer 1600 limit 3000
    tc qdisc add dev eth1 root handle 1: cbq avpkt 1000 bandwidth 10Mbit

    Simulate 768k down and 128k up:

    tc qdisc replace dev eth0 root handle 1:0 tbf rate 768kbit burst 2048 latency 100ms
    tc qdisc replace dev eth1 root handle 2:0 tbf rate 128kbit burst 2048 latency 100ms

Kill Netem:

To end any Netem protocols running, I use:
tc qdisc del dev eth0 root

Automating Netem as part of Cucumber Automation

First, I made a change in the env.rb file within the features folder.  In that file, in the begin block, I added the highlighted part:

def environment
  (ENV['ENVI'] ||= 'proxy').downcase.to_sym
end

Before do  |scenario|
  p "Starting #{scenario}"
  if environment == :int
    @browser = Watir::Browser.new(:remote, :url=>"http://[my qa selenium grid server]:4444/wd/hub", :desired_capabilities=> browser_name)
    @browser.goto "http://[my integration test env]:8080"
  elsif environment == :local
    @browser = Watir::Browser.new browser_name
    @browser.goto "http://[my integration test env]:8080"
  elsif environment == :proxy
    profile = Selenium::WebDriver::Firefox::Profile.new
    proxy = Selenium::WebDriver::Proxy.new(:http => "[my centos VM running netem and squid goes here]:3128")
    profile.proxy = proxy
    driver = Selenium::WebDriver.for :firefox, :profile => profile
    @browser = Watir::Browser.new(driver)
    @browser.goto "http://[my integration test env]:8080"

  else
    @browser = Watir::Browser.new browser_name
    @browser.goto "http://[alternate test env]:8080"
  end
end

So now, when I send the command: Cucumber features/performance_test_Asia.feature ENVI=proxy
it will kick off cucumber, launch Firefox - which will use the proxy we set up on the Linux VM.  that Linux VM, if using Netem to trigger expected latency from Asia - will simulate real world response times.

Configuring Jenkins

Now that the automation works, we want to turn Netem latency on before a test, and off after a test.  The bet way to do this, that I've found is to use Jenkins.

Jenkins configuration on the Netem VM

In my case I have that Linux VM with the proxy and netem. I put Jenkins on it.  I created jobs pertaining to netem.  Jobs like:
1. Start Netem with Latency for Asia (70ms-140ms)
2. Start Netem with Latency for US-Michigan (10ms-80ms)
3. Start Netem with Latency for Europe (70ms - 100ms)
4. Start Netem with 0.3 percent packet loss
5. Stop Netem

Each job is just running a shell script on that linux VM.  I.e. a stand alone job to simply run
sudo tc qdisc add dev eth0 root netem delay 100ms 70ms
for example.

The stop Netem job, simply runs:
 sudo tc qdisc del dev eth0 root


In my case, I needed to use the ssh plugin for Jenkins. This allows me to sudo commands. So although i'm not ssh'ing anywhere, the SSH plugin allows me to authenticate on the same box with the account that has sudo privledges.  I have more on setting that up in a separate blog.

At any rate, here's what you do next.... you go to a job, like "Start Netem with Latency for Asia..." in the job details, right click the build link and copy out the URL.  Paste that build URL for each job.

These URL's will look something like: 
http://[Your Jenkins]:8080/view/[Your Project]/job/Stop%20Netem/build?delay=0sec

Jenkins configuration on the Automation Environment

Step 1

Back at the main automation suite (hopefully you have jenkins running jobs there - if not, you'll need to set Jenkins up), create a job.  This will be a free standing job. All it will do is call a script to hit those URL's you copied.

How do you do that? Well you could curl it, if your automation jenkins is on a linux env.  Or you could do wget... or... you could script it in ruby/python/groovy... or in my case, I use Jmeter.  I simply have a Jmeter script that hits that URL.

So to recap:
In the automation environment, I have a Jenkins job that is a parent job. It starts Netem, by calling a Jmeter script to hit the appropriate Netem URL on the Linux VM.  As long as this environment can talk to the linux proxy server, this will work fine.

Step 2

Next I create another job in the automation Jenkins. This job will execute the Cucumber script.  It is a simple stand alone job that runs the command line:
cucumber ENVI=proxy features/my_asia_performance.feature

Step 3

After creating that Cucumber job, I link it as a child to the job we made in step 1.  In this way, Step 1, sends a command to turn on Netem on the Linux VM.  Then that job is configured (in the configure of the job) to launch a project when it's finished. That project will be the automation job we made in step 2.

Step 4

After setting the cucumber automation as a child project of the Netem job in step 1, we will now make the last job.  This will again be a stand alone job, it will simply stop Netem.  It will be a child of the Cucumber job/project.

So create a new Jenkins job, and have it curl/wget or use a script to hit the URL to the linux Jenkins that will launch the Netem kill job.

Step 5.

Edit the Cucumber job, so that after it finishes, it will launch a project - this being the Netem stop/kill job you created in Step 4.

All done.

To run it, simply run the first job in step 1.

That job will turn Netem on, setting the appropriate latency - then when that job finishes it will automatically launch the browser automation job (which is recording the response times) - then after that job finishes, it will launch the job to turn off the Netem latency.

Pipping Out the Output to CSV

Most people don't want to read log data to find response times.  I got a request to keep this data I was gathering in a common flat file.  So I decided to go with CSV.

 What I did was use the native CSV class in Ruby to handle this:
  CSV.open("C:\\#{locale}_{$TestTime}.csv", "ab") do |csv|
    csv << ["Save Action","#{@CF_save_end}", Time.now]
  end

I'll go into more detail on this in my next blog post.

22 April 2013

Automated Verification of VOIP Audio


I've created a Presentation that goes over these points as well:
http://prezi.com/-29ebxieb4ek/copy-of-rtp-re-assembly/

*UPDATE*
I found this awesome work, using google's translate api, to transcode the audio to text:
http://cheateinstein.com/category-shell/using-google-voice-api-to-transcribe-audio/

I've now used this at the final end of the process, to verify the text heard is what is expected!!

I've been working on this VOIP/SIP automation framework for a few months now. I started with a Cucumber framework, and then added on with some VOIP/SIP specific tools like SIPP and SIPCLI.

I got to where the test harness' I built with these tools, would use Jenkins to push button (or on a schedule or build commit) drive traffic to a phone number... verify it reached it by acknowledgements sent back.  But what if the phone number was going to the wrong destination, and sent back acknowledgements?

At that point I used TollFreeForwarding.com's technology to set a email alert as an endpoint on a phone number. For example, you call: 888-888-8888 and you get an IVR. you press 1, and are sent to a voicemail - you pass in audio and hang up. Then TollFreeForwarding.com emails the configured email on the account, the recording.

It was better, but it required a voicemail to email application at every end point. It also doesn't verify that audio actually occurred on the call. What if no audio played back? Or there was significant jitter to not understand it?

To further this testing, I started thinking of recording the call and using some sort of analysis of the recording to verify it's what was expected.  

This is my first draft at answering that need.  It can be improved.  But it's a step in the right direction.

What I'm doing

  1. This automation dials a number, with a known IVR or greeting.  
  2. It does a packet capture during the recording
  3. It filters out the RTP channels from the packet capture and then creates a wav out of the pcap file.
  4. Once there is a wav file, it runs diagnostics on it... generating some visual graphs like the image on this blog... but more importantly (and more useful) it generates audio information that I use as a footprint for the audio playback.
  5. This audio is also sent to google who transcribes it and sends me back the text which is compared to the expected string.

Tools used

  1. sipp to drive an automated command line sip call
  2. tshark (command line version of wireshark)
  3. jenkins (for the GUI to drive and schedule these tests)
  4. sox (linux based audio conversion and analysis tool)
  5. some shell scripting

How it Works

The test has a parent job, that kicks off two sub jobs.  These sub jobs run simultaneously.  One does a phone call to a phone number with a recording Greeting/IVR.  The other job runs a shell script that maintains the test itself.  The second job uses tshark to record the packets and filter the rtp, then uses sox to convert the raw audio to a wav and do some analysis on the wav.

The Shell Script

First I set tshark to record for a specific duration, that I think will encompass the call:
tshark -a duration:20 -w /jenkins/userContent/sip_1call.pcap

I assign a variable to a tsark task to scan the RTP packets and find the hex value for the RTP packets (I learned these three parts from a online tutorial, but lost the bookmark):
ssrc=$(tshark -n -r /jenkins/userContent/sip_1call.pcap -R rtp -T fields -e rtp.ssrc -Eseparator=, | sort -u | awk 'FNR ==1 {print}')

The above would return a hex value like:
0x344292302

Which is followed by:
sudo tshark -n -r /jenkins/userContent/sip_1call.pcap -R rtp -R "rtp.ssrc == $ssrc" -T fields -e rtp.payload | tee payloads

The above looks for that Hex value captured previously, and holds that as a variable, payload.

Finally, we have a for statement in the shell script to convert the payload value from above, to a raw audio file:
for payload in `cat payloads`; do IFS=:; for byte in $payload; do printf "\\x$byte" >> /jenkins/userContent/sip_1call.raw; done; done

At this point I had a raw audio file. I found a linux tool called sox that was  a good fit for this conversion... so I installed it and added these lines into my script...
Sox is then invoked to convert the raw audio to a wav:
sox -t raw -r 8000 -v 4 -c 1 -U /jenkins/userContent/sip_1call.raw /jenkins/userContent/sip_1call.wav


Then I run a couple more Sox commands:
This one creates stats, which Jenkins captures in the log file of the test run:
sox /var/lib/jenkins/userContent/sip_audio_1call.wav -n stat

The stats generated will look like this:
Samples read:             15680
Length (seconds):      1.960000
Scaled by:         2147483647.0
Maximum amplitude:     0.425659
Minimum amplitude:    -0.285034
Midline amplitude:     0.070313
Mean    norm:          0.043354
Mean    amplitude:    -0.000055
RMS     amplitude:     0.070984
Maximum delta:         0.243896
Minimum delta:         0.000000
Mean    delta:         0.019919
RMS     delta:         0.034190
Rough   frequency:          613
Volume adjustment:        2.349
 
The two highlighted values seem to be consistent with the same audio.  At this point, that's what the test assertion is based on.  I have a better plan in the works for a future upgrade to this test.  But for now, I'm using the rough frequency and max amplitude to determine the pass / fail criteria.

Is it perfect? No. It's potential for false negatives. The rough frequency *could* change, but so far it hasn't for the same audio I expect.

Spectograms
If your into spectrogram's (and who isn't?), then sox will also output one if you like, I end the shell script with this:

sox /jenkins/userContent/sip_1call.wav -n spectrogram -y 2 -l -o /jenkins/userContent/sip_1call.png

If anyone has any other tools that can pull out more data, please let me know.

The Upshot?

One shell script, called by Jenkins, running 3 tools gets this job done.

Verify Audio via Speech To Text

A few people approached me and mentioned rough frequency may not remain constant as the test call goes through different hops.  So I began to investigate this some more... I found this guy:
http://cheateinstein.com/category-shell/using-google-voice-api-to-transcribe-audio/

he had created a way to use a shell script to send audio files to google for transcription.

I modified his script a little to work for my needs, and added a text assertion.  If the text fails comparison then I exit the script with a error code, which forces jenkins to regard this as a total failure.

Here's the part I added to the bottom of my previous script:
echo "1 - Translate with SOX - Convert WAV to FLAC with 16000"
sox /jenkins/userContent/sip_audio_1call.wav input.flac rate 16k
echo "2 - Submit to Google Voice API"
wget -q -U "Mozilla/5.0" --post-file input.flac --header="Content-Type: audio/x-flac; rate=16000" -O - "http://www.google.com/speech-api/v1/recognize?lang=en-us&client=chromium" > output.ret
echo "3 - Extract recognized text"

cat output.ret | sed 's/.*utterance":"//' | sed 's/","confidence.*//' > output.txt
echo "4 - Display text"
a=`cat output.txt`
echo $a
b="tollfreeforwarding.com"
if [ "$a" = "tollfreeforwarding.com" ];
then
        echo "Verified audio is tollfreeforwarding.com"
else
        echo "FAIL audio is not tollfreeforwarding.com"
        exit 666


fi;


In my scenario, I've seeded the phone greeting on the number that is called to be an announcement audio that says, "Toll Free Forwarding Dot Com"  which google turns correctly to "tollfreeforwarding.com" and I validate against that.

19 April 2013

Converting RDP in pcap to Audio Wav files

After following a lot of different tutorials (some of which worked some of which didn't), I came up with a shell script using a couple tools to scrape a packet capture file, pull out the rdp packets, and then convert them back into audio.

For me this will be useful in automated testing.  I currently drive automated SIP calls via SIPCLI and ruby for a variety of tests at work.  But how do I know I get the right end point?  In the past, I'd have the phone number I dial, record voice mail and send me an email, and the sipcli client would send over text to speech audio.

But I dont always have the luxury of being able to configure the phone number to voice mail. 

I've been wanting to do a packet capture during the test and convert it back to audio afterwards, then do a wav comparison on the expected audio vs. the captured audio.

Tools used:

tshark
sox

These are both linux tools. 
tshark is a command line version of wireshark.  It's installed on centos boxes using yum install wireshark-gnome.
sox via yum install sox

Sox is a audio analysis tool that is run from the command line. 

Test Script:

After looking at some examples online of different tools, I pieced this together from other people's examples, with a few modifications. It seems to work for me:

Contents of pcap_to_wav.sh:


ssrc=$(sudo tshark -n -r capture.pcap -R rtp -T fields -e rtp.ssrc -Eseparator=, | sort -u)

echo $ssrc

sudo tshark -n -r capture.pcap -R rtp -R "rtp.ssrc == $ssrc" -T fields -e rtp.payload | tee payloads

for payload in `cat payloads`; do IFS=:; for byte in $payload; do printf "\\x$byte" >> sound.raw; done; done

echo 'sox has converted pcap to wav file'
sudo sox -t raw -r 8000 -c 1 -U sound.raw capture3d.wav

That's it!

basically if you have sudo access, you can run this and it will take the pcap and find the rdp packets, then make that a raw audio file... sox is then used to convert the raw file to a wav file.

At this point, you can further use sox to compare one wav to another wav.

08 April 2013

Selenium Grid Up and Running with Cucumber Automation

Setting up Selenium Grid with Cucumber


This is more of an advanced topic I think. I did a lot of research to come up with the solution I use here at TollFreeForwarding.com

I have tests that would take hours to run sequentially.  To bring this back to normalcy, I use Selenium Grid to farm the jobs to multiple VM's simultaneously.  This way the total time for all tests to complete is the longest test I have (5min.)

To get to this point, the Cucumber tests have to be modular.  They have to be broken up. You can't have one giant cucumber test.  Selenium Grid can't take one giant test and farm out each Scenario.  Instead you have to have multiple features.

For example:
If you  have a UI you automate and you have coverage for areas like -
  • Account Creation
  • Profile CRUD actions
  • Forum Post CRUD actions
  • Calendar Schedules
  • Call Customer Support (WebPhone)
  • Admin: Create Menus for Customers
  • Admin: Make new announcements for Customers
  • Admin: CRUD actions on gallery uploads/edits/deletes
Then I'd make each of these it's own feature. 

Imagine each of the eight features above took 5 min to complete.  That's a total of 40min if they are run sequentially.  Meaning you'd start the tests and come back in 40min.  What if you could get that down to 5 min?

You can!

Run them all at the same time, across multiple VMs.  That way they all run in parallel and finish in 5min.

This is where Selenium Grid comes in.

Install the Grid

On the Main VM that will run this job, you will install the Selenium Grid Hub.  This is rather simple to do.  You basically have to have java installed, and you run a command like:
[path to your selenium server standalone jar]/selenium-server-standalone-2.31.0.jar -role hub

I found our VM's might sometimes restart (IT restarts and what not) so I added a batch file on this Windows VM and use the Windows Scheduler to assign a new task of "on restart run this batch file" the batch file has this content:
@echo off
"C:\Program Files (x86)\Java\jdk1.7.0_17\bin\java.exe" -jar "C:\Selenium-grid\selenium-server-standalone-2.31.0.jar" -role hub

Which tells the server "run Java to execute the Selenium server stand alone with the parameter role of "hub."

Node VM's must be configured

On each VM that will be used to get jobs from the Grid hub (referred to as 'nodes'), you will need to set them up.  These boxes/VM's again need to have JAVA installed and need to have the same selenium-server-standalone jar.  But it will be run with the role 'node.'

Here's an example:
java -jar selenium-server-standalone-2.31.0.jar" -role node -hub http://qa1.ifn.com:4444/grid/register -browser browserName=chrome,maxInstances=5

Similar to the Hub, I also created a batch file and used the windows scheduler to make sure that on a restart the batch file executes... it has this code:

C:\Program Files (x86)\Java\jre7\bin\java.exe" -jar "C:\Selenium-grid\selenium-server-standalone-2.31.0.jar" -role node -hub http://mydomain.com:4444/grid/register -browser browserName=chrome,maxInstances=5

Note that http://mydomain.com:4444 is the domain that the hub is running on.  This command is registering the browser Chrome and saying it has 5 max instances (this means it will run up to 5 chrome tests simultaneously on this box.)  you can of course change that.


Setting Up Cucumber

Since I use Jenkins to run all the Cucumber jobs, I want to be able to specify parameters from a command line that will:
  • Run the Cucumber Features
  • Specify the Browser for the test
  • Specify if the grid will be used or not
To do that, I use the env.rb file in Cucumber's /Features/support  directory.

Here's an example of what I did to create these parameters to be used from the command line....
First I added some code:

def browser_name
  (ENV['BROWSER'] ||= 'firefox').downcase.to_sym
end

def environment
  (ENV['ENVI'] ||= 'int').downcase.to_sym
end

This is setting the parameter "browser" and "envi" (for environment) to be called on the command line.  It is also giving a default value for each.  By default the browser is Firefox and the environment is int (for integration.)

Below that code, I wrote this before statement:
Before do  |scenario|
  p "Starting #{scenario}"
  if environment == :int
    @browser = Watir::Browser.new(:remote, :url=>"http://mydomain.com:4444/wd/hub", :desired_capabilities=> browser_name)
    @browser.goto "http://integration.env.com:8080"
  elsif environment == :local
    @browser = Watir::Browser.new browser_name
    @browser.goto "http://integration.env.com:8080"
  end
end

This block above says that before we run the Cucumber scenarios, we must define a few things.  First if the environment is passed in as 'int' (our default) we will define @browser to be equal to Watir::Browser.new(:remote, :url=>"http://mydomain.com:4444/wd/hub", :desired_capabilities=>browser_name)

So if environment is int, we're defining the @browser to be going to selenium grid to send all the traffic.  If a browser is passed in, we're also sending that along. 

Test this out by sending a command from the project you have like:
cucumber envi=int browser=firefox

You can monitor the jobs get picked up by different vm's from the Grid console.

Jenkins Configuration

Once you have Selenium Grid set up, go to Jenkins and make a new basic job. 

This job will just do a Windows Shell command.

You'll want to use the same command line parameter that was working to test your test above. For example, if you have a feature called "Account Creation" you would do something like this:
cucumber features/account_creationl.feature BROWSER=firefox ENVI=int

If you did that for each job, you'd have all your jobs going to the grid.  You could make a parent job that runs all features simultaneously.

To do that you create a project that has only one function, to kick off downstream projects... each feature would be it's own project/job.  So the parent project would launch downstream jobs of: account creation, profile crud actions, forum crud actions, etc.  So they all run in parallel and are sent to the grid.

Gotcha's

There's a gotcha.  IE webdriver doesn't like to be used remotely.  For this one outlier, I use IE in a serial fashion.

So in my Jenkins I have my jobs on tabs.  The first time is Functional Tests (these run in Firefox), the second tab is Chrome Tests, the third is IE tests.

On the first tab I have the jobs configured to run with FF only.  On the second to run with Chrome only. The third tab runs in IE only AND does NOT SEND the jobs to the grid. 

This way, I get the Firefox functional tests finished in 5min, Chrome finished in 5 min and IE takes the normal time of 40min.  It's still better then 40min a pop.



27 March 2013

How I built My SIP Automation Testing Framework from the Ground Up

This is a pretty long and detailed (long) blog post.  I'll try and break it out later to smaller posts. I've tried to capture all the details I can, so that anyone can set up something like this relatively easily.

My current SIP Automation Framework, uses all open source tools to:

  • Generate performance tests with our SIP framework
  • Generate reports
  • Generate Graphs
  • Capture PCAP's during the 



A Tour Around the Framework

Dashboard

My Dashboard looks like this.  I reskinned Jenkins and gave it a bit of a visual polish.  I also added some new links on the left "Get Call Control Logs," "Latest PCAP," "Archived PCAP" and so forth.  In the main section the different SIPP preset tests are listed, with their failure/pass criteria.  At the bottom of the main section are some visual Graphs of the SIPP results.


Left Nav

The left nav has the new items mentioned above.  Call Control is a button to grab the logs off one of our servers.  Automation Home, goes to the QA Automation page.  Latest PCAP loads the last Packet Capture done with tshark.  Archived PCAPs points to a repository of PCAPs.

Main Body

The Main Body lists the jobs for the project.  In this case, "jobs" equates to a preset SIPP scenario with command line parameters.  It also shows the current pass/fail value in the first column, and the history of pass/fails on the far right column.

Bottom Graphs

The bottom graphs are utlizing a plugin to display JPG images. I'm creating the graphs with CLICharts, during the test run.  Jenkins points to these jpg's via the plugin.

Test Detail Page

Inside a SIPP Test (or job), I've modified it a bit to  have a larger graph, and some useful links.

The links are, Historic Data (which is Historic Graph data) and PCAP Captures (historic PCAP captures.) These links basically point to Workspace locations, so Jenkins is already set to show you a file structure like this: 

Tools Used

The tools I used were:

  • Jenkins
  • SIPP
  • JMETER
  • CLICharts
  • Tshark/Wireshark
  • a bit of Shell Scripting

OS Used

I started on Windows, migrated to OSX and finally ended up putting the whole thing on a Centos VM.  OSX couldn't keep up with the pace of SIPP and performance testing for our needs. I would get spikes.  This could be the machine I was using, but either way I didn't want the framework to be running on a machine in my office. 

I opted for a blade server hosting a Centos VM for me.  It's very fast and the SIPP performance spikes went down drastically... indicating it wasn't code related, but test machine related.

Jenkins

I'm using Jenkins CI to maintain all the SIP tests. My reasoning for this was two fold.  
  1. SIPP was going to be used as our main SIP test/performance tool. It uses XML and Command Line params.  I wanted to capture test presets that had both of these. Jenkins will let me build a job that runs a SIPP command line with params i need, and specify the scenario file. So everything is maintained together
  2. Using Jenkins CI was useful for us, since our developers use Jenkins, they could send a command to run the SIP tests once a build is ready.
To install Jenkins, I've installed it on Windows as well as OSX... When I went with the blade install I had our network admin install Jenkins CI there... I'm assuming he did a yum install jenkins  but i'm not sure.  

Either way you can find more info on installing Jenkins from: http://jenkins-ci.org/

SIPP

To install SIPP on CentOS, I just did a sudo yum install sipp
SIPP is a robust Open Source command line tool to generate SIP calls and traffic. It has two main aspects:
  1. Scenario Files
  2. Command Line Parameters
SIPP is one of the most complex command line tools I've used.  Hence the need to capture the command line params tha tare useful... as there are so many.  

Scenario files are just XML files that build packets.  You are litterally building out a packet in XML, and then saying what you expect as a response.  

SIPP comes with some default scenario files.  Unfortunatley those scenario files needed to be modified by me to work with our team's development.  There are certain responses we send, or do not send, that don't match the default scenario files.  

I won't get into writing your own scenario file, as I think it's outside the overall scope here of building the framework.  I'll post more on the scenario file creation in a later post.

For more information on SIPP, check out: http://sipp.sourceforge.net/

JMeter

To install Jmeter on Centos, I just did a sudo yum install for apache jmeter

I  have a specific need to update my test framework before SIP Load testing... to do this I use JMeter to quickly send some POST or PUTs to end points to set up the data I need. I think this is uncommon for most people doing SIP testing - so I won't cover this aspect in this post.

CLICharts

To install CLICharts on Centos, I just did the yum install.

SIPP will output it's performance data to a CSV file, if you choose.  I found a very useful unix tool called CLICharts.  CLICharts will take a csv file and do all sorts of analysis to it.  In my case, I use it to generate graphs at the end of a test. These jpg graphs are put in a folder that Jenkins points to and references with hard links.  This way each test run, updates the Jenkins page with the latest graph of reuslts.

For more information on CLICharts check out: http://clichart.sourceforge.net/docs/quickstart.html

Tshark/Wireshark

This was a bit different... you need to do a yum install for wireshark-gnome.  That will install a version of wireshark that is usable from the command line (just doing a yum install wireshark will install it in /sbin and not /usr/bin, so you can't just use it as a commandline)... and of course you get the tshark command line tool.

I use this tool to run during the SIPP test.  Since I know the total duration of a SIPP test, I set the default duration of the Packet Capture to the same duration.  Basically I wrote a shell script that calls tshark, sets a duration, and outputs a file.  The file is also copied to a archive folder and has the date/time stamp added to the file name.  These can be referenced by Jenkins as you'll see further in this post.

Putting It All Together

So if you have all the tools above installed, lets put it all together and get a SIP Automation Framework up and running!

Jenkins Plugins needed:

In Jenkins, install these plugins (SSH Slaves, Sidebar Link, Green Balls, Dashboard View, and if you want to restyle the CSS of your Jenkins, get the Simple Theme plugin.)

Once you've installed those Jenkins plugins, restart Jenkins.

Set Up SSH Slaves

Go into Jenkins / Configuration
Scroll down to SSH  remote hosts
Fill this out with the host that is running Jenkins, and you should have sudo access to this box.

You'll be using SSH to sudo your commands, even though you aren't "ssh'ing" anywyere, this is how Jenkins will log in as you to sudo commands.  

Setting Up A SIPP Job in Jenkins

First, open Jenkins... by default it's your localhost:8080.  
When you have Jenkins up, click "New Job"
Choose "Build a free-style software project"

Give the project a name and click OK.
Click "Advanced" button on the Advanced Project Options.
Then check "use customer workspace" and put the path/directory to your sipp scenario files.

Check off "Execute shell script on remote host using ssh"
in the SSH site dropdown make sure it's the same one you input on "Set Up SSH Slaves" above.
in the Pre build script, type:
cd [your directory to your sipp scenario files]
sudo sipp -s [service/number you want to dial] -sf [path to scenario xml file] -m [amount of calls you want to run] -mi [your ip of the jenkins server (media IP)] -d [pause duration override, if you need it] -trace_rtt -trace_err -stat_delimiter ,

The reason you need to sudo sipp here, is because if you are sending pcap audio, you need to use -mi which creates a live socket. You can't create a live socket unless you are sudo'd.

If you run this, it should work.

Adding tshark PCAP captures

I use a parent project, it kicks off the above as a downstream. Meaning, I have a parent project called "2000 calls @10 calls per min to Integration," that job just kicks off a downstream job... two actually.  The first is what we just created above (the SIPP job) and the second job it kicks off simultaneously is the tshark pcap capture.  They run at the same time.

How you do this is simple:
On the Server:
create a shell script in the folder with  your scenario files.  Input something like this:
sudo tshark -a duration:201 -w sip_test.pcap
sudo mv sip_test.pcap /var/lib/jenkins/userContent/
sudo chmod 777 /var/lib/jenkins/userContent/sip_test.pcap
sudo cp /var/lib/jenkins/userContent/sip_test.pcap /home/jenkins/sipp-3.3/pcap/archived/sip_test$(date +%F-%T).pcap

Ok, so this script will run tshark for 201 seconds.  My SIPP tests runs for 200 seconds.  it outputs a pcap file to this directory.  Then I move it to my Jenkins UserContent folder.  I change the permissions of this file so it can be read by other users. Finally I copy this file, and change the name of it to a date/time stamp and drop it in an archive.

Update this script to fit your specific needs and directory structure, then run it. Make sure it works.

Jenkins Packet Capture Job

Back in Jenkins, create a new Jenkins job, just like before.  Give it a name like "Capture PCAP."  This we'll run a SSH script just like before. But instead of running SIPP, we're going to call that shell script (i.e. sudo /somepath/yourscript.sh)

Ok, back at that parent job I just spoke about... remember, the job that just kicks off jobs?  Scroll down to the bottom and click Add Post Build Porcess and choose to build a project(s).  In the field input your SIPP project followed by a comma, and the job you just made to run the tshark script.

Run the Parent job, and verify that you get the pcap in your workspace.

Linking the PCAP to Jenkins

One plugin we installed for Jenkins was called Side Bar.  This lets us make our own Left Nav links.  Go ahead click Jenkins / Manage Jenkins / Configuration and in the section "Additional Side Bar Links" click add.  If you want to use your own icon, use the uploader to upload it. it will by default upload it to Jenkin's userContent folder.  It will provide you the path after it uploads.

For the Link Text, input whatever you want (I.e. "Latest PCAP File")
For the Link URL, point to the PCAP file location on your server.

Linking to Archived PCAPs

If you want to link to older PCAPs, well you would have had to do the script bit I did above, where I copy the pcap to a archive folder.  Hopefully this folder you copy to is also the same folder used in your Project workspace.  

That way you can go to the project workspace and point to your archive folder.  You do that by clicking on the project, then "workspace" and click the file structure till you get to your archive folder.  Copy that URL. Use that URL as your link.

Adding a Downstream Job to do Graphing and Clean Up

Ok, now that you got sipp to work, it should be dumping a csv file to your workspace. Great. lets graph that.

Creating a script to do Charting

Back on your Jenkins server, go to your workspace you're using for the other jobs (i.e. the folder with your sipp scenarios.

Create a new file
In that file enter something like this:
clichart -cfvl 0,1 *_rtt.csv -o /var/lib/jenkins/userContent/charts/chart1.jpg
cp /var/lib/jenkins/userContent/charts/chart1.jpg /home/jenkins/sipp-3.3/charts/sipcalls$(date +%F-%T).jpg
convert /var/lib/jenkins/userContent/charts/chart1.jpg -resize 510 /var/lib/jenkins/userContent/charts/chart1-s.jpg
mv *.csv archived_csv/

This runs clichart and picks the correct column of the default output of SIPP.  so keep the parameters.  I tell it to use whatever csv file is in the folder, and make a jpg of the results chart.  Remember our previous script auto archives csv files after each run... so only one csv should be there at a time.

Then ti copies the jpb to an archive and makes  a thumbnail using convert.  Convert is a special call from another command line tool that I forget the name of... 

Jenkins running your script

Using the process earlier, create another Jenkins job.  This time give it a name like "Graph Data."  In the job details, check off "Execute Shell script on remote host using ssh.  Make sure the SSH site is correct and in the Pre build script do:
cd [to your directory with the chart shell script]
sudo ./[your chart script.sh]

Save...

Getting Graphs on Jenkins Dashboard

There's a plugin we installed for Jenkins called Dashboard. if you Edit the View of your Dashboard or Main Tab, and scroll down you'll see some options for Dashboard Portlets... You can have "top, left, right, bottom" portlets... and one portlet option is IMAGE.  Choose Image and link to the hard link of your jpt graph from clicharts.

Run the Parent....
you should see:
  1. Parent calls two jobs: SIPP and tshark
  2. When those jobs finish up, a downstream job charts the data and clearns up csv files in the main folder
  3. All images you've hard linked to, should work to show graphs.
  4. All PCAP's you hard linked too should show PCAPS
  5. your PCAP archive links should now show the archives