How I built My SIP Automation Testing Framework from the Ground Up

This is a pretty long and detailed (long) blog post.  I'll try and break it out later to smaller posts. I've tried to capture all the details I can, so that anyone can set up something like this relatively easily.

My current SIP Automation Framework, uses all open source tools to:

• Generate performance tests with our SIP framework
• Generate reports
• Generate Graphs
• Capture PCAP's during the 

A Tour Around the Framework

Dashboard

My Dashboard looks like this.  I reskinned Jenkins and gave it a bit of a visual polish.  I also added some new links on the left "Get Call Control Logs," "Latest PCAP," "Archived PCAP" and so forth.  In the main section the different SIPP preset tests are listed, with their failure/pass criteria.  At the bottom of the main section are some visual Graphs of the SIPP results.

Left Nav

The left nav has the new items mentioned above.  Call Control is a button to grab the logs off one of our servers.  Automation Home, goes to the QA Automation page.  Latest PCAP loads the last Packet Capture done with tshark.  Archived PCAPs points to a repository of PCAPs.

Main Body

The Main Body lists the jobs for the project.  In this case, "jobs" equates to a preset SIPP scenario with command line parameters.  It also shows the current pass/fail value in the first column, and the history of pass/fails on the far right column.

Bottom Graphs

The bottom graphs are utlizing a plugin to display JPG images. I'm creating the graphs with CLICharts, during the test run.  Jenkins points to these jpg's via the plugin.

Test Detail Page

Inside a SIPP Test (or job), I've modified it a bit to  have a larger graph, and some useful links.

The links are, Historic Data (which is Historic Graph data) and PCAP Captures (historic PCAP captures.) These links basically point to Workspace locations, so Jenkins is already set to show you a file structure like this: 

Tools Used

The tools I used were:

• Jenkins
• SIPP
• JMETER
• CLICharts
• Tshark/Wireshark
• a bit of Shell Scripting

OS Used

I started on Windows, migrated to OSX and finally ended up putting the whole thing on a Centos VM.  OSX couldn't keep up with the pace of SIPP and performance testing for our needs. I would get spikes.  This could be the machine I was using, but either way I didn't want the framework to be running on a machine in my office. 

I opted for a blade server hosting a Centos VM for me.  It's very fast and the SIPP performance spikes went down drastically... indicating it wasn't code related, but test machine related.

Jenkins

I'm using Jenkins CI to maintain all the SIP tests. My reasoning for this was two fold.  

1. SIPP was going to be used as our main SIP test/performance tool. It uses XML and Command Line params.  I wanted to capture test presets that had both of these. Jenkins will let me build a job that runs a SIPP command line with params i need, and specify the scenario file. So everything is maintained together
2. Using Jenkins CI was useful for us, since our developers use Jenkins, they could send a command to run the SIP tests once a build is ready.

To install Jenkins, I've installed it on Windows as well as OSX... When I went with the blade install I had our network admin install Jenkins CI there... I'm assuming he did a yum install jenkins  but i'm not sure.  

Either way you can find more info on installing Jenkins from: http://jenkins-ci.org/

SIPP

To install SIPP on CentOS, I just did a sudo yum install sipp

SIPP is a robust Open Source command line tool to generate SIP calls and traffic. It has two main aspects:

1. Scenario Files
2. Command Line Parameters

SIPP is one of the most complex command line tools I've used.  Hence the need to capture the command line params tha tare useful... as there are so many.  

Scenario files are just XML files that build packets.  You are litterally building out a packet in XML, and then saying what you expect as a response.  

SIPP comes with some default scenario files.  Unfortunatley those scenario files needed to be modified by me to work with our team's development.  There are certain responses we send, or do not send, that don't match the default scenario files.  

I won't get into writing your own scenario file, as I think it's outside the overall scope here of building the framework.  I'll post more on the scenario file creation in a later post.

For more information on SIPP, check out: http://sipp.sourceforge.net/

JMeter

To install Jmeter on Centos, I just did a sudo yum install for apache jmeter

I  have a specific need to update my test framework before SIP Load testing... to do this I use JMeter to quickly send some POST or PUTs to end points to set up the data I need. I think this is uncommon for most people doing SIP testing - so I won't cover this aspect in this post.

CLICharts

To install CLICharts on Centos, I just did the yum install.

SIPP will output it's performance data to a CSV file, if you choose.  I found a very useful unix tool called CLICharts.  CLICharts will take a csv file and do all sorts of analysis to it.  In my case, I use it to generate graphs at the end of a test. These jpg graphs are put in a folder that Jenkins points to and references with hard links.  This way each test run, updates the Jenkins page with the latest graph of reuslts.

For more information on CLICharts check out: http://clichart.sourceforge.net/docs/quickstart.html

Tshark/Wireshark

This was a bit different... you need to do a yum install for wireshark-gnome.  That will install a version of wireshark that is usable from the command line (just doing a yum install wireshark will install it in /sbin and not /usr/bin, so you can't just use it as a commandline)... and of course you get the tshark command line tool.

I use this tool to run during the SIPP test.  Since I know the total duration of a SIPP test, I set the default duration of the Packet Capture to the same duration.  Basically I wrote a shell script that calls tshark, sets a duration, and outputs a file.  The file is also copied to a archive folder and has the date/time stamp added to the file name.  These can be referenced by Jenkins as you'll see further in this post.

Putting It All Together

So if you have all the tools above installed, lets put it all together and get a SIP Automation Framework up and running!

Jenkins Plugins needed:

In Jenkins, install these plugins (SSH Slaves, Sidebar Link, Green Balls, Dashboard View, and if you want to restyle the CSS of your Jenkins, get the Simple Theme plugin.)

Once you've installed those Jenkins plugins, restart Jenkins.

Set Up SSH Slaves

Go into Jenkins / Configuration

Scroll down to SSH  remote hosts

Fill this out with the host that is running Jenkins, and you should have sudo access to this box.

You'll be using SSH to sudo your commands, even though you aren't "ssh'ing" anywyere, this is how Jenkins will log in as you to sudo commands.  

Setting Up A SIPP Job in Jenkins

First, open Jenkins... by default it's your localhost:8080.  

When you have Jenkins up, click "New Job"

Choose "Build a free-style software project"

Give the project a name and click OK.

Click "Advanced" button on the Advanced Project Options.

Then check "use customer workspace" and put the path/directory to your sipp scenario files.

Check off "Execute shell script on remote host using ssh"

in the SSH site dropdown make sure it's the same one you input on "Set Up SSH Slaves" above.

in the Pre build script, type:

cd [your directory to your sipp scenario files]

sudo sipp -s [service/number you want to dial] -sf [path to scenario xml file] -m [amount of calls you want to run] -mi [your ip of the jenkins server (media IP)] -d [pause duration override, if you need it] -trace_rtt -trace_err -stat_delimiter ,

The reason you need to sudo sipp here, is because if you are sending pcap audio, you need to use -mi which creates a live socket. You can't create a live socket unless you are sudo'd.

If you run this, it should work.

Adding tshark PCAP captures

I use a parent project, it kicks off the above as a downstream. Meaning, I have a parent project called "2000 calls @10 calls per min to Integration," that job just kicks off a downstream job... two actually.  The first is what we just created above (the SIPP job) and the second job it kicks off simultaneously is the tshark pcap capture.  They run at the same time.

How you do this is simple:

On the Server:

create a shell script in the folder with  your scenario files.  Input something like this:

sudo tshark -a duration:201 -w sip_test.pcap

sudo mv sip_test.pcap /var/lib/jenkins/userContent/

sudo chmod 777 /var/lib/jenkins/userContent/sip_test.pcap

sudo cp /var/lib/jenkins/userContent/sip_test.pcap /home/jenkins/sipp-3.3/pcap/archived/sip_test$(date +%F-%T).pcap

Ok, so this script will run tshark for 201 seconds.  My SIPP tests runs for 200 seconds.  it outputs a pcap file to this directory.  Then I move it to my Jenkins UserContent folder.  I change the permissions of this file so it can be read by other users. Finally I copy this file, and change the name of it to a date/time stamp and drop it in an archive.

Update this script to fit your specific needs and directory structure, then run it. Make sure it works.

Jenkins Packet Capture Job

Back in Jenkins, create a new Jenkins job, just like before.  Give it a name like "Capture PCAP."  This we'll run a SSH script just like before. But instead of running SIPP, we're going to call that shell script (i.e. sudo /somepath/yourscript.sh)

Ok, back at that parent job I just spoke about... remember, the job that just kicks off jobs?  Scroll down to the bottom and click Add Post Build Porcess and choose to build a project(s).  In the field input your SIPP project followed by a comma, and the job you just made to run the tshark script.

Run the Parent job, and verify that you get the pcap in your workspace.

Linking the PCAP to Jenkins

One plugin we installed for Jenkins was called Side Bar.  This lets us make our own Left Nav links.  Go ahead click Jenkins / Manage Jenkins / Configuration and in the section "Additional Side Bar Links" click add.  If you want to use your own icon, use the uploader to upload it. it will by default upload it to Jenkin's userContent folder.  It will provide you the path after it uploads.

For the Link Text, input whatever you want (I.e. "Latest PCAP File")

For the Link URL, point to the PCAP file location on your server.

Linking to Archived PCAPs

If you want to link to older PCAPs, well you would have had to do the script bit I did above, where I copy the pcap to a archive folder.  Hopefully this folder you copy to is also the same folder used in your Project workspace.  

That way you can go to the project workspace and point to your archive folder.  You do that by clicking on the project, then "workspace" and click the file structure till you get to your archive folder.  Copy that URL. Use that URL as your link.

Adding a Downstream Job to do Graphing and Clean Up

Ok, now that you got sipp to work, it should be dumping a csv file to your workspace. Great. lets graph that.

Creating a script to do Charting

Back on your Jenkins server, go to your workspace you're using for the other jobs (i.e. the folder with your sipp scenarios.

Create a new file

In that file enter something like this:

clichart -cfvl 0,1 *_rtt.csv -o /var/lib/jenkins/userContent/charts/chart1.jpg

cp /var/lib/jenkins/userContent/charts/chart1.jpg /home/jenkins/sipp-3.3/charts/sipcalls$(date +%F-%T).jpg

convert /var/lib/jenkins/userContent/charts/chart1.jpg -resize 510 /var/lib/jenkins/userContent/charts/chart1-s.jpg

mv *.csv archived_csv/

This runs clichart and picks the correct column of the default output of SIPP.  so keep the parameters.  I tell it to use whatever csv file is in the folder, and make a jpg of the results chart.  Remember our previous script auto archives csv files after each run... so only one csv should be there at a time.

Then ti copies the jpb to an archive and makes  a thumbnail using convert.  Convert is a special call from another command line tool that I forget the name of... 

Jenkins running your script

Using the process earlier, create another Jenkins job.  This time give it a name like "Graph Data."  In the job details, check off "Execute Shell script on remote host using ssh.  Make sure the SSH site is correct and in the Pre build script do:

cd [to your directory with the chart shell script]

sudo ./[your chart script.sh]

Save...

Getting Graphs on Jenkins Dashboard

There's a plugin we installed for Jenkins called Dashboard. if you Edit the View of your Dashboard or Main Tab, and scroll down you'll see some options for Dashboard Portlets... You can have "top, left, right, bottom" portlets... and one portlet option is IMAGE.  Choose Image and link to the hard link of your jpt graph from clicharts.

Run the Parent....

you should see:

1. Parent calls two jobs: SIPP and tshark
2. When those jobs finish up, a downstream job charts the data and clearns up csv files in the main folder
3. All images you've hard linked to, should work to show graphs.
4. All PCAP's you hard linked too should show PCAPS
5. your PCAP archive links should now show the archives

My SIP Testing Framework

After reskinning Jenkins, adding in some more functionality, and graphing capabilities, the SIP testing tool looks like this:

Jenkins: Error opening terminal: unknown.

I do a lot of shell scripts called from Jenkins.  Most of the time, I am logged in as a specific user.  In one case, I was logged in as me, to do a sudo command.  I got this error in the response:

Error opening terminal: unknown.

To fix it, I found a site online:
https://bbs.archlinux.org/viewtopic.php?pid=739747

The fix for me was to edit my user's .bashrc file and add:
export TERM=xterm

After that Jenkins ran the script fine.

Chrome Webdriver and Coordinate Errors

If you get coordinate errors with Chrome Webdriver, the most likely culprit (that I've found) is that this is due to a UI element obstructing the view of something you are clicking.

Yes, it's weird.  Normally webdriver would say "click this ID/Name/Class" and it clicks it no matter if a window is up or not.

But with Chrome Webdriver it's a bit different.

Take the screenshot below for example:

In this test, I first had Chrome Webdriver click on that menu.  that menu stays up when webdriver runs (not in a regular Chrome session), so then when it comes time to click "Save" it will throw an exception:

Element is not clickable at point (109, 604). Other element would receive the click:

The trick is to get all  your UI elements back away so you can send that click.  In my Case I either don't pull that particular menu, or I click it again to close it.

Buidling Automatic Graphs of SIPp SIP data

I finally got some automated SIPp results to look like this:

The screenshot above is my Jenkins install.  Each time I run this sip test of 2,000 calls it updates the graph there dynamically.

Here's how I did it.

I create a "Free Style" jenkins job.
In that job, I basically create a call to run a shell script.  The script is the SIPP call.  It has all the parameters for hitting my target box, phone number, and the number of calls.

One parameter on the sipp call is to output the RTT (-trace_rtt) - so after the test a csv file will be in the folder.  I also change the delimter from semi-colon to comma (why do they default to semi-colon???)

I set Jenkins to then fire a downstream job.  This job is another free-style job. it runs a shell script that basically runs CLICharts.

What is CLICharts?  CLICharts is amazing.

To install it, check out:
http://clichart.sourceforge.net/docs/quickstart.html

Basically I created a shell script that runs CLICharts headlessly.  Basically it's like this:
sudo clichart -cfvl 0,1 *rtt.csv -o /var/lib/jenkins/userContent/charts/chart1.jpg

That tells clichart to not care about dates (since I dont have dates it would understand) and tells it to only care about columns 0 and 1.  Finally to output this chart to a folder in userContent (oh yeah, I'm using Simple Theme plugin for Jenkins so this userContent folder is used by that to link to my own graphics.)

Downstream Jobs

So the Downstream Jenkins job just runs shell script.

On a tip I saw online, I used the main project, and edited it to add my own img src tag. This img tag points to the /chart1.jpg above.  Each time the test is run, it updates the jpg and therefore the page has a new graph soon after the test is run.

I also  set up a post build action on this Downstream job... this one emails me and others the csv file and the jpg.

Finally, after the test a second downstream job is kicked off to simply move the csv files to an archive folder (so only one rtt.csv file exists at a time in the main folder.) 

Other Set Up I had to Do

I had to do a few things to make this work... like add the ssh plugin to Jenkins. Even though I'm not
ssh'ing, the ssh plugin sets the user I'm sudoing as on the Jenkins box.  That user was updated in the /etc sudoer's file (via visudo) so that NOPASSWD:ALL was set for that user (so no pass prompt.)

Jenkins running shell scripts as sudo

I was getting this error:

sudo: sorry, you must have a tty to run sudo

To get Jenkins to run scripts that require sudo access, I had to do the following in the sudoer's file:
1. Fist in /etc running visudo I had to update the user Jenkins sudo's as to [user] (ALL)= NOPASSWD:ALL
2. Second I had to turn off the requiretty.  You can do t his per user, or global.

In Jenkins I added the SSH plugin and even though I'm not actually ssh'ing it seems to use this ssh user/pass/key to run sudo.  I set that ot the same host i'm on and the user i want Jenkins to use, that's in the sudoer file.

Jenkins and Cucumber Reports

I've provided details on this set up, but here's a visual walk through:

My Web Automation Jenkins looks like this the image below.  Each job listed (Announcements, Call Forwarding, etc.) is a specific feature file that Cucumber will run - so each entry runs a multitude of tests.  Each Test is sent to a Selenium Grid hub and is farmed to run on one of several VM's.  This simultaneous testing brings down the overall end to end duration significantly (from 45min to 6min) :

When you mouse over a job that has run, it will offer the reports view in the menu, like so:

Clicking on the Cucumber Reports link will first present you with a summary of results:

If you click the test link in the Feature Statistics table, it will break down more details on what failed and passed: