How I built My SIP Automation Testing Framework from the Ground Up

This is a pretty long and detailed (long) blog post.  I'll try and break it out later to smaller posts. I've tried to capture all the details I can, so that anyone can set up something like this relatively easily.

My current SIP Automation Framework, uses all open source tools to:

• Generate performance tests with our SIP framework
• Generate reports
• Generate Graphs
• Capture PCAP's during the 

A Tour Around the Framework

Dashboard

My Dashboard looks like this.  I reskinned Jenkins and gave it a bit of a visual polish.  I also added some new links on the left "Get Call Control Logs," "Latest PCAP," "Archived PCAP" and so forth.  In the main section the different SIPP preset tests are listed, with their failure/pass criteria.  At the bottom of the main section are some visual Graphs of the SIPP results.

Left Nav

The left nav has the new items mentioned above.  Call Control is a button to grab the logs off one of our servers.  Automation Home, goes to the QA Automation page.  Latest PCAP loads the last Packet Capture done with tshark.  Archived PCAPs points to a repository of PCAPs.

Main Body

The Main Body lists the jobs for the project.  In this case, "jobs" equates to a preset SIPP scenario with command line parameters.  It also shows the current pass/fail value in the first column, and the history of pass/fails on the far right column.

Bottom Graphs

The bottom graphs are utlizing a plugin to display JPG images. I'm creating the graphs with CLICharts, during the test run.  Jenkins points to these jpg's via the plugin.

Test Detail Page

Inside a SIPP Test (or job), I've modified it a bit to  have a larger graph, and some useful links.

The links are, Historic Data (which is Historic Graph data) and PCAP Captures (historic PCAP captures.) These links basically point to Workspace locations, so Jenkins is already set to show you a file structure like this: 

Tools Used

The tools I used were:

• Jenkins
• SIPP
• JMETER
• CLICharts
• Tshark/Wireshark
• a bit of Shell Scripting

OS Used

I started on Windows, migrated to OSX and finally ended up putting the whole thing on a Centos VM.  OSX couldn't keep up with the pace of SIPP and performance testing for our needs. I would get spikes.  This could be the machine I was using, but either way I didn't want the framework to be running on a machine in my office. 

I opted for a blade server hosting a Centos VM for me.  It's very fast and the SIPP performance spikes went down drastically... indicating it wasn't code related, but test machine related.

Jenkins

I'm using Jenkins CI to maintain all the SIP tests. My reasoning for this was two fold.  

1. SIPP was going to be used as our main SIP test/performance tool. It uses XML and Command Line params.  I wanted to capture test presets that had both of these. Jenkins will let me build a job that runs a SIPP command line with params i need, and specify the scenario file. So everything is maintained together
2. Using Jenkins CI was useful for us, since our developers use Jenkins, they could send a command to run the SIP tests once a build is ready.

To install Jenkins, I've installed it on Windows as well as OSX... When I went with the blade install I had our network admin install Jenkins CI there... I'm assuming he did a yum install jenkins  but i'm not sure.  

Either way you can find more info on installing Jenkins from: http://jenkins-ci.org/

SIPP

To install SIPP on CentOS, I just did a sudo yum install sipp

SIPP is a robust Open Source command line tool to generate SIP calls and traffic. It has two main aspects:

1. Scenario Files
2. Command Line Parameters

SIPP is one of the most complex command line tools I've used.  Hence the need to capture the command line params tha tare useful... as there are so many.  

Scenario files are just XML files that build packets.  You are litterally building out a packet in XML, and then saying what you expect as a response.  

SIPP comes with some default scenario files.  Unfortunatley those scenario files needed to be modified by me to work with our team's development.  There are certain responses we send, or do not send, that don't match the default scenario files.  

I won't get into writing your own scenario file, as I think it's outside the overall scope here of building the framework.  I'll post more on the scenario file creation in a later post.

For more information on SIPP, check out: http://sipp.sourceforge.net/

JMeter

To install Jmeter on Centos, I just did a sudo yum install for apache jmeter

I  have a specific need to update my test framework before SIP Load testing... to do this I use JMeter to quickly send some POST or PUTs to end points to set up the data I need. I think this is uncommon for most people doing SIP testing - so I won't cover this aspect in this post.

CLICharts

To install CLICharts on Centos, I just did the yum install.

SIPP will output it's performance data to a CSV file, if you choose.  I found a very useful unix tool called CLICharts.  CLICharts will take a csv file and do all sorts of analysis to it.  In my case, I use it to generate graphs at the end of a test. These jpg graphs are put in a folder that Jenkins points to and references with hard links.  This way each test run, updates the Jenkins page with the latest graph of reuslts.

For more information on CLICharts check out: http://clichart.sourceforge.net/docs/quickstart.html

Tshark/Wireshark

This was a bit different... you need to do a yum install for wireshark-gnome.  That will install a version of wireshark that is usable from the command line (just doing a yum install wireshark will install it in /sbin and not /usr/bin, so you can't just use it as a commandline)... and of course you get the tshark command line tool.

I use this tool to run during the SIPP test.  Since I know the total duration of a SIPP test, I set the default duration of the Packet Capture to the same duration.  Basically I wrote a shell script that calls tshark, sets a duration, and outputs a file.  The file is also copied to a archive folder and has the date/time stamp added to the file name.  These can be referenced by Jenkins as you'll see further in this post.

Putting It All Together

So if you have all the tools above installed, lets put it all together and get a SIP Automation Framework up and running!

Jenkins Plugins needed:

In Jenkins, install these plugins (SSH Slaves, Sidebar Link, Green Balls, Dashboard View, and if you want to restyle the CSS of your Jenkins, get the Simple Theme plugin.)

Once you've installed those Jenkins plugins, restart Jenkins.

Set Up SSH Slaves

Go into Jenkins / Configuration

Scroll down to SSH  remote hosts

Fill this out with the host that is running Jenkins, and you should have sudo access to this box.

You'll be using SSH to sudo your commands, even though you aren't "ssh'ing" anywyere, this is how Jenkins will log in as you to sudo commands.  

Setting Up A SIPP Job in Jenkins

First, open Jenkins... by default it's your localhost:8080.  

When you have Jenkins up, click "New Job"

Choose "Build a free-style software project"

Give the project a name and click OK.

Click "Advanced" button on the Advanced Project Options.

Then check "use customer workspace" and put the path/directory to your sipp scenario files.

Check off "Execute shell script on remote host using ssh"

in the SSH site dropdown make sure it's the same one you input on "Set Up SSH Slaves" above.

in the Pre build script, type:

cd [your directory to your sipp scenario files]

sudo sipp -s [service/number you want to dial] -sf [path to scenario xml file] -m [amount of calls you want to run] -mi [your ip of the jenkins server (media IP)] -d [pause duration override, if you need it] -trace_rtt -trace_err -stat_delimiter ,

The reason you need to sudo sipp here, is because if you are sending pcap audio, you need to use -mi which creates a live socket. You can't create a live socket unless you are sudo'd.

If you run this, it should work.

Adding tshark PCAP captures

I use a parent project, it kicks off the above as a downstream. Meaning, I have a parent project called "2000 calls @10 calls per min to Integration," that job just kicks off a downstream job... two actually.  The first is what we just created above (the SIPP job) and the second job it kicks off simultaneously is the tshark pcap capture.  They run at the same time.

How you do this is simple:

On the Server:

create a shell script in the folder with  your scenario files.  Input something like this:

sudo tshark -a duration:201 -w sip_test.pcap

sudo mv sip_test.pcap /var/lib/jenkins/userContent/

sudo chmod 777 /var/lib/jenkins/userContent/sip_test.pcap

sudo cp /var/lib/jenkins/userContent/sip_test.pcap /home/jenkins/sipp-3.3/pcap/archived/sip_test$(date +%F-%T).pcap

Ok, so this script will run tshark for 201 seconds.  My SIPP tests runs for 200 seconds.  it outputs a pcap file to this directory.  Then I move it to my Jenkins UserContent folder.  I change the permissions of this file so it can be read by other users. Finally I copy this file, and change the name of it to a date/time stamp and drop it in an archive.

Update this script to fit your specific needs and directory structure, then run it. Make sure it works.

Jenkins Packet Capture Job

Back in Jenkins, create a new Jenkins job, just like before.  Give it a name like "Capture PCAP."  This we'll run a SSH script just like before. But instead of running SIPP, we're going to call that shell script (i.e. sudo /somepath/yourscript.sh)

Ok, back at that parent job I just spoke about... remember, the job that just kicks off jobs?  Scroll down to the bottom and click Add Post Build Porcess and choose to build a project(s).  In the field input your SIPP project followed by a comma, and the job you just made to run the tshark script.

Run the Parent job, and verify that you get the pcap in your workspace.

Linking the PCAP to Jenkins

One plugin we installed for Jenkins was called Side Bar.  This lets us make our own Left Nav links.  Go ahead click Jenkins / Manage Jenkins / Configuration and in the section "Additional Side Bar Links" click add.  If you want to use your own icon, use the uploader to upload it. it will by default upload it to Jenkin's userContent folder.  It will provide you the path after it uploads.

For the Link Text, input whatever you want (I.e. "Latest PCAP File")

For the Link URL, point to the PCAP file location on your server.

Linking to Archived PCAPs

If you want to link to older PCAPs, well you would have had to do the script bit I did above, where I copy the pcap to a archive folder.  Hopefully this folder you copy to is also the same folder used in your Project workspace.  

That way you can go to the project workspace and point to your archive folder.  You do that by clicking on the project, then "workspace" and click the file structure till you get to your archive folder.  Copy that URL. Use that URL as your link.

Adding a Downstream Job to do Graphing and Clean Up

Ok, now that you got sipp to work, it should be dumping a csv file to your workspace. Great. lets graph that.

Creating a script to do Charting

Back on your Jenkins server, go to your workspace you're using for the other jobs (i.e. the folder with your sipp scenarios.

Create a new file

In that file enter something like this:

clichart -cfvl 0,1 *_rtt.csv -o /var/lib/jenkins/userContent/charts/chart1.jpg

cp /var/lib/jenkins/userContent/charts/chart1.jpg /home/jenkins/sipp-3.3/charts/sipcalls$(date +%F-%T).jpg

convert /var/lib/jenkins/userContent/charts/chart1.jpg -resize 510 /var/lib/jenkins/userContent/charts/chart1-s.jpg

mv *.csv archived_csv/

This runs clichart and picks the correct column of the default output of SIPP.  so keep the parameters.  I tell it to use whatever csv file is in the folder, and make a jpg of the results chart.  Remember our previous script auto archives csv files after each run... so only one csv should be there at a time.

Then ti copies the jpb to an archive and makes  a thumbnail using convert.  Convert is a special call from another command line tool that I forget the name of... 

Jenkins running your script

Using the process earlier, create another Jenkins job.  This time give it a name like "Graph Data."  In the job details, check off "Execute Shell script on remote host using ssh.  Make sure the SSH site is correct and in the Pre build script do:

cd [to your directory with the chart shell script]

sudo ./[your chart script.sh]

Save...

Getting Graphs on Jenkins Dashboard

There's a plugin we installed for Jenkins called Dashboard. if you Edit the View of your Dashboard or Main Tab, and scroll down you'll see some options for Dashboard Portlets... You can have "top, left, right, bottom" portlets... and one portlet option is IMAGE.  Choose Image and link to the hard link of your jpt graph from clicharts.

Run the Parent....

you should see:

1. Parent calls two jobs: SIPP and tshark
2. When those jobs finish up, a downstream job charts the data and clearns up csv files in the main folder
3. All images you've hard linked to, should work to show graphs.
4. All PCAP's you hard linked too should show PCAPS
5. your PCAP archive links should now show the archives