Automating SIP Testing with SIPp, FreeSwitch and Cucumber

SIP testing is pretty new to me.  In fact, 3 months ago, I didn't know what SIP was and I knew little of VOIP.  Today, I work at a company who's business is phone routing, virtual PBX, fax routing, VOIP, SIP, etc.

While I was able to set up the Front End web app automation, the development leads wanted me to also start building out a framework to test the SIP calls and the PBX software (FreeSwitch.)

Problem:

Create an automated framework that can be used to load test and functional test FreeSwitch.

Analysis:

I looked at a lot of opensource solutions, I was hoping for FS_Spec and some other ruby based solutions to work. But I had nothing but issues with them. In fact most software in this realm hasn't been updated since '09 or so.  In my research I did find one Open Source tool that was highly rated all around:
SIPp.

SIPp even comes pre-installed on the security distro of Linux (BackTrack.)  SIPp is pretty powerful, but it has a few drawbacks:

• It's not geared to functional tests. SIPp is really a load engine.  
• The reporting isn't great.  
• Can't run a suite of tests out of the box
• Doesn't have built in features to test simultaneous UAC and UAS (sender/receiver) - it expects a person to have multiple terminal sessions running, and one running a UAC and the other a UAS. 

 The above issues, broke down to too much human intervention.  What was desired, was something similar to the front end automation: you run a functional test, it either passes or fails, and the results are gathered for the test suite.

Solution:

I started with SIPp as the solution for doing the bulk of the tests, and then worked to build a framework around it to run the tests in an automated fashion.

What I came up with was Cucumber running SIPp functional and Load scripts.
The end result is, I just run:
cucumber
It runs all the tests (i.e. a few seconds later, it's driving traffic to my desk phone... Desk phone is ringing... yay.) Test fails/passes are tallied and the results are output to screen and file.

In the end, Cucumber is optional.  It's not a requirement here. For me, I'm only using Cucumber to tally the results.  I may opt to drop it all together as it has a high overhead.  The tests would be faster if just a collection of Ruby/Groovy or Scala scripts.

How it All Works:

To start it off, I had to see how the development servers work (i.e. FreeSwitch.)  So first things first, if you're going to test SIP, I recommend installing the Virtual PBX you are using in development (Asterix, FreeSwitch, etc.) on a local QA box.

Other tools you'll want installed on the same box are:
Wireshark
SIPp
SoftPhone for testing (I.e. X-Lite or "telephone")
Scripting Language (Ruby/Groovy/Python/Scala Scripts)
Optionally: Cucumber 
Optionally: Fully Functional Call Control with virtual numbers

I recommend only using OSX or a Linux distro for this set up.  Getting this to work on a PC is too difficult. 

You'll want to spend time with FreeSwitch and WireShark to see  how the packets are sent back and forth... and how to read the exchange.  Then you'll see how to write the tests.

Writing the SIP tests all boils down to two steps:

• The main test is going to be written in SIPp.  It's a command line SIP load tool.  But you can constrain it to only drive one test, for functional testing.  
• Use a Scripting Language (or Cucumber) to run the SIPp tests you wrote.

I detail each step below...

Details: Setting Up FreeSwitch

The easiest way to do this is on OSX or Linux.  Just do a git clone from the FreeSwitch repo... you can follow the instructions here: http://wiki.freeswitch.org/wiki/Installation_and_Tips_for_Mac_OS_X
Make sure you also do the step to make your audio files.

On Mac you'll prob need to bring in the libjpeg on my mac... to do this i used brew install libjpeg.

Configuration Setup:

This is where it gets a little tricky with FreeSwitch set up.

1. You'll need to know or change the default password... this is found in the vars.xml file  in /conf inside of the Freeswitch directory.  The instructions to do this are actually in the comments of the vars.xml file. 
2. Next, you'll want to change  /conf/sip_profiles/sip_profiles/internal.xml to update the inbounx-acl is localnet.auto 
3. As a tester, you're probably going to be  using an Internal Profile (meaning your machine's internal IP) for testing.  Developers will probably not use that set up. For testing though, you probably will.  When pointing to your Internal/local IP, it will bind that to a specific profile called "internal."  By default that folder is empty.  What I did, to make it quick and easy was this step:  I copied the files in the Default profile to the Internal Profile:  /conf/directory/default/* to /conf/sip_profiles/internal/* 
4. What you just copied over are actual user accounts and extensions.  They need to be modified.... Edit them individually and change the reference of "default" to "internal" and save them out.

Start Up FreeSwitch

Ok lets start it up, go to the freeswitch bin directory (i.e. /usr/local/freeswitch/bin) and type ./freeswitch

Once it's up, type: sofia status
If everything worked out, you should see 4 rows in a table display.  One will be an Alias, and it references the Internal Profile.  This Alias should have your local IP.   

You can tab for methods/commands... like this, type sofia, then hit TAB.  you'll see all the available options.  For example: sofia global siptrace on  is a very useful command.

Register with your local FreeSwitch

You can now launch a softphone (like X-Lite) and set it up to talk to login as a default user you copied over (i.e. 1000) and use the default password (what's in the vars.xml file) and then for domain set your local IP.

If everything is set up correctly, it should REGISTER on the FreeSwitch.

If you have a second softphone (i.e. "telephone"), you can also register as a different user (i.e. 1001) and then call the other user (i.e. 1000).

Once you make a call you should start seeing the FreeSwitch server display a lot of activity.  If you  have: sofia global siptrace on running, it will categorize the events.  You should see events like: INVITE, ACK, BYE.

Details: Using Wireshark

Wireshark Installation Overview

Wireshark is incredibly useful here.  If you are installing Wireshark on a MAC, you'll need to set up X11... on Mountain Lion, you'll need to use XQuartz.  Once installed, you can then bind Wireshark to XQuartz/X11 - I wont go into all the setup details here, but you'll need to restart your Mac for the changes to take effect (or logout/login.)

When Wireshark is up and running, you'll want to listen to the interface: lo0 for  your local traffic.

What can Wireshark show me?

What can't it show you.... it shows everything.  It not only records all the packets, this tool also has a whole Telephony menu.  From there you can build VOIP diagrams of the call flows. This is very useful in seeing what is sent and expected back for each test.  Wireshark can also copy the RTP stream!  That means it can hear the audio  you send, and verify the audio sent (say a recorded wav) is what's received!  

Using Wireshark to Generate a Flow Diagram

Go ahead and launch XQuartz, then Wireshark. 

Now, click on the interface lo0 and start recording.

Make sure FreeSwitch is up and running and that it is taking local traffic (i.e. sofia status shows alias and internal profiles up and running)

Make a call from one softphone to the other

Stop the Wireshark recording.

At first you'll see a dump of all packets captured on that interface.  

Now, go to the Telephony menu, and click "VOIP."  That will load only the VOIP related packets.  

Click the button, Select All.  

Now click Make Flow Diagram.

From that window, save it out.  In OSX the initial display usually doesn't render well. but once you save it out, it looks great.

This flow diagram will show you everything going on in a call, and removes all the uneeded data, making this more human readable. 

Using Wireshark to listen/verify the audio of a Call

Go ahead and launch XQuartz, then Wireshark. 

Now, click on the interface lo0 and start recording.

Make sure FreeSwitch is up and running and that it is taking local traffic (i.e. sofia status shows alias and internal profiles up and running)

Make a call from one softphone to the other

Stop the Wireshark recording.

Similar to before, click the Telephony menu.

Click the RTP sub menu and the "Show Streams" option.

You'll see packets here on each line item. click through on one, and click "Analyze."

You'll get the audio of the call/transmission. 

Using Wireshark Programmatically

My goal is to use it in automation testing.  To do this, I'll be calling the Wireshark utilities from the command line and getting back results in the command line interface.  

Details: SIPp

Now lets look at SIPp.  If you're new to SIP, you can think of SIPp as a command line version of JMeter.  It was designed to drive lots of traffic (hundreds or thousands of calls a min) to a PBX.

However, SIPp can be set up to run just one call through a functional scenario.

To install SIPp, just follow the instructions over at: http://sipp.sourceforge.net/

Pre-Installed Tests

SIPp has built in tests.  These include UAC, UAS, UAC_PCAP, REGEX, and more.  The whole list is off their main site.  If you don't know what UAC or UAS is.... you should probably read up on SIP.  But to just basically summarize it, SIP is a P2P system.  At any given time, one user is both the server and client.  They are sending data and receiving.  so UAC and UAS is a client server scenario.  

If you just want to see SIPp work, you can type:

./sipp uac [your local ip]

There's no Params, so it will run it's default load for the UAC test. 

If you want to see what the built in UAC test is actually doing, just do this:

./sipp -sd uac >> [filename]

Working with Parameters and SIPp

There's some sites out there with useful cheat sheets for SIPp... here's one such site:

http://tomeko.net/other/sipp/sipp_cheatsheet.php?lang=pl

That site also has some tests they wrote, which you can download and see how they work.  They also have tests that utilize CSV files which contain a set of users you want in your load.

For my tests, I use a lot of functional tests, so I don't want to slam 10,000 phone calls against the server. Instead I want to run one call, one time, to do something specific:

./sipp -s [phone number configured on call control to redirect to my desk phone] -m 1 -l 1 -recv_timeout 6000 -sf [path to my test xml file] [IP of our integration / uat freeswitch]

So here's what those parameters are saying:

Run SIPP against the QA Integration/UAT env. and dial that number configured in Call Control (-s means service, which could be a extension, phone number, etc.) -m tells it to run once -l tells it to run only one test at a time.  -recv_timeout 6000 says to run this test for only 6 seconds.  Finally -sf is the path to my test.... and of course i end it with the IP of the FS box.

Writing your Own SIPp tests

I wont go into too much detail here.  I'll save it for a future blog.  But to start with, you can take an existing built in test, and export it using the -sd option I mention above, or you can review some tests other people are writing out there and have put on github or elsewhere.

Automating All This

Now to automate this.

Initially I used straight Ruby for this... but then I decided having Cucumber run the tests, gives me the added benefit of compiling test results. I'm on the fence if that's necessary at this point or not.

Basically, I found it too hard to use an existing API with what we use here in house as our PBX.  I tried SIPr, FS_Switch and a variety of other tools. Most of these haven't been touched in many years and have known bugs flagged against them.  Rather then fight with that, I choose to use SIPp.

SIPp is an industry standard, but it doesn't have an open API.  To automate this, I use SIPp as is, and build the shell around it.

It's really very simple:

Create a script to drive all the SIPp tests.  

Assert via REGEX the output for expectations.

Mark pass / fail.

So I'm basically doing the core work in SIPp.  I write out my test, with the parameters to call it effectively.

Then I use Cucumber (you could use straight Ruby or Groovy or Scala, Python, etc.) to do this:

Given /^a SIP call is made to another SIP account$/ do

   @calling = %x{/Users/brainwarner/sipp-3.3/sipp -s 1000 -m 1 -l 1 -recv_timeout 6000 -sf /Users/brianwarner/sipp-3.3/calling.xml 10.98.1.1}

end

Then /^a SIP 180 ringing should return$/ do

   assert @calling.include?("180")

end

The above is a simple set of two steps in a step definition file.  Basically it's saying ok, I'm going to execute sipp with my sipp test, against the specific IP of my UAT server and I'm using those parameters to make sure it's just one test, run once and ends after 6 seconds.

Then I'm calling a Ruby method "include" which is like a REGEX, to search the output that's in that class variable @calling for the specific ID I expect for calling.  

Why not use REGEX? well every regex I tried, passed even when it failed. Initially i would search the output for a invalid value (i.e. 777iwin!), and it would always pass, no matter how I wrote the regex!  I ended up using this method because it actually works well.

Simple. But it works.

Future Steps In Automation

So this is the Framework.  

You could easily see how that simple test could be transformed to something more impressive:

%x{....} calling one UAS user session

%x{...} calling another session in UAC test mode... and then have them talk to each other.

Since we're calling 3rd party command line utilities, you can also run the command line wireshark! meaning at test time you could use Wireshark to sniff packets, or verify RTP audio.... now that's cool.