Project: Making a SIP GUI test harness with Grails and SIPP - Part III

Part I, discussed this project and walked through setting up SIPP scenarios and Grails (making a basic Grails project.)

Part II, discussed adding in a Service Layer to do the SIPP call and added some call validation.

Now What?

So now what? Well you can add on to your test harness. You can add other domain classes, controllers and services as needed.

For example, what if you wanted to have a different interface, where a user just enters a phone number and a proxy, and the service not only uses sipp but iterates through a list of carriers?

You could do that by adding in a new Domain object, and generate controllers/views... then make a service to handle the logic.  Like in the service you might define a list:
def carriers = ['Sprint','TCast','Verizon'] and iterate through it with a for loop, like:

for (c in carriers) {
     def carrier = c
     def dialNum = "sipp -s ${dnis} ${ipprox} -sf ${carrier}_outbound_call.xml -l 1 -m 1 -d 5000".execute().text
 .... 

Or what if you wanted a Service that would handle some audio analysis, so you could turn the audio of the phone call to a string and compare it against an expected value?  

You can keep adding onto the project with more and more cases.  

In the end, the code is centralized, and we have a UI for free. This UI can be used by us or other testers.

Tomcat

So lastly, there is the aspect of deployment.  For QA, we usually run our own servers and environments.  In my case, I have a variety of Linux Centos VM's available to me.  I basically put Tomcat up on one of those guys. 

I'm assuming you got Tomcat up and running, as I wont cover setting it up in this post.

To deploy a Grails application, you basically do what you would do in Java - you create a WAR file and drop the WAR file in the Tomcat webapp folder. 

So there's two ways I can think of doing this.... the hard way and the easy way.

The Hard Way

The hard way, basically means you do it the long way.  In this method, you simply open the grails command line in Intellij and type:
grails war
and hit enter or click ok.  

It will generate a war file and drop it in your project under /target.

Then you would SCP/FTP that up to your Tomcat server

That's not so hard.  But there's an easier way....

The Easy Way

First you need to set this up.  After that it's easy.  

The easy way is easy, because once it's set up, you don't do any FTP/SCP or WAR Generation. You just run a command and it does all that.  

To get here, you will first need to do a few things - 

On the Tomcat server, edit the tomcat-users.xml file.  In that file you want to add a role group (if you don't have it) called "manager-script."

Add a user with the role  manager-script... jot down the username and pass you gave it... you'll be adding that in the Grails configuration.

In Grails 2.2.4 Tomcat deployment plugin is already installed.  What you need to do is configure it so that when you deploy via your IDE, it will do all the WAR generation and SCP to your Tomcat server, for you.

**NOTE: I'm having issues with Grails 2.3Pre Release at this time, with Tomcat deployments**

In Grails, you open your /conf/Config.groovy file and add this:

/**
 * TOMCAT DEPLOY CONFIG
 */
tomcat.deploy.username="grails"
tomcat.deploy.password="pass"
tomcat.deploy.url="http://[your Tomcat server]/manager/text"
/**
 * END TOMCAT CONFIG
 */

That's it.  

Now to deploy to Tomcat, you do this command: grails tomcat deploy 
from the command line within the IDE. 

Project: Making a SIP GUI test harness with Grails and SIPP - Part II

In Part I, we set up SIPP, SIPP Scenarios and created a Grails application with a controller and a domain class.

What we'll start with in Part II is the creation of a service.

But first... what is a service?

A service is a piece of functionality that coordinates logic with the Domain layer. 

We'll use a service to run the all the logic that's needed for the application.  By separating the code out into a service, we move the logic into a layer that isn't the same as the controller.  If we didn't do this, the controller would be handling all the functionality.

Grails looks to split the core functionality and logic into a separate layer (i.e. Service layer) and leave the controller as handing the routing.  In this model of development the controller is left to return values, render values or redirect to a specific view or other controller.

So again, the 3 R's of controllers are:

• Render
• Redirect
• Return

Render is not a common use of a controller.  Returning values and Redirection is the most common activity of a controller.

So now that that's out there, what do we need to do with this service?

We need to put the logic in there, that handles the user input.  Remember our tool offers a UI that a end user can hit, pull up a form and submit a phone number, carrier and proxy.  We want those values to plug into SIPP, and let SIPP make the call and we'll validate if it's successful or not.

Service

To create a new service, in Intellij you can right click the project in the project tree, and then choose 

New | Grails Service

Give it a name... like DialerService

Great, now in the project tree, open the services folder.  You should see your new service listed there.  Go ahead and double click it to open it up.

Let's go ahead and add some logic in here. 

def sipp(phone,ipprox,carrier){
    def dialNum = "sipp -s ${phone} ${ipprox} -sf /carrierTests/outbound/${carrier}_outbound_call.xml -l 1 -m 1 -d 5000".execute().text
    [dialNumResult:dialNum]
    }
}

Let's review what's going on here.  

The service has a method called sipp, which takes three parameters (phone number, the proxy ip and the carrier) - these three parameters are the ones being passed in by the end user on the form.  

The sipp method has a piece of functionality defined as dialNum.  dialNum basically runs our SIPP command line call.  What's in quotes is the string we will execute.  The ${} is a gString... or groovy string.  Groovy strings are there to pull in those parameter values being passed through.  The .text() may not seem logical, but I actually want it there.  It will take the output from SIPP (with the call data) and convert that to text for us.  That way if someone says "ok the call went through, but how do I know what it did?" they can have a record of the actual call process.

Lastly, what's that square bracket stuff?

The square bracket is a return.  We are saying set up this variable dialNumResult to have the value of dialNum... and pass it back.

In Grails, the last line of a method is the last action returned.  In this case, we're returning the value of dialNum.  

Controller

Back in the controller, we want to pass the variables we are getting from the user input (via the form) through to the service we made.  

So under def outboundCallInstance = new OutboundCall(params) add these lines:

def carrier = params.carrier

def phone = params.phone

def ipprox = params.ipProxy

Great, so do you see how it's working? We're taking the user input on the form, which is going to the controller... and the controller is routing it to the service.  The service will do it's logic/functionality and return the result.  

But this isn't quite done yet.  We are grabbing the data from the form, sure enough, but we aren't passing it to the service.

Above the save method in the controller add this:

def dialerService

In Intellij, it will automatically know that this is a reference to the service we made and put a special icon next to it in the margin.  

Why did we reference the service in camel case, when it's really named DialerService?  Well that's just how you do it.  We're referencing the service in a class and this is how we do it.  

Below the lines we added (after the parmas.ipProxy line) add this line:

outboundCallInstance.results = dialerService.sipp(phone, ipprox, carrier).dialNumResult

If you set it up right it should auto complete a lot of that for you.  We're saying, hey we defined this service, now lets talk to it... we want the outbound call instance's results to be the dialer service and pass into it the 3 variables: 

• phone
• ipproxy
• carrier

...and we expect the dialNumResult returned (this is the output from sipp)

So it should look something like this:

def dialerService
    def save() {
        def outboundCallInstance = new OutboundCall(params)
        def carrier = params.carrier
        def phone= params.phone
        def ipprox = params.ipProxy
        outboundCallInstance.results = dialerService.sipp(phone, ipprox, carrier).dialNumRes

        ....

There will be other code in there, just leave it as it was generated.

To recap:
We are:

1. Collecting user input on a form
2. passing that to a controller that sends it on to a Service
3. The service does some logic on it and returns a result
4. All this is saved in the db

Validation

If you were to run this, it should make a single call via SIPP to your phone number via a proxy of some sort.  You can test this out by running the application again and filling out the form with valid data for your test environment.

But it's not really validating the phone call.  

First I'll show how to do some simple validation of the SIPP call with some logic in the controller... later on I'll show a different way to validate with some logic in the service.

Back in the controller, let's go to that save method again, and under the outboundCallInstance.results = line, let's add this if statement:

if (dialerService.sipp(dnis,ipprox,carrier).dialNumRes =~ //){
            outboundCallInstance.pass = "PASS"
        } else {
            outboundCallInstance.pass = "FAIL"
        }

This is pretty simple and it won't work. Not yet.  We haven't added the REGEX.

The IF statement is saying, IF the dialer service return =~  (that funny symbol means we're going to use Regular expressions here) // (and what's inside the slashes is what we look for.  If the regular expression matches, then we will assign the string value "PASS" to the Domain Class variable "pass" and if it fails the regular expression, we will assign the string "FAIL" to that Domain Class variable.

If you know SIPP output you know it's difficult to regex on.  This is because SIPP outputs it's results like this:
Successful Call   |0|1
Failed Call         |0|0

That's pretty crazy right?  Because if you make that a map, it would look like this:
Successful Call, 0, 1 Failed Call.

But it's not a failed call, it's a successful call!  Silly SIPP output!

Regex Hell

After many hours of Regular Expression Hell, I created this regex that works for SIPP results:
Successful call\s+([0])\s+1

It basically says, Look for the words Successful call, on the same line, find the 0, and the fond the 1.

That was my solution, yours might be better.

so now the controller should look like:

if (dialerService.sipp(dnis,ipprox,carrier).dialNumRes =~ /Successful call\s+([0])\s+1/){
            outboundCallInstance.pass = "PASS"
        } else {
            outboundCallInstance.pass = "FAIL"
        }

If you were to run this now, it will most likely fail.  Either pass when it should fail, or fail when it should pass. The reason, is the Pipes in the SIPP output.  I've tried a lot of solutions, including escaping the pipes in the Regex, but it still doesn't work right.

To fix it, we need to open the service and add this to the return:

[dialNumRes:dialNum.replaceAll("\\|", "")]

So at the end of dialNum, add a

.replaceAll("\\|", "")

This removes the pipes in the output.  So what's returned is returned without Pipes.

That should be it. It should work just like that. On to Part III (Deployment)

SIPP & Jenkins Details

Per request, I'm adding some detail on the use of Jenkins and how I configured it to run the jobs and retain the details.

Why Jenkins?

First, why did I use Jenkins? Several reasons:

1. Jenkins is a build process, so by having the tests in Jenkins, I can kick off tests when a different job finishes building... i.e. developers push a new version of call control, and that starts the tests running
2. Jenkins allows me to retain historical data easily
3. Jenkins is push button, so anyone can come along and run the job.

Jekins Set Up with SIPP

Regarding my set up, from a high level, I needed Jenkins to kick off two simultaneous jobs... one is Running the SIPP load, and the other is capturing a PCAP during the test.  The PCAP capture may not be necessary for your tests... But if you do want it, you'll need to run the jobs simultaneously.

Simultaneous Jobs

In Jenkins I use the Multi Job plugin: https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin 

You then create a new project/job and set it up as a Multi-Job. On the job details, you  add a build option for a Multi Job Phase.  Then you can add multiple jobs to one phase.

This lets Jenkins run multiple jobs simultaneously.

Why not just have the parent job call the child jobs like job1, job2, etc.? 

the answer is due to collecting pass/fail criteria.  If you just have a Jenkins job kick off other jobs, the parent job will always pass, even if the children fail. 

The MultiJob Plugin will fail the parent job, if the children jobs fail.

Jenkins Running Sipp

To run the SIPP command I have a Jenkins job just for that.  In the Job details, under "Build Environment" I check "Execute Shell Script on Remote Host using SSH" (I believe this is avail with the SSH plugin.)  NOTE: You must define your SSH host and login on the main Jenkins configuration.
In the Execute Shell Script text box, I add:
cd /sipp-3.3
sudo sipp -s [Phone number] [Proxy] -sf /uac_pcap_g711.xml -m 2000 -mi [Proxy media IP] -d 1200 -trace_rtt -trace_err -stat_delimiter ,

That's it.  Now when this job is run, it will cd to the sipp folder, then run the sudo of sipp to call the number using our specific IP for the proxy. 

Packet Capture

In the packet capture job, I do the same thing as the SIPP job, I check off "Execute Shell Script..." but I point to a script I've made... like pcap.sh on the file structure.  In that file, I have a call to run tshark for X seconds and Output the pcap file to a specific folder.

That's it.

Then back to the MultiJob parent, it points to these two jobs, as one part of one phase.

Buidling Automatic Graphs of SIPp SIP data

I finally got some automated SIPp results to look like this:

The screenshot above is my Jenkins install.  Each time I run this sip test of 2,000 calls it updates the graph there dynamically.

Here's how I did it.

I create a "Free Style" jenkins job.
In that job, I basically create a call to run a shell script.  The script is the SIPP call.  It has all the parameters for hitting my target box, phone number, and the number of calls.

One parameter on the sipp call is to output the RTT (-trace_rtt) - so after the test a csv file will be in the folder.  I also change the delimter from semi-colon to comma (why do they default to semi-colon???)

I set Jenkins to then fire a downstream job.  This job is another free-style job. it runs a shell script that basically runs CLICharts.

What is CLICharts?  CLICharts is amazing.

To install it, check out:
http://clichart.sourceforge.net/docs/quickstart.html

Basically I created a shell script that runs CLICharts headlessly.  Basically it's like this:
sudo clichart -cfvl 0,1 *rtt.csv -o /var/lib/jenkins/userContent/charts/chart1.jpg

That tells clichart to not care about dates (since I dont have dates it would understand) and tells it to only care about columns 0 and 1.  Finally to output this chart to a folder in userContent (oh yeah, I'm using Simple Theme plugin for Jenkins so this userContent folder is used by that to link to my own graphics.)

Downstream Jobs

So the Downstream Jenkins job just runs shell script.

On a tip I saw online, I used the main project, and edited it to add my own img src tag. This img tag points to the /chart1.jpg above.  Each time the test is run, it updates the jpg and therefore the page has a new graph soon after the test is run.

I also  set up a post build action on this Downstream job... this one emails me and others the csv file and the jpg.

Finally, after the test a second downstream job is kicked off to simply move the csv files to an archive folder (so only one rtt.csv file exists at a time in the main folder.) 

Other Set Up I had to Do

I had to do a few things to make this work... like add the ssh plugin to Jenkins. Even though I'm not
ssh'ing, the ssh plugin sets the user I'm sudoing as on the Jenkins box.  That user was updated in the /etc sudoer's file (via visudo) so that NOPASSWD:ALL was set for that user (so no pass prompt.)

SIP Testing Basics

This is a walk through on what it took to get a single SIP load test generated, using SIPP.  Getting the first one was hard, so I wanted to document these steps in case it helps anyone else, and for future reference.

Tools

There's not a lot of SIP load tools out there.  For SIP load, there's really one standard that keeps coming up in discussions: SIPP.  SIPP, which is at: http://sipp.sourceforge.net/ is a command line tool that drives various types of SIP traffic to a proxy. There's a lot of potential with this tool, it actually is highly configurable and has quite a few amazing capabilities, as seen in their documentation (http://sipp.sourceforge.net/)

SIPP is also one of the most difficult things to learn. If you are new to Session Initiation Protocol (SIP) and just starting to test it, learning about SIP and SIPP is pretty daunting.  However, I have one advice, stick with SIPP.  I've tried every open source tool out there, and for SIP load testing, SIPP is by far the best.

After I got SIPP scripts up and running, I then started integrating them into Jenkins, and letting Jenkins maintain the scripts and email me the results of each run (more on that in a separate blog post: http://www.continuous-qa.com/2013/02/sipp-and-jenkins.html

SIP Basics

SIP stands for Session Initiation Protocol.  I won't pretend to be an expert on it.  I was thrown into the world of SIP as a need arose for SIP Load testing where I work.

Basically, SIP is a protocol that handles the handshaking and transfer of data for VOIP services... Voice over the Internet.  But this isn't limited to just handling tools like Skype, Teamspeak, etc.  In fact most phone calls, at some point, become SIP calls.  You may pick up a hard line, dial a number, it could route to your PBX and at some or many points in the transaction go to a SIP PBX.

SIP is a standard of packet protocols. It requires certain things to occur in specific sequences.  There is room for some variance, but in general everything works within a standard format.

Simple Call

In a simple call where one phone dials another, and SIP is being utilized, we'd see something like this occur:
12136667777@someplace.com INVITE -----> 1000@somepbx.com
100 Trying <-------- Phone 2
200 OK <----------- Phone 2
ACK ---------------->

In the above example there's some similarities with the Web world... first off the Sip addresses look like emails: 12136667777@someplace.com and 1000@somepbx.com

That's how SIP accounts are registered. The first is an example of a potential phone number... listed on a proxy or pbx.  the other is an example of extension 1000 on a pbx.

The responses also look like HTTP status codes... that's because they are related.  A list of SIP status codes can be found here:
http://www.websitepulse.com/kb/sip_status_codes.html

You'll notice a lot of similarities with HTTP codes... 400 range are errors as are 500, 200's are usually "OK's" and so forth.

Details of a Simple Call

Using Wireshark or other packet sniffers you can grab the packets and see what they look like. Below are some examples of the packets from my local network:

Invite

----------------------------------------------- 2013-02-05 16:52:14:961.441
UDP message sent (505 bytes):

INVITE sip:1000@10.98.6.26:5060 SIP/2.0
Via: SIP/2.0/UDP 10.98.7.36:5061;branch=z9hG4bK-1769-1-0
From: sipp <sip:sipp@10.98.7.36:5061>;tag=1769SIPpTag001
To: 1000 <sip:1000@10.98.6.26:5060>
Call-ID: 1-1769@10.98.7.36
CSeq: 1 INVITE
Contact: sip:sipp@10.98.7.36:5061
Max-Forwards: 70
Subject: Performance Test
Content-Type: application/sdp
Content-Length:   131

v=0
o=user1 53655765 2353687637 IN IP4 10.98.7.36
s=-
c=IN IP4 10.98.7.36
t=0 0
m=audio 6000 RTP/AVP 0
a=rtpmap:0 PCMU/8000

100 Trying

----------------------------------------------- 2013-02-05 16:52:14:961.782
UDP message received [332] bytes :

SIP/2.0 100 Trying
Via: SIP/2.0/UDP 10.98.7.36:5061;branch=z9hG4bK-1769-1-0;received=10.98.6.26
From: sipp <sip:sipp@10.98.7.36:5061>;tag=1769SIPpTag001
To: 1000 <sip:1000@10.98.6.26:5060>
Call-ID: 1-1769@10.98.7.36
CSeq: 1 INVITE
User-Agent: FreeSWITCH-mod_sofia/1.3.13b+git~20130204T173656Z~f3206e5ff1
Content-Length: 0

200 OK

----------------------------------------------- 2013-02-05 16:52:15:2.033
UDP message received [1066] bytes :

SIP/2.0 200 OK
Via: SIP/2.0/UDP 10.98.7.36:5061;branch=z9hG4bK-1769-1-0;received=10.98.6.26
From: sipp <sip:sipp@10.98.7.36:5061>;tag=1769SIPpTag001
To: 1000 <sip:1000@10.98.6.26:5060>;tag=cUcc2K9Z5Ut1F
Call-ID: 1-1769@10.98.7.36
CSeq: 1 INVITE
Contact: <sip:1000@10.98.6.26:5060;transport=udp>
User-Agent: FreeSWITCH-mod_sofia/1.3.13b+git~20130204T173656Z~f3206e5ff1
Accept: application/sdp
Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REGISTER, REFER, NOTIFY, PUBLISH, SUBSCRIBE
Supported: timer, precondition, path, replaces
Allow-Events: talk, hold, conference, presence, dialog, line-seize, call-info, sla, include-session-description, presence.winfo, message-summary, refer
Content-Type: application/sdp
Content-Disposition: session
Content-Length: 187
Remote-Party-ID: "1000" <sip:1000@10.98.6.26>;party=calling;privacy=off;screen=no

v=0
o=FreeSWITCH 1360084930 1360084931 IN IP4 10.98.6.26
s=FreeSWITCH
c=IN IP4 10.98.6.26
t=0 0
m=audio 27004 RTP/AVP 0
a=rtpmap:0 PCMU/8000
a=silenceSupp:off - - - -
a=ptime:20

ACK

----------------------------------------------- 2013-02-05 16:52:15:2.159
UDP message sent (351 bytes):

ACK sip:1000@10.98.6.26:5060 SIP/2.0
Via: SIP/2.0/UDP 10.98.7.36:5061;branch=z9hG4bK-1769-1-5
From: sipp <sip:sipp@10.98.7.36:5061>;tag=1769SIPpTag001
To: 1000 <sip:1000@10.98.6.26:5060>;tag=cUcc2K9Z5Ut1F
Call-ID: 1-1769@10.98.7.36
CSeq: 1 ACK
Contact: sip:sipp@10.98.7.36:5061
Max-Forwards: 70
Subject: Performance Test
Content-Length: 0

SIPP Sample Scenario

SIPP tests SIP by using scenario files to simulate the flow of a call. Then, using parameters passed on the command line, it will run X load, Y times, with Z packet loss %, etc.  So SIPP is working on both a command line and a flat file.

The fat file, henceforth referred to as Scenario file is basically an XML document.  If you are familiar with XML then the Scenario file will look very familiar.  The scenario file will set the constraints of the test.  It will say "i'm sending a packet like this for the invite... now after that's sent i'm ready to accept 100 trying...." and so forth. 

There are variety of built in scenarios and of course, you can make your own. 

Here's what a Scenario File looks like:

<scenario name="Basic Sipstone UAC" >
  <!-- In client mode (sipp placing calls), the Call-ID MUST be         -->
  <!-- generated by sipp. To do so, use [call_id] keyword.                -->
  <send retrans="500" >
    <![CDATA[

      INVITE sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>
      Call-ID: [call_id]
      CSeq: 1 INVITE
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Type: application/sdp
      Content-Length: [len]

      v=0
      o=user1 53655765 2353687637 IN IP[local_ip_type] [local_ip]
      s=-
      c=IN IP[media_ip_type] [media_ip]
      t=0 0
      m=audio [media_port] RTP/AVP 0
      a=rtpmap:0 PCMU/8000

    ]]>
  </send>

  <recv response="100"
        optional="true" start_rtd="Performance" >
  </recv>

  <recv response="180" optional="true">
  </recv>

  <recv response="183" optional="true">
  </recv>

  <!-- By adding rrs="true" (Record Route Sets), the route sets         -->
  <!-- are saved and used for following messages sent. Useful to test   -->
  <!-- against stateful SIP proxies/B2BUAs.                             -->
  <recv response="200" rtd="Performance">
  </recv>

  <!-- Packet lost can be simulated in any send/recv message by         -->
  <!-- by adding the 'lost = "10"'. Value can be [1-100] percent.       -->
  <send>
    <![CDATA[

      ACK sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>[peer_tag_param]
      Call-ID: [call_id]
      CSeq: 1 ACK
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Length: 0

    ]]>
  </send>

  <!-- This delay can be customized by the -d command-line option       -->
  <!-- or by adding a 'milliseconds = "value"' option here.             -->
  <pause/>

  <!-- The 'crlf' option inserts a blank line in the statistics report. -->
  <send retrans="500">
    <![CDATA[

      BYE sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>[peer_tag_param]
      Call-ID: [call_id]
      CSeq: 2 BYE
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Length: 0

    ]]>
  </send>

  <recv response="200" crlf="true"  >
  </recv>

  <!-- definition of the response time repartition table (unit is ms)   -->
  <ResponseTimeRepartition value="10, 20, 30, 40, 50, 100, 150, 200"/>

  <!-- definition of the call length repartition table (unit is ms)     -->
  <CallLengthRepartition value="10, 50, 100, 500, 1000, 5000, 10000"/>

</scenario>

Breaking Down The Scenario

In the above example, there's a flow going on.  It's easier to read in an IDE like  Intellij.  The Invite block looks like this:
  <send retrans="500" >
    <![CDATA[

      INVITE sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>
      Call-ID: [call_id]
      CSeq: 1 INVITE
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Type: application/sdp
      Content-Length: [len]

      v=0
      o=user1 53655765 2353687637 IN IP[local_ip_type] [local_ip]
      s=-
      c=IN IP[media_ip_type] [media_ip]
      t=0 0
      m=audio [media_port] RTP/AVP 0
      a=rtpmap:0 PCMU/8000

    ]]>
  </send>

It's very similar to our real packet from way up above in this post.  There are items in this though in square brackets.  Those items are variables. These are either auto generated or set via the command line parameters.  by default the call_id is auto generated, but some of the other items need to be supplied as a parameter via the command line (such as the service, proxy ip, etc.)

After this invite block we see some xml nodes called recv response.  These allow the test to receive these responses back.
  <recv response="100"
        optional="true" start_rtd="Performance" >
  </recv>

  <recv response="180" optional="true">
  </recv>

  <recv response="183" optional="true">
  </recv>
Say for example we didn't have recv response="100"... and we send an invite.  If we get a 100 response back, the test will fail. It will say "expecting 180 (the next on the test) but got a 100 instead.

I also have a tag there called "start_rtd"  this starts my response time timer. In this test, I'm organizing the response time between the first 100 I get after an Invite is sent, and the 200 OK.  So it's saying, I've sent an Invite, how long does it take to get a 200 OK? 

Further down we have another packet set up for ACK and then for BYE (disconnect.) 

Built In Scenarios

SIPP comes with some built in scenarios... for example, if you download and install sipp, you could go to the sipp directory and do a ./sipp -sn uac [ip of  your proxy] and it start generating sip load on your proxy based on that scenario.

Notice the -sn parameter with the value uac.  The -sn means use a default scenario.  The value is the name of it. it's built it. it's not a flat file. However, if you want to see what uac does, just do this:

./sipp -sd uac   (that will print to screen)
./sipp -sd uac >> uac.xml (that will print it out to an xml doc)

SIPP has several built in scenarios:

• UAS
• UAC
• UAC_Pcap

UAS is a server, it would listen for incoming calls...
UAC is a client sending SIP calls.
In truth, all SIP calls have each party being both the Server and the Client.

UAC_PCAP is a special test, it will send a PCAP (more on this later) to the recipient.  A PCAP is a packet capture.  In this case it's audio.  In other words, they made a sample audio call, and recorded the packets that created the Audio channel in RTP (RTP is a network layer protocol that handles media streaming.)

Be advised, I was not able to get the built in scenarios to fully 100% work with my PBX here.  My PBX would send some extra calls or expect some extra calls and the tests had to be modified.  You'll find out what you really need, by making a call from your softphone to the pbx, and recording the traffic with Wireshark and reviewing the results... I'll detail that in a different blog post.

Using Your Own Scenarios

As mentioned above, you'll no doubt need to use your own scenarios.  When you modify theirs or create your own, you call it from the command line with -sf [path to your file]

Organizing the Tests

As you can get an idea, these tests are in two parts - Scenario and Parameters.  Because of this, it gets challenging organizing and keeping the actual test.  You can easily store the scenario flat files, but how do you store all your parameters that drive the tests?

I opted for using Jenkins.  Jenkins is a great way to save a test, as a "project" and "build" it on demand.  The command lines are stored in the shell script for the project, and the command line calls the scenario file.  I have more on that in my previous blog post. 

Parameters 

SIPP is half scenario and half parameter driven.  The parameters are supplied at the time of the test, via the command line inputs.  I've mentioned some previously to call your own scenario files or use the built in ones... there are tons of parameters you can set... the most common are:
The Service Name (the To: you are sending to)
The Proxy IP of the PBX
Username/pass to authenticate on the PBX

I also use the reporting parameters.  These are all prepended with "trace" like -trace_err (outputs error logs), -trace_screen outputs the general screen stats that displayed and -trace_rtt outputs the response time values to a CSV file (you have to have set a start and stop timer for response time in the test for this to work.)

All these and more can be seen in the documentation on the SIPP repository.

UDP Vs. TCP

This wasn't a surprise to me, but for some I've talked to, it is a bit surprising.  In the web world, mostly TCP is used. But with rapid data transfer, such as an Online Game (Planetside 2, my favorite online game), or VOIP, UDP is used.  UDP doesn't have any delays checking and needing all the packets delivered. If TCP is used, the phone call would be broken and waiting for lost packets. Packets are going to get lost, so what VOIP does (from my limited vision into VOIP) is use UDP and smooth out the call when packets get lost.  There is  a threshold at which lost packets create a loss in audio. 

Which is why there is a need for load testing. It's very important to Load test the SIP services and make sure that they can handle the expected load and that the packet loss is within the SLA. 

SIPP and Jenkins

Flow

I start with the Dashboard having a project which first sets up the data for the test:

The JMeter set up is pretty straight forward... it's the parent project and the Working directory is set ot the JMeter installation folder. The Shell script is set to  kick off the JMeter script that seeds the data for the test.  Afterwards it kicks off the downstream project, which runs the load.

The load itself is just the SIPP command line:

The Post Process here is to email out the CSV results to a distribution list.... and clean up the test data.

Organizing SIPP with Jenkins

In my organization, I work on a team that's business is both a web application and a SIP protocol.  To that end I work in both Java created code and C created code.

The UI I automated with Cucumber, but when it came time to create Load and Performance tests of the SIP side of things, I turned to SIPP.

SIPP is a command line unix tool that lets you drive SIP traffic to a virtual PBX proxy (FreeSWITCH, Asterix, etc.)  You can simulate calls by driving RTP traffic (as recorded pcaps) as well as send DTMF tones to navigate IVR's.

SIPP tests are controlled equally via the command line, as well as in a scenario file.  This is where it gets cumbersome.  You can store all the scenario files, and you can put them in git or subversion... but you still need to remember all those command line parameters for your tests.

I decided to utilize Jenkins for this.

I set up a local installation of Jenkins to handle the Load Tests only.  Each Jenkins job is really just a drive of a command line.  For example, a Jenkins build called "2,000 SIP calls @ 10 calls per second" would be a build process of simply running a shell script: ./sipp -s [service name] [your proxy] -sf [your scenario file] -r 10 [sets the rate to 10 per 1 sec] -m 2000 [the number of calls] -trace_rtt [calculates the response time you defined in your scenario file and outputs the results to a csv file] -trace_err [outputs any errors]

When this Jenkins job is built, it runs the command line that drives the scenario file with those parameters specified.

Then I have Jenkins mail me the csv file created.  Integrating the output to Jenkins would be great, but it requires reworking some performance plugins.  At this point, I'll just handle the graphs in excel from the csv data.

Upstream

I created a Higher Level project to set the data. This occurred when some of the load tests were run, the number they were hitting as being configured incorrectly via the Web Application.  Not knowing the state of the previous web application would cause the tests to sometimes get directed to the wrong locations.... i.e. all the load going to a forwarding action to another number, or going to a fax line.

To resolve this, I added a simple JMeter script as the parent project.  It does several put requests, setting all the data as expected, then immediately kicks off the downstream project which runs the SIPP tool to kick off the SIP load.

Further Downstream

After the Email is sent out, I didn't want a ton of CSV files cluttering up the workspace, so I have a final clean up project that moves the csv's to archive.

End Result

The end result is a fully automated SIP load process, via a web interface that can be accessed anywhere on the network.

A user goes to Jenkins, clicks on a Project like "2,000 SIP calls to Integration @ 10 calls per second" and clicks build.  In a few min, an email with results is sent out to them.

It can be further improved, but it's a great start that really makes organizing the SIP scenarios and command line parameters feasible, as well as giving an easy to use interface to run load tests ad hoc.

Automating SIP Testing with SIPp, FreeSwitch and Cucumber

SIP testing is pretty new to me.  In fact, 3 months ago, I didn't know what SIP was and I knew little of VOIP.  Today, I work at a company who's business is phone routing, virtual PBX, fax routing, VOIP, SIP, etc.

While I was able to set up the Front End web app automation, the development leads wanted me to also start building out a framework to test the SIP calls and the PBX software (FreeSwitch.)

Problem:

Create an automated framework that can be used to load test and functional test FreeSwitch.

Analysis:

I looked at a lot of opensource solutions, I was hoping for FS_Spec and some other ruby based solutions to work. But I had nothing but issues with them. In fact most software in this realm hasn't been updated since '09 or so.  In my research I did find one Open Source tool that was highly rated all around:
SIPp.

SIPp even comes pre-installed on the security distro of Linux (BackTrack.)  SIPp is pretty powerful, but it has a few drawbacks:

• It's not geared to functional tests. SIPp is really a load engine.  
• The reporting isn't great.  
• Can't run a suite of tests out of the box
• Doesn't have built in features to test simultaneous UAC and UAS (sender/receiver) - it expects a person to have multiple terminal sessions running, and one running a UAC and the other a UAS. 

 The above issues, broke down to too much human intervention.  What was desired, was something similar to the front end automation: you run a functional test, it either passes or fails, and the results are gathered for the test suite.

Solution:

I started with SIPp as the solution for doing the bulk of the tests, and then worked to build a framework around it to run the tests in an automated fashion.

What I came up with was Cucumber running SIPp functional and Load scripts.
The end result is, I just run:
cucumber
It runs all the tests (i.e. a few seconds later, it's driving traffic to my desk phone... Desk phone is ringing... yay.) Test fails/passes are tallied and the results are output to screen and file.

In the end, Cucumber is optional.  It's not a requirement here. For me, I'm only using Cucumber to tally the results.  I may opt to drop it all together as it has a high overhead.  The tests would be faster if just a collection of Ruby/Groovy or Scala scripts.

How it All Works:

To start it off, I had to see how the development servers work (i.e. FreeSwitch.)  So first things first, if you're going to test SIP, I recommend installing the Virtual PBX you are using in development (Asterix, FreeSwitch, etc.) on a local QA box.

Other tools you'll want installed on the same box are:
Wireshark
SIPp
SoftPhone for testing (I.e. X-Lite or "telephone")
Scripting Language (Ruby/Groovy/Python/Scala Scripts)
Optionally: Cucumber 
Optionally: Fully Functional Call Control with virtual numbers

I recommend only using OSX or a Linux distro for this set up.  Getting this to work on a PC is too difficult. 

You'll want to spend time with FreeSwitch and WireShark to see  how the packets are sent back and forth... and how to read the exchange.  Then you'll see how to write the tests.

Writing the SIP tests all boils down to two steps:

• The main test is going to be written in SIPp.  It's a command line SIP load tool.  But you can constrain it to only drive one test, for functional testing.  
• Use a Scripting Language (or Cucumber) to run the SIPp tests you wrote.

I detail each step below...

Details: Setting Up FreeSwitch

The easiest way to do this is on OSX or Linux.  Just do a git clone from the FreeSwitch repo... you can follow the instructions here: http://wiki.freeswitch.org/wiki/Installation_and_Tips_for_Mac_OS_X
Make sure you also do the step to make your audio files.

On Mac you'll prob need to bring in the libjpeg on my mac... to do this i used brew install libjpeg.

Configuration Setup:

This is where it gets a little tricky with FreeSwitch set up.

1. You'll need to know or change the default password... this is found in the vars.xml file  in /conf inside of the Freeswitch directory.  The instructions to do this are actually in the comments of the vars.xml file. 
2. Next, you'll want to change  /conf/sip_profiles/sip_profiles/internal.xml to update the inbounx-acl is localnet.auto 
3. As a tester, you're probably going to be  using an Internal Profile (meaning your machine's internal IP) for testing.  Developers will probably not use that set up. For testing though, you probably will.  When pointing to your Internal/local IP, it will bind that to a specific profile called "internal."  By default that folder is empty.  What I did, to make it quick and easy was this step:  I copied the files in the Default profile to the Internal Profile:  /conf/directory/default/* to /conf/sip_profiles/internal/* 
4. What you just copied over are actual user accounts and extensions.  They need to be modified.... Edit them individually and change the reference of "default" to "internal" and save them out.

Start Up FreeSwitch

Ok lets start it up, go to the freeswitch bin directory (i.e. /usr/local/freeswitch/bin) and type ./freeswitch

Once it's up, type: sofia status
If everything worked out, you should see 4 rows in a table display.  One will be an Alias, and it references the Internal Profile.  This Alias should have your local IP.   

You can tab for methods/commands... like this, type sofia, then hit TAB.  you'll see all the available options.  For example: sofia global siptrace on  is a very useful command.

Register with your local FreeSwitch

You can now launch a softphone (like X-Lite) and set it up to talk to login as a default user you copied over (i.e. 1000) and use the default password (what's in the vars.xml file) and then for domain set your local IP.

If everything is set up correctly, it should REGISTER on the FreeSwitch.

If you have a second softphone (i.e. "telephone"), you can also register as a different user (i.e. 1001) and then call the other user (i.e. 1000).

Once you make a call you should start seeing the FreeSwitch server display a lot of activity.  If you  have: sofia global siptrace on running, it will categorize the events.  You should see events like: INVITE, ACK, BYE.

Details: Using Wireshark

Wireshark Installation Overview

Wireshark is incredibly useful here.  If you are installing Wireshark on a MAC, you'll need to set up X11... on Mountain Lion, you'll need to use XQuartz.  Once installed, you can then bind Wireshark to XQuartz/X11 - I wont go into all the setup details here, but you'll need to restart your Mac for the changes to take effect (or logout/login.)

When Wireshark is up and running, you'll want to listen to the interface: lo0 for  your local traffic.

What can Wireshark show me?

What can't it show you.... it shows everything.  It not only records all the packets, this tool also has a whole Telephony menu.  From there you can build VOIP diagrams of the call flows. This is very useful in seeing what is sent and expected back for each test.  Wireshark can also copy the RTP stream!  That means it can hear the audio  you send, and verify the audio sent (say a recorded wav) is what's received!  

Using Wireshark to Generate a Flow Diagram

Go ahead and launch XQuartz, then Wireshark. 

Now, click on the interface lo0 and start recording.

Make sure FreeSwitch is up and running and that it is taking local traffic (i.e. sofia status shows alias and internal profiles up and running)

Make a call from one softphone to the other

Stop the Wireshark recording.

At first you'll see a dump of all packets captured on that interface.  

Now, go to the Telephony menu, and click "VOIP."  That will load only the VOIP related packets.  

Click the button, Select All.  

Now click Make Flow Diagram.

From that window, save it out.  In OSX the initial display usually doesn't render well. but once you save it out, it looks great.

This flow diagram will show you everything going on in a call, and removes all the uneeded data, making this more human readable. 

Using Wireshark to listen/verify the audio of a Call

Go ahead and launch XQuartz, then Wireshark. 

Now, click on the interface lo0 and start recording.

Make sure FreeSwitch is up and running and that it is taking local traffic (i.e. sofia status shows alias and internal profiles up and running)

Make a call from one softphone to the other

Stop the Wireshark recording.

Similar to before, click the Telephony menu.

Click the RTP sub menu and the "Show Streams" option.

You'll see packets here on each line item. click through on one, and click "Analyze."

You'll get the audio of the call/transmission. 

Using Wireshark Programmatically

My goal is to use it in automation testing.  To do this, I'll be calling the Wireshark utilities from the command line and getting back results in the command line interface.  

Details: SIPp

Now lets look at SIPp.  If you're new to SIP, you can think of SIPp as a command line version of JMeter.  It was designed to drive lots of traffic (hundreds or thousands of calls a min) to a PBX.

However, SIPp can be set up to run just one call through a functional scenario.

To install SIPp, just follow the instructions over at: http://sipp.sourceforge.net/

Pre-Installed Tests

SIPp has built in tests.  These include UAC, UAS, UAC_PCAP, REGEX, and more.  The whole list is off their main site.  If you don't know what UAC or UAS is.... you should probably read up on SIP.  But to just basically summarize it, SIP is a P2P system.  At any given time, one user is both the server and client.  They are sending data and receiving.  so UAC and UAS is a client server scenario.  

If you just want to see SIPp work, you can type:

./sipp uac [your local ip]

There's no Params, so it will run it's default load for the UAC test. 

If you want to see what the built in UAC test is actually doing, just do this:

./sipp -sd uac >> [filename]

Working with Parameters and SIPp

There's some sites out there with useful cheat sheets for SIPp... here's one such site:

http://tomeko.net/other/sipp/sipp_cheatsheet.php?lang=pl

That site also has some tests they wrote, which you can download and see how they work.  They also have tests that utilize CSV files which contain a set of users you want in your load.

For my tests, I use a lot of functional tests, so I don't want to slam 10,000 phone calls against the server. Instead I want to run one call, one time, to do something specific:

./sipp -s [phone number configured on call control to redirect to my desk phone] -m 1 -l 1 -recv_timeout 6000 -sf [path to my test xml file] [IP of our integration / uat freeswitch]

So here's what those parameters are saying:

Run SIPP against the QA Integration/UAT env. and dial that number configured in Call Control (-s means service, which could be a extension, phone number, etc.) -m tells it to run once -l tells it to run only one test at a time.  -recv_timeout 6000 says to run this test for only 6 seconds.  Finally -sf is the path to my test.... and of course i end it with the IP of the FS box.

Writing your Own SIPp tests

I wont go into too much detail here.  I'll save it for a future blog.  But to start with, you can take an existing built in test, and export it using the -sd option I mention above, or you can review some tests other people are writing out there and have put on github or elsewhere.

Automating All This

Now to automate this.

Initially I used straight Ruby for this... but then I decided having Cucumber run the tests, gives me the added benefit of compiling test results. I'm on the fence if that's necessary at this point or not.

Basically, I found it too hard to use an existing API with what we use here in house as our PBX.  I tried SIPr, FS_Switch and a variety of other tools. Most of these haven't been touched in many years and have known bugs flagged against them.  Rather then fight with that, I choose to use SIPp.

SIPp is an industry standard, but it doesn't have an open API.  To automate this, I use SIPp as is, and build the shell around it.

It's really very simple:

Create a script to drive all the SIPp tests.  

Assert via REGEX the output for expectations.

Mark pass / fail.

So I'm basically doing the core work in SIPp.  I write out my test, with the parameters to call it effectively.

Then I use Cucumber (you could use straight Ruby or Groovy or Scala, Python, etc.) to do this:

Given /^a SIP call is made to another SIP account$/ do

   @calling = %x{/Users/brainwarner/sipp-3.3/sipp -s 1000 -m 1 -l 1 -recv_timeout 6000 -sf /Users/brianwarner/sipp-3.3/calling.xml 10.98.1.1}

end

Then /^a SIP 180 ringing should return$/ do

   assert @calling.include?("180")

end

The above is a simple set of two steps in a step definition file.  Basically it's saying ok, I'm going to execute sipp with my sipp test, against the specific IP of my UAT server and I'm using those parameters to make sure it's just one test, run once and ends after 6 seconds.

Then I'm calling a Ruby method "include" which is like a REGEX, to search the output that's in that class variable @calling for the specific ID I expect for calling.  

Why not use REGEX? well every regex I tried, passed even when it failed. Initially i would search the output for a invalid value (i.e. 777iwin!), and it would always pass, no matter how I wrote the regex!  I ended up using this method because it actually works well.

Simple. But it works.

Future Steps In Automation

So this is the Framework.  

You could easily see how that simple test could be transformed to something more impressive:

%x{....} calling one UAS user session

%x{...} calling another session in UAC test mode... and then have them talk to each other.

Since we're calling 3rd party command line utilities, you can also run the command line wireshark! meaning at test time you could use Wireshark to sniff packets, or verify RTP audio.... now that's cool.