SIP Testing Basics

This is a walk through on what it took to get a single SIP load test generated, using SIPP.  Getting the first one was hard, so I wanted to document these steps in case it helps anyone else, and for future reference.

Tools

There's not a lot of SIP load tools out there.  For SIP load, there's really one standard that keeps coming up in discussions: SIPP.  SIPP, which is at: http://sipp.sourceforge.net/ is a command line tool that drives various types of SIP traffic to a proxy. There's a lot of potential with this tool, it actually is highly configurable and has quite a few amazing capabilities, as seen in their documentation (http://sipp.sourceforge.net/)

SIPP is also one of the most difficult things to learn. If you are new to Session Initiation Protocol (SIP) and just starting to test it, learning about SIP and SIPP is pretty daunting.  However, I have one advice, stick with SIPP.  I've tried every open source tool out there, and for SIP load testing, SIPP is by far the best.

After I got SIPP scripts up and running, I then started integrating them into Jenkins, and letting Jenkins maintain the scripts and email me the results of each run (more on that in a separate blog post: http://www.continuous-qa.com/2013/02/sipp-and-jenkins.html

SIP Basics

SIP stands for Session Initiation Protocol.  I won't pretend to be an expert on it.  I was thrown into the world of SIP as a need arose for SIP Load testing where I work.

Basically, SIP is a protocol that handles the handshaking and transfer of data for VOIP services... Voice over the Internet.  But this isn't limited to just handling tools like Skype, Teamspeak, etc.  In fact most phone calls, at some point, become SIP calls.  You may pick up a hard line, dial a number, it could route to your PBX and at some or many points in the transaction go to a SIP PBX.

SIP is a standard of packet protocols. It requires certain things to occur in specific sequences.  There is room for some variance, but in general everything works within a standard format.

Simple Call

In a simple call where one phone dials another, and SIP is being utilized, we'd see something like this occur:
12136667777@someplace.com INVITE -----> 1000@somepbx.com
100 Trying <-------- Phone 2
200 OK <----------- Phone 2
ACK ---------------->

In the above example there's some similarities with the Web world... first off the Sip addresses look like emails: 12136667777@someplace.com and 1000@somepbx.com

That's how SIP accounts are registered. The first is an example of a potential phone number... listed on a proxy or pbx.  the other is an example of extension 1000 on a pbx.

The responses also look like HTTP status codes... that's because they are related.  A list of SIP status codes can be found here:
http://www.websitepulse.com/kb/sip_status_codes.html

You'll notice a lot of similarities with HTTP codes... 400 range are errors as are 500, 200's are usually "OK's" and so forth.

Details of a Simple Call

Using Wireshark or other packet sniffers you can grab the packets and see what they look like. Below are some examples of the packets from my local network:

Invite

----------------------------------------------- 2013-02-05 16:52:14:961.441
UDP message sent (505 bytes):

INVITE sip:1000@10.98.6.26:5060 SIP/2.0
Via: SIP/2.0/UDP 10.98.7.36:5061;branch=z9hG4bK-1769-1-0
From: sipp <sip:sipp@10.98.7.36:5061>;tag=1769SIPpTag001
To: 1000 <sip:1000@10.98.6.26:5060>
Call-ID: 1-1769@10.98.7.36
CSeq: 1 INVITE
Contact: sip:sipp@10.98.7.36:5061
Max-Forwards: 70
Subject: Performance Test
Content-Type: application/sdp
Content-Length:   131

v=0
o=user1 53655765 2353687637 IN IP4 10.98.7.36
s=-
c=IN IP4 10.98.7.36
t=0 0
m=audio 6000 RTP/AVP 0
a=rtpmap:0 PCMU/8000

100 Trying

----------------------------------------------- 2013-02-05 16:52:14:961.782
UDP message received [332] bytes :

SIP/2.0 100 Trying
Via: SIP/2.0/UDP 10.98.7.36:5061;branch=z9hG4bK-1769-1-0;received=10.98.6.26
From: sipp <sip:sipp@10.98.7.36:5061>;tag=1769SIPpTag001
To: 1000 <sip:1000@10.98.6.26:5060>
Call-ID: 1-1769@10.98.7.36
CSeq: 1 INVITE
User-Agent: FreeSWITCH-mod_sofia/1.3.13b+git~20130204T173656Z~f3206e5ff1
Content-Length: 0

200 OK

----------------------------------------------- 2013-02-05 16:52:15:2.033
UDP message received [1066] bytes :

SIP/2.0 200 OK
Via: SIP/2.0/UDP 10.98.7.36:5061;branch=z9hG4bK-1769-1-0;received=10.98.6.26
From: sipp <sip:sipp@10.98.7.36:5061>;tag=1769SIPpTag001
To: 1000 <sip:1000@10.98.6.26:5060>;tag=cUcc2K9Z5Ut1F
Call-ID: 1-1769@10.98.7.36
CSeq: 1 INVITE
Contact: <sip:1000@10.98.6.26:5060;transport=udp>
User-Agent: FreeSWITCH-mod_sofia/1.3.13b+git~20130204T173656Z~f3206e5ff1
Accept: application/sdp
Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REGISTER, REFER, NOTIFY, PUBLISH, SUBSCRIBE
Supported: timer, precondition, path, replaces
Allow-Events: talk, hold, conference, presence, dialog, line-seize, call-info, sla, include-session-description, presence.winfo, message-summary, refer
Content-Type: application/sdp
Content-Disposition: session
Content-Length: 187
Remote-Party-ID: "1000" <sip:1000@10.98.6.26>;party=calling;privacy=off;screen=no

v=0
o=FreeSWITCH 1360084930 1360084931 IN IP4 10.98.6.26
s=FreeSWITCH
c=IN IP4 10.98.6.26
t=0 0
m=audio 27004 RTP/AVP 0
a=rtpmap:0 PCMU/8000
a=silenceSupp:off - - - -
a=ptime:20

ACK

----------------------------------------------- 2013-02-05 16:52:15:2.159
UDP message sent (351 bytes):

ACK sip:1000@10.98.6.26:5060 SIP/2.0
Via: SIP/2.0/UDP 10.98.7.36:5061;branch=z9hG4bK-1769-1-5
From: sipp <sip:sipp@10.98.7.36:5061>;tag=1769SIPpTag001
To: 1000 <sip:1000@10.98.6.26:5060>;tag=cUcc2K9Z5Ut1F
Call-ID: 1-1769@10.98.7.36
CSeq: 1 ACK
Contact: sip:sipp@10.98.7.36:5061
Max-Forwards: 70
Subject: Performance Test
Content-Length: 0

SIPP Sample Scenario

SIPP tests SIP by using scenario files to simulate the flow of a call. Then, using parameters passed on the command line, it will run X load, Y times, with Z packet loss %, etc.  So SIPP is working on both a command line and a flat file.

The fat file, henceforth referred to as Scenario file is basically an XML document.  If you are familiar with XML then the Scenario file will look very familiar.  The scenario file will set the constraints of the test.  It will say "i'm sending a packet like this for the invite... now after that's sent i'm ready to accept 100 trying...." and so forth. 

There are variety of built in scenarios and of course, you can make your own. 

Here's what a Scenario File looks like:

<scenario name="Basic Sipstone UAC" >
  <!-- In client mode (sipp placing calls), the Call-ID MUST be         -->
  <!-- generated by sipp. To do so, use [call_id] keyword.                -->
  <send retrans="500" >
    <![CDATA[

      INVITE sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>
      Call-ID: [call_id]
      CSeq: 1 INVITE
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Type: application/sdp
      Content-Length: [len]

      v=0
      o=user1 53655765 2353687637 IN IP[local_ip_type] [local_ip]
      s=-
      c=IN IP[media_ip_type] [media_ip]
      t=0 0
      m=audio [media_port] RTP/AVP 0
      a=rtpmap:0 PCMU/8000

    ]]>
  </send>

  <recv response="100"
        optional="true" start_rtd="Performance" >
  </recv>

  <recv response="180" optional="true">
  </recv>

  <recv response="183" optional="true">
  </recv>

  <!-- By adding rrs="true" (Record Route Sets), the route sets         -->
  <!-- are saved and used for following messages sent. Useful to test   -->
  <!-- against stateful SIP proxies/B2BUAs.                             -->
  <recv response="200" rtd="Performance">
  </recv>

  <!-- Packet lost can be simulated in any send/recv message by         -->
  <!-- by adding the 'lost = "10"'. Value can be [1-100] percent.       -->
  <send>
    <![CDATA[

      ACK sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>[peer_tag_param]
      Call-ID: [call_id]
      CSeq: 1 ACK
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Length: 0

    ]]>
  </send>

  <!-- This delay can be customized by the -d command-line option       -->
  <!-- or by adding a 'milliseconds = "value"' option here.             -->
  <pause/>

  <!-- The 'crlf' option inserts a blank line in the statistics report. -->
  <send retrans="500">
    <![CDATA[

      BYE sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>[peer_tag_param]
      Call-ID: [call_id]
      CSeq: 2 BYE
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Length: 0

    ]]>
  </send>

  <recv response="200" crlf="true"  >
  </recv>

  <!-- definition of the response time repartition table (unit is ms)   -->
  <ResponseTimeRepartition value="10, 20, 30, 40, 50, 100, 150, 200"/>

  <!-- definition of the call length repartition table (unit is ms)     -->
  <CallLengthRepartition value="10, 50, 100, 500, 1000, 5000, 10000"/>

</scenario>

Breaking Down The Scenario

In the above example, there's a flow going on.  It's easier to read in an IDE like  Intellij.  The Invite block looks like this:
  <send retrans="500" >
    <![CDATA[

      INVITE sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>
      Call-ID: [call_id]
      CSeq: 1 INVITE
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Type: application/sdp
      Content-Length: [len]

      v=0
      o=user1 53655765 2353687637 IN IP[local_ip_type] [local_ip]
      s=-
      c=IN IP[media_ip_type] [media_ip]
      t=0 0
      m=audio [media_port] RTP/AVP 0
      a=rtpmap:0 PCMU/8000

    ]]>
  </send>

It's very similar to our real packet from way up above in this post.  There are items in this though in square brackets.  Those items are variables. These are either auto generated or set via the command line parameters.  by default the call_id is auto generated, but some of the other items need to be supplied as a parameter via the command line (such as the service, proxy ip, etc.)

After this invite block we see some xml nodes called recv response.  These allow the test to receive these responses back.
  <recv response="100"
        optional="true" start_rtd="Performance" >
  </recv>

  <recv response="180" optional="true">
  </recv>

  <recv response="183" optional="true">
  </recv>
Say for example we didn't have recv response="100"... and we send an invite.  If we get a 100 response back, the test will fail. It will say "expecting 180 (the next on the test) but got a 100 instead.

I also have a tag there called "start_rtd"  this starts my response time timer. In this test, I'm organizing the response time between the first 100 I get after an Invite is sent, and the 200 OK.  So it's saying, I've sent an Invite, how long does it take to get a 200 OK? 

Further down we have another packet set up for ACK and then for BYE (disconnect.) 

Built In Scenarios

SIPP comes with some built in scenarios... for example, if you download and install sipp, you could go to the sipp directory and do a ./sipp -sn uac [ip of  your proxy] and it start generating sip load on your proxy based on that scenario.

Notice the -sn parameter with the value uac.  The -sn means use a default scenario.  The value is the name of it. it's built it. it's not a flat file. However, if you want to see what uac does, just do this:

./sipp -sd uac   (that will print to screen)
./sipp -sd uac >> uac.xml (that will print it out to an xml doc)

SIPP has several built in scenarios:

• UAS
• UAC
• UAC_Pcap

UAS is a server, it would listen for incoming calls...
UAC is a client sending SIP calls.
In truth, all SIP calls have each party being both the Server and the Client.

UAC_PCAP is a special test, it will send a PCAP (more on this later) to the recipient.  A PCAP is a packet capture.  In this case it's audio.  In other words, they made a sample audio call, and recorded the packets that created the Audio channel in RTP (RTP is a network layer protocol that handles media streaming.)

Be advised, I was not able to get the built in scenarios to fully 100% work with my PBX here.  My PBX would send some extra calls or expect some extra calls and the tests had to be modified.  You'll find out what you really need, by making a call from your softphone to the pbx, and recording the traffic with Wireshark and reviewing the results... I'll detail that in a different blog post.

Using Your Own Scenarios

As mentioned above, you'll no doubt need to use your own scenarios.  When you modify theirs or create your own, you call it from the command line with -sf [path to your file]

Organizing the Tests

As you can get an idea, these tests are in two parts - Scenario and Parameters.  Because of this, it gets challenging organizing and keeping the actual test.  You can easily store the scenario flat files, but how do you store all your parameters that drive the tests?

I opted for using Jenkins.  Jenkins is a great way to save a test, as a "project" and "build" it on demand.  The command lines are stored in the shell script for the project, and the command line calls the scenario file.  I have more on that in my previous blog post. 

Parameters 

SIPP is half scenario and half parameter driven.  The parameters are supplied at the time of the test, via the command line inputs.  I've mentioned some previously to call your own scenario files or use the built in ones... there are tons of parameters you can set... the most common are:
The Service Name (the To: you are sending to)
The Proxy IP of the PBX
Username/pass to authenticate on the PBX

I also use the reporting parameters.  These are all prepended with "trace" like -trace_err (outputs error logs), -trace_screen outputs the general screen stats that displayed and -trace_rtt outputs the response time values to a CSV file (you have to have set a start and stop timer for response time in the test for this to work.)

All these and more can be seen in the documentation on the SIPP repository.

UDP Vs. TCP

This wasn't a surprise to me, but for some I've talked to, it is a bit surprising.  In the web world, mostly TCP is used. But with rapid data transfer, such as an Online Game (Planetside 2, my favorite online game), or VOIP, UDP is used.  UDP doesn't have any delays checking and needing all the packets delivered. If TCP is used, the phone call would be broken and waiting for lost packets. Packets are going to get lost, so what VOIP does (from my limited vision into VOIP) is use UDP and smooth out the call when packets get lost.  There is  a threshold at which lost packets create a loss in audio. 

Which is why there is a need for load testing. It's very important to Load test the SIP services and make sure that they can handle the expected load and that the packet loss is within the SLA. 

SIPP and Jenkins

Flow

I start with the Dashboard having a project which first sets up the data for the test:

The JMeter set up is pretty straight forward... it's the parent project and the Working directory is set ot the JMeter installation folder. The Shell script is set to  kick off the JMeter script that seeds the data for the test.  Afterwards it kicks off the downstream project, which runs the load.

The load itself is just the SIPP command line:

The Post Process here is to email out the CSV results to a distribution list.... and clean up the test data.

Organizing SIPP with Jenkins

In my organization, I work on a team that's business is both a web application and a SIP protocol.  To that end I work in both Java created code and C created code.

The UI I automated with Cucumber, but when it came time to create Load and Performance tests of the SIP side of things, I turned to SIPP.

SIPP is a command line unix tool that lets you drive SIP traffic to a virtual PBX proxy (FreeSWITCH, Asterix, etc.)  You can simulate calls by driving RTP traffic (as recorded pcaps) as well as send DTMF tones to navigate IVR's.

SIPP tests are controlled equally via the command line, as well as in a scenario file.  This is where it gets cumbersome.  You can store all the scenario files, and you can put them in git or subversion... but you still need to remember all those command line parameters for your tests.

I decided to utilize Jenkins for this.

I set up a local installation of Jenkins to handle the Load Tests only.  Each Jenkins job is really just a drive of a command line.  For example, a Jenkins build called "2,000 SIP calls @ 10 calls per second" would be a build process of simply running a shell script: ./sipp -s [service name] [your proxy] -sf [your scenario file] -r 10 [sets the rate to 10 per 1 sec] -m 2000 [the number of calls] -trace_rtt [calculates the response time you defined in your scenario file and outputs the results to a csv file] -trace_err [outputs any errors]

When this Jenkins job is built, it runs the command line that drives the scenario file with those parameters specified.

Then I have Jenkins mail me the csv file created.  Integrating the output to Jenkins would be great, but it requires reworking some performance plugins.  At this point, I'll just handle the graphs in excel from the csv data.

Upstream

I created a Higher Level project to set the data. This occurred when some of the load tests were run, the number they were hitting as being configured incorrectly via the Web Application.  Not knowing the state of the previous web application would cause the tests to sometimes get directed to the wrong locations.... i.e. all the load going to a forwarding action to another number, or going to a fax line.

To resolve this, I added a simple JMeter script as the parent project.  It does several put requests, setting all the data as expected, then immediately kicks off the downstream project which runs the SIPP tool to kick off the SIP load.

Further Downstream

After the Email is sent out, I didn't want a ton of CSV files cluttering up the workspace, so I have a final clean up project that moves the csv's to archive.

End Result

The end result is a fully automated SIP load process, via a web interface that can be accessed anywhere on the network.

A user goes to Jenkins, clicks on a Project like "2,000 SIP calls to Integration @ 10 calls per second" and clicks build.  In a few min, an email with results is sent out to them.

It can be further improved, but it's a great start that really makes organizing the SIP scenarios and command line parameters feasible, as well as giving an easy to use interface to run load tests ad hoc.

Cucumber Conditionals

I had an interesting experience with a GXT environment.  Since GXT generates the HTML code, I had some constraints to work around.

In this particular case, I had drop downs that were locked down to these specifications:
  - drop downs were not treated with the "select" tag
  - drop downs were read only

this meant, if I had a language drop down with values: English, Spanish, French

I had to find a way to select any of them, via a test.  Since the code in this case couldn't be modified to switch the drop downs to "selectors" I was stuck. Watir Webdriver expects drop downs to be accessed with the .select method.  In this case I had to use the .text_field method.

Since the fields were read only, I couldn't pass in values to select them.

My Solution
I ended up writing a conditional test in Cucumber itself.

The test would be a scenario outline defining the values in the drop down for the test, like this:

Scenario Outline: Testing each value in the language drop down
  Given a user at .....
  When they choose a specific language of <language>
  And Save .....
  Then the <language> will be set
  Examples:
  |language|
  |English|
  |Spanish|
  |French|

In the step definition, I wrote a conditional at the When and Thens.... the When would be like:
if arg == "English
   @browser.text_field(:id=>"the id on this field").send_keys :arrow_down, :return

elsif arg == "Spanish"
   @browser.text_field(:id=>"the id on this field").send_keys 2.times{:arrow_down)
   @browser.send_keys :return

and so forth.

For many choices I would use a case/when switch instead of a series of if's, but you get the idea.

Is it the best idea? No.

But it was optimal, as some would say. It got a solution out quickly, when I had a constraint that forced me into a specific path.

Cucumber Using Multiple Test Environments

I came across a need to run my tests across multiple environments.  For example:
Integration Environment, vs QA 1 vs QA 2.  Different versions or branches of the web app could be stood up in either location.

The method I used to achieve this was with the use of hooks.  But I hit a little snag, and found out my error.

What I did was this:

In my /support/env.rb file (within the Cucumber folder), I have this:
def browser_name
  (ENV['BROWSER'] ||= 'firefox').downcase.to_sym
end

def environment
  (ENV['ENVI'] ||= 'int').downcase.to_sym
end

Before do  |scenario|
  p "Starting #{scenario}"
  if environment == :int
    @browser = Watir::Browser.new browser_name
    @browser.goto "http://integration.atmysite.com:8080"
    @browser.send_keys :return
  elsif environment == :qa1
    @browser = Watir::Browser.new browser_name
    @browser.goto "http://qa1.atmysite.com:8080"
    @browser.send_keys :return
  else
    @browser = Watir::Browser.new browser_name
    @browser.goto "http://qa2.atmysite.com:8080"
    @browser.send_keys :return
  end
end

The first method defines the browser from the command line, this is useful for use with Jenkins or other batch file runs, that I can set the browser to be used in the test via the command line.  I learned this from an online resource someplace.

The second method there is similar - I based this on the previous. I'm saying take the value ENVI and it's value... then in the Before block I say:
If that environment method is :int, then run this login block... if it's :qa1, run this login block... else go to this other location.

I'll prob enhance that to where's instead of if statements at some point.

Where I initially errored on this, I was setting the equality to "int" rather then "int"... and I couldn't figure out why it wasn't working.  So I debugged it, by putting in a puts environment during the test run.  It was outputting :int.  So the parameter value was getting set as a symbol (i.e. :something.)

Once I updated the if statements to be :int, :qa1, :qa2 everything worked great.

Now I dont need the login helper I was using.  This before hook takes care of the login, based on the environment value I pass.

Automating SIP Testing with SIPp, FreeSwitch and Cucumber

SIP testing is pretty new to me.  In fact, 3 months ago, I didn't know what SIP was and I knew little of VOIP.  Today, I work at a company who's business is phone routing, virtual PBX, fax routing, VOIP, SIP, etc.

While I was able to set up the Front End web app automation, the development leads wanted me to also start building out a framework to test the SIP calls and the PBX software (FreeSwitch.)

Problem:

Create an automated framework that can be used to load test and functional test FreeSwitch.

Analysis:

I looked at a lot of opensource solutions, I was hoping for FS_Spec and some other ruby based solutions to work. But I had nothing but issues with them. In fact most software in this realm hasn't been updated since '09 or so.  In my research I did find one Open Source tool that was highly rated all around:
SIPp.

SIPp even comes pre-installed on the security distro of Linux (BackTrack.)  SIPp is pretty powerful, but it has a few drawbacks:

• It's not geared to functional tests. SIPp is really a load engine.  
• The reporting isn't great.  
• Can't run a suite of tests out of the box
• Doesn't have built in features to test simultaneous UAC and UAS (sender/receiver) - it expects a person to have multiple terminal sessions running, and one running a UAC and the other a UAS. 

 The above issues, broke down to too much human intervention.  What was desired, was something similar to the front end automation: you run a functional test, it either passes or fails, and the results are gathered for the test suite.

Solution:

I started with SIPp as the solution for doing the bulk of the tests, and then worked to build a framework around it to run the tests in an automated fashion.

What I came up with was Cucumber running SIPp functional and Load scripts.
The end result is, I just run:
cucumber
It runs all the tests (i.e. a few seconds later, it's driving traffic to my desk phone... Desk phone is ringing... yay.) Test fails/passes are tallied and the results are output to screen and file.

In the end, Cucumber is optional.  It's not a requirement here. For me, I'm only using Cucumber to tally the results.  I may opt to drop it all together as it has a high overhead.  The tests would be faster if just a collection of Ruby/Groovy or Scala scripts.

How it All Works:

To start it off, I had to see how the development servers work (i.e. FreeSwitch.)  So first things first, if you're going to test SIP, I recommend installing the Virtual PBX you are using in development (Asterix, FreeSwitch, etc.) on a local QA box.

Other tools you'll want installed on the same box are:
Wireshark
SIPp
SoftPhone for testing (I.e. X-Lite or "telephone")
Scripting Language (Ruby/Groovy/Python/Scala Scripts)
Optionally: Cucumber 
Optionally: Fully Functional Call Control with virtual numbers

I recommend only using OSX or a Linux distro for this set up.  Getting this to work on a PC is too difficult. 

You'll want to spend time with FreeSwitch and WireShark to see  how the packets are sent back and forth... and how to read the exchange.  Then you'll see how to write the tests.

Writing the SIP tests all boils down to two steps:

• The main test is going to be written in SIPp.  It's a command line SIP load tool.  But you can constrain it to only drive one test, for functional testing.  
• Use a Scripting Language (or Cucumber) to run the SIPp tests you wrote.

I detail each step below...

Details: Setting Up FreeSwitch

The easiest way to do this is on OSX or Linux.  Just do a git clone from the FreeSwitch repo... you can follow the instructions here: http://wiki.freeswitch.org/wiki/Installation_and_Tips_for_Mac_OS_X
Make sure you also do the step to make your audio files.

On Mac you'll prob need to bring in the libjpeg on my mac... to do this i used brew install libjpeg.

Configuration Setup:

This is where it gets a little tricky with FreeSwitch set up.

1. You'll need to know or change the default password... this is found in the vars.xml file  in /conf inside of the Freeswitch directory.  The instructions to do this are actually in the comments of the vars.xml file. 
2. Next, you'll want to change  /conf/sip_profiles/sip_profiles/internal.xml to update the inbounx-acl is localnet.auto 
3. As a tester, you're probably going to be  using an Internal Profile (meaning your machine's internal IP) for testing.  Developers will probably not use that set up. For testing though, you probably will.  When pointing to your Internal/local IP, it will bind that to a specific profile called "internal."  By default that folder is empty.  What I did, to make it quick and easy was this step:  I copied the files in the Default profile to the Internal Profile:  /conf/directory/default/* to /conf/sip_profiles/internal/* 
4. What you just copied over are actual user accounts and extensions.  They need to be modified.... Edit them individually and change the reference of "default" to "internal" and save them out.

Start Up FreeSwitch

Ok lets start it up, go to the freeswitch bin directory (i.e. /usr/local/freeswitch/bin) and type ./freeswitch

Once it's up, type: sofia status
If everything worked out, you should see 4 rows in a table display.  One will be an Alias, and it references the Internal Profile.  This Alias should have your local IP.   

You can tab for methods/commands... like this, type sofia, then hit TAB.  you'll see all the available options.  For example: sofia global siptrace on  is a very useful command.

Register with your local FreeSwitch

You can now launch a softphone (like X-Lite) and set it up to talk to login as a default user you copied over (i.e. 1000) and use the default password (what's in the vars.xml file) and then for domain set your local IP.

If everything is set up correctly, it should REGISTER on the FreeSwitch.

If you have a second softphone (i.e. "telephone"), you can also register as a different user (i.e. 1001) and then call the other user (i.e. 1000).

Once you make a call you should start seeing the FreeSwitch server display a lot of activity.  If you  have: sofia global siptrace on running, it will categorize the events.  You should see events like: INVITE, ACK, BYE.

Details: Using Wireshark

Wireshark Installation Overview

Wireshark is incredibly useful here.  If you are installing Wireshark on a MAC, you'll need to set up X11... on Mountain Lion, you'll need to use XQuartz.  Once installed, you can then bind Wireshark to XQuartz/X11 - I wont go into all the setup details here, but you'll need to restart your Mac for the changes to take effect (or logout/login.)

When Wireshark is up and running, you'll want to listen to the interface: lo0 for  your local traffic.

What can Wireshark show me?

What can't it show you.... it shows everything.  It not only records all the packets, this tool also has a whole Telephony menu.  From there you can build VOIP diagrams of the call flows. This is very useful in seeing what is sent and expected back for each test.  Wireshark can also copy the RTP stream!  That means it can hear the audio  you send, and verify the audio sent (say a recorded wav) is what's received!  

Using Wireshark to Generate a Flow Diagram

Go ahead and launch XQuartz, then Wireshark. 

Now, click on the interface lo0 and start recording.

Make sure FreeSwitch is up and running and that it is taking local traffic (i.e. sofia status shows alias and internal profiles up and running)

Make a call from one softphone to the other

Stop the Wireshark recording.

At first you'll see a dump of all packets captured on that interface.  

Now, go to the Telephony menu, and click "VOIP."  That will load only the VOIP related packets.  

Click the button, Select All.  

Now click Make Flow Diagram.

From that window, save it out.  In OSX the initial display usually doesn't render well. but once you save it out, it looks great.

This flow diagram will show you everything going on in a call, and removes all the uneeded data, making this more human readable. 

Using Wireshark to listen/verify the audio of a Call

Go ahead and launch XQuartz, then Wireshark. 

Now, click on the interface lo0 and start recording.

Make sure FreeSwitch is up and running and that it is taking local traffic (i.e. sofia status shows alias and internal profiles up and running)

Make a call from one softphone to the other

Stop the Wireshark recording.

Similar to before, click the Telephony menu.

Click the RTP sub menu and the "Show Streams" option.

You'll see packets here on each line item. click through on one, and click "Analyze."

You'll get the audio of the call/transmission. 

Using Wireshark Programmatically

My goal is to use it in automation testing.  To do this, I'll be calling the Wireshark utilities from the command line and getting back results in the command line interface.  

Details: SIPp

Now lets look at SIPp.  If you're new to SIP, you can think of SIPp as a command line version of JMeter.  It was designed to drive lots of traffic (hundreds or thousands of calls a min) to a PBX.

However, SIPp can be set up to run just one call through a functional scenario.

To install SIPp, just follow the instructions over at: http://sipp.sourceforge.net/

Pre-Installed Tests

SIPp has built in tests.  These include UAC, UAS, UAC_PCAP, REGEX, and more.  The whole list is off their main site.  If you don't know what UAC or UAS is.... you should probably read up on SIP.  But to just basically summarize it, SIP is a P2P system.  At any given time, one user is both the server and client.  They are sending data and receiving.  so UAC and UAS is a client server scenario.  

If you just want to see SIPp work, you can type:

./sipp uac [your local ip]

There's no Params, so it will run it's default load for the UAC test. 

If you want to see what the built in UAC test is actually doing, just do this:

./sipp -sd uac >> [filename]

Working with Parameters and SIPp

There's some sites out there with useful cheat sheets for SIPp... here's one such site:

http://tomeko.net/other/sipp/sipp_cheatsheet.php?lang=pl

That site also has some tests they wrote, which you can download and see how they work.  They also have tests that utilize CSV files which contain a set of users you want in your load.

For my tests, I use a lot of functional tests, so I don't want to slam 10,000 phone calls against the server. Instead I want to run one call, one time, to do something specific:

./sipp -s [phone number configured on call control to redirect to my desk phone] -m 1 -l 1 -recv_timeout 6000 -sf [path to my test xml file] [IP of our integration / uat freeswitch]

So here's what those parameters are saying:

Run SIPP against the QA Integration/UAT env. and dial that number configured in Call Control (-s means service, which could be a extension, phone number, etc.) -m tells it to run once -l tells it to run only one test at a time.  -recv_timeout 6000 says to run this test for only 6 seconds.  Finally -sf is the path to my test.... and of course i end it with the IP of the FS box.

Writing your Own SIPp tests

I wont go into too much detail here.  I'll save it for a future blog.  But to start with, you can take an existing built in test, and export it using the -sd option I mention above, or you can review some tests other people are writing out there and have put on github or elsewhere.

Automating All This

Now to automate this.

Initially I used straight Ruby for this... but then I decided having Cucumber run the tests, gives me the added benefit of compiling test results. I'm on the fence if that's necessary at this point or not.

Basically, I found it too hard to use an existing API with what we use here in house as our PBX.  I tried SIPr, FS_Switch and a variety of other tools. Most of these haven't been touched in many years and have known bugs flagged against them.  Rather then fight with that, I choose to use SIPp.

SIPp is an industry standard, but it doesn't have an open API.  To automate this, I use SIPp as is, and build the shell around it.

It's really very simple:

Create a script to drive all the SIPp tests.  

Assert via REGEX the output for expectations.

Mark pass / fail.

So I'm basically doing the core work in SIPp.  I write out my test, with the parameters to call it effectively.

Then I use Cucumber (you could use straight Ruby or Groovy or Scala, Python, etc.) to do this:

Given /^a SIP call is made to another SIP account$/ do

   @calling = %x{/Users/brainwarner/sipp-3.3/sipp -s 1000 -m 1 -l 1 -recv_timeout 6000 -sf /Users/brianwarner/sipp-3.3/calling.xml 10.98.1.1}

end

Then /^a SIP 180 ringing should return$/ do

   assert @calling.include?("180")

end

The above is a simple set of two steps in a step definition file.  Basically it's saying ok, I'm going to execute sipp with my sipp test, against the specific IP of my UAT server and I'm using those parameters to make sure it's just one test, run once and ends after 6 seconds.

Then I'm calling a Ruby method "include" which is like a REGEX, to search the output that's in that class variable @calling for the specific ID I expect for calling.  

Why not use REGEX? well every regex I tried, passed even when it failed. Initially i would search the output for a invalid value (i.e. 777iwin!), and it would always pass, no matter how I wrote the regex!  I ended up using this method because it actually works well.

Simple. But it works.

Future Steps In Automation

So this is the Framework.  

You could easily see how that simple test could be transformed to something more impressive:

%x{....} calling one UAS user session

%x{...} calling another session in UAC test mode... and then have them talk to each other.

Since we're calling 3rd party command line utilities, you can also run the command line wireshark! meaning at test time you could use Wireshark to sniff packets, or verify RTP audio.... now that's cool.