Grails - Creating an Auto Dialer - Refactored with Services

After getting some formal Grails training, I decided to refactor this tool I made for our internal provisioning team.

The Original tool was discussed previously here.

Don't Put Too Much Logic in the Controller

I learned that a good MVC practice is to not put too much logic/work in the controller - but to think of controllers as routers.  The body of logic should be pulled out into services.

My previous version of the tool had all the work in the controller.  I had one controller doing this:

• Parsing a user submitted CSV file
• Dialing each number in the CSV
• Emailing the results of each number being active or disconnected, to the requested email

I've changed this, so that the controller just does CSV parsing, and two separate services are handing the phone number dialing and the email sending.

The Refactored Controller

The controller now looks like this:

@Grab('com.xlson.groovycsv:groovycsv:1.0')
import static com.xlson.groovycsv.CsvParser.parseCsv


class CsvImportController {
    def emailService
    def dialerService

    def save() {

  def emailTo = params.emailAdd
  if (emailTo =~ /@myinternaldomain/) {
  def csv = request.getFile('myfile').inputStream.text
  
  def data = parseCsv(csv)

  runAsync{
   for(line in data) {
             def phone = line.Phone
            dialerService.dialNumber(phone)
   
   }
   
            emailService.sendingEmail(emailTo)

   }
  }else{
   render(view:"error") {
     div(id:"error", "E-mail Format Error: E-mail must be from @myinternaldomain.com")
    }
  }
 }
}

The two service calls (dialerService.dialNumber(phone) and emailService.sendingEmail(emailTo), now clean up the controller quite a bit.

Services

There are two new services now:
DialerService and EmailService

The DailerService looks like this:

class DialerService {

    def dialNumber(String phone) {

        println "Trying... $phone"
        def dialNum = "sipcli/sipcli.exe $phone -d [proxy ip goes here] -o 4 -t \"This is a test. this is a test. this is a test. this is a test\"-l 3".execute()
        def outFile = new File("grails-app/test.txt")
        if (dialNum.text =~ /success/){
            outFile << ("PASS on number $phone\r\n")

        } else {
            System.getProperty("line.separator")
            outFile.append("FAIL  on number $phone\r\n")

        }
    }
}

The EmailService looks like this:

class EmailService {

    def sendingEmail(String emailTo) {
        println "Emailing Report To: " + emailTo
        sendMail {
            multipart true
            to "${emailTo}"
            from "brian@someemail.com"
            subject "Provisioning Report"
            body 'Please find the attached Provisioning Report...'
            attachBytes 'grails-app/test.txt','text/csv', new File('grails-app/test.txt').readBytes()
        }
        println "Attempting to delete results file..."
        def delFile = new File("grails-app/test.txt").delete()
    }
}

GRAILS - Dynamic Scaffolding and MYSQL Set Up

Part of my development learning, I put together some notes on Grails. In this post, it covers mostly the use of dynamic scaffolding and the connection to a mysql server, as well as the use of environments. 

Scaffolding is the MVC format seem in Rails and Grails. It allows a developer to set up CRUD (Create/Update/Delete) actions super fast.   Dynamic Scaffolding is interesting because it doesn't need actual templates. You write very little code and Grails actual creates the pages and linkage for you... Grails will even create the dynamic db tables and do all the updates/deletes for you!


    Goals
        1. create a new grails app
        2. learn to use dynamic scaffolding
        3. use environments
        4. set up a mysql db
        5. connect to the mysql db
        6. run app in prod mode.

1. Create a new grails app

    Easy: grails create-app [name]
   

2. Dynamic Scaffolding

    A. First Create the Domain Class
        grails create-domain-class com.blog.Post
    B. Next Create the Scaffold controller
        grails create-scaffold-controller com.blog.Post

3. Environments

    By default we run the app in "dev" which uses H2 memory db.
    We can run other db's per environment, or have env. settings.
    For example, we can set up MYSQL for "prod" and leave H2 for dev
    To run with a spec env you do a grails prod run-app

4. Set up a MYSQL db

    First make sure MYSQL server is running
    In the MYSQL Workbench, create a new connection if you like
    Then create a new Schema - this is your db name.

5. Connect to the MYSQL db

    Back in Grails, open the file:
        /grails-app/conf/BuildConfig.groovy
        and uncomment:
        runtime 'mysql:mysql-connector-java:5.1.22'
    Next open the file:
        /grails-app/conf/DataSource.groovy
        update the url to be your msql server...
        ie "jdbc:mysql://localhost/blog"
        add
            username = [your mysql user]
        add
            password = [your mysql pass]

6. Run the App In Prod mode

    grails prod run-app

You can monitor the db you created, and watch as you
    create
    edit
    delete
via the app, the db will reflect the changes.

SIPP Simultaneous Jobs

The question came up today, about tshark... specifically if tshark/wireshark is smart enough to know which SIPP call is being captured in the media.

By default, no. 

However, you can get around this to run simultaneous jobs if you design your architecture to make use of Virtual Machines.  In my case, I have 5 Virtual Machines.  Each one runs a set of SIPP tests on a hourly schedule. 

If the SIPP tests run on top of each other (one call to phone number X and another to phone number Y), tcast will get a packet capture of both media streams... but it won't know which stream goes to whom.  So if stream #1 fails and stream #2 passes... it won't know what test to pass.

Virtual Machines

I get around this limitation by using Virtual Machines:
I set up the schedule so that Virtual Machine #1, handles Outbound tests only, with specific carriers.  I need each test to run every hour, so I have the ability to run about 6 tests... each sep. by 10min. 

For my purposes, if a test fails I retry it.  So... Test #1 runs, fails, it retries in 2min. I retry up to 3 times. This gets me 8-10 min worth of testing.  So I'm limited to 6 tests I can run per hour.

So for my design, I have Virtual Machine #1 that runs 6 outbound tests an hour.  Virtual Machine #2 runs 6 inbound tests an hour... and so forth.  This way they can overlap.

Each virtual machine has it's own Jenkins, driving it's own jobs (their own packet capture and sipp call.) 

Cloning

The easiest way to do this, is get one Virtual Machine all set up with Jenkins.  Once you have it working really well, clone it to multiple other VM's and change the jobs on each Jenkins of each VM.

SIPP & Jenkins Details

Per request, I'm adding some detail on the use of Jenkins and how I configured it to run the jobs and retain the details.

Why Jenkins?

First, why did I use Jenkins? Several reasons:

1. Jenkins is a build process, so by having the tests in Jenkins, I can kick off tests when a different job finishes building... i.e. developers push a new version of call control, and that starts the tests running
2. Jenkins allows me to retain historical data easily
3. Jenkins is push button, so anyone can come along and run the job.

Jekins Set Up with SIPP

Regarding my set up, from a high level, I needed Jenkins to kick off two simultaneous jobs... one is Running the SIPP load, and the other is capturing a PCAP during the test.  The PCAP capture may not be necessary for your tests... But if you do want it, you'll need to run the jobs simultaneously.

Simultaneous Jobs

In Jenkins I use the Multi Job plugin: https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin 

You then create a new project/job and set it up as a Multi-Job. On the job details, you  add a build option for a Multi Job Phase.  Then you can add multiple jobs to one phase.

This lets Jenkins run multiple jobs simultaneously.

Why not just have the parent job call the child jobs like job1, job2, etc.? 

the answer is due to collecting pass/fail criteria.  If you just have a Jenkins job kick off other jobs, the parent job will always pass, even if the children fail. 

The MultiJob Plugin will fail the parent job, if the children jobs fail.

Jenkins Running Sipp

To run the SIPP command I have a Jenkins job just for that.  In the Job details, under "Build Environment" I check "Execute Shell Script on Remote Host using SSH" (I believe this is avail with the SSH plugin.)  NOTE: You must define your SSH host and login on the main Jenkins configuration.
In the Execute Shell Script text box, I add:
cd /sipp-3.3
sudo sipp -s [Phone number] [Proxy] -sf /uac_pcap_g711.xml -m 2000 -mi [Proxy media IP] -d 1200 -trace_rtt -trace_err -stat_delimiter ,

That's it.  Now when this job is run, it will cd to the sipp folder, then run the sudo of sipp to call the number using our specific IP for the proxy. 

Packet Capture

In the packet capture job, I do the same thing as the SIPP job, I check off "Execute Shell Script..." but I point to a script I've made... like pcap.sh on the file structure.  In that file, I have a call to run tshark for X seconds and Output the pcap file to a specific folder.

That's it.

Then back to the MultiJob parent, it points to these two jobs, as one part of one phase.

Grails - Creating an Auto Dialer

I had a need for an internal tool where I work.

The premise is that our company buys and stores various numbers that they lease out later.  However, if those numbers do not have activity in a certain amount of time, they can become deactivated.

In the past, the company had individuals who would (every few months) get a list of current numbers in the risk zone for deactivation (hundreds of numbers) and dial each one to make sure it still went to our menu. 

So the Provisioning Team reached out to me to ask me if this could be automated....

They're requirements were that:
    a) they have lists of phone numbers (DNIS) that need to be called. this list could be a CSV
    b) they would like to upload it to a tool which would dial each number validating if it was active or not
    c) they would be notified of the results somehow.

I put together this tool and here's how I did it...

Basically I built an auto dialer with Grails and SIPCLI.  I could have used SIPP, but since I foresaw a future need to do text to speech, I opted for sipcli.  The downside to sipcli is that it is a windows tool... so you're tied to the Windows Env in this case.  But you could easily repurpose the methodology here to work with Linux using SIPP instead.

Using Grails, I created an application with:
grails create-app dialer
then I cd'd into the dialer directory and created the main controller:
grails create-controller csvImport
Once done I opened up my IDE and imported the Grails project (I use Grails Tools Suite.)

View

In the View I did this to capture the user's CSV file:
        <div id="status" role="complementary">
            <h1>Upload your CSV file of phone numbers</h1>
            <g:form controller="CsvImport" method="post" action="save"
                enctype ="multipart/form-data">
                   E-mail Results to a @mydomain.com address:<br>
                <g:textField type="field" name="emailAdd" value="brian@mydomain.com" style='width: 500px;' required=""/><p/>
                <input type="file" name="myfile" required/>
                <g:actionSubmit value="Start Process" action="save"/>
            </g:form>
        </div>

The controller looks like this:
   def save() {
        def emailTo = params.emailAdd
           if (emailTo =~ /@mydomain.com/) {
        def csv = request.getFile('myfile').inputStream.text
       
        def data = parseCsv(csv)
        runAsync{
            for(line in data) {
            println "Trying... $line.Phone"
            def dialNum = "sipcli/sipcli.exe $line.Phone -d **.**.*.*** [masked IP of our proxy] -o 4 -t \"This is a test. this is a test. this is a test. this is a test\"-l 3".execute()
            def outFile = new File("grails-app/test.txt")
            if (dialNum.text =~ /success/){               
                outFile << ("PASS on number $line.Phone\r\n")

            } else {
                System.getProperty("line.separator")
                outFile.append("FAIL on number $line.Phone\r\n")

            }
            }
 
            sendMail {
                multipart true
                to "${emailTo}"
                from "SOMEONE@ADDRESS.COM"
                subject "Provisioning Report"
                body 'Please find the attached Provisioning Report...'
                attachBytes 'grails-app/test.txt','text/csv', new File('grails-app/test.txt').readBytes()
            }
} else {
   render "Error: Email does not conform to @mydomain.com" }
            }
           
         }

To explain how it works...
The view is pretty easy to understand, it just takes a file and passes it to the controller.

Controller

The controller isn't doing any special validation, since this is a internal tool.  It takes the CSV and expects to have a column header called "Phone."  Phone will have a list of DNIS (or numbers) that need to be called.

CSV Parsing

To parse the CSV I import:
@Grab('com.xlson.groovycsv:groovycsv:1.0')
import static com.xlson.groovycsv.CsvParser.parseCsv
Then use this call def csv = request.getFile('myfile').inputStream.text to grab the input and finally pass it to the parser:
def data = parseCsv(csv)

Running Asynchronously

I don't want the  user to wait 30min for 500 phone numbers to be dialed... so instead, I send them immediately to a  page... this is handled with runAsync... this is a plugin called executor.  The Grails Executor plugin will run a closure asynchronously so you can do other stuff while the longer method runs.

Inside the runAsync closure is this for loop and if statement:
            for(line in data) {
            println "Trying... $line.Phone"
            def dialNum = "sipcli/sipcli.exe $line.Phone -d **.**.*.*** [masked IP of our proxy] -o 4 -t \"This is a test. this is a test. this is a test. this is a test\"-l 3".execute()
            def outFile = new File("grails-app/test.txt")
            if (dialNum.text =~ /success/){               
                outFile << ("PASS on number $line.Phone\r\n")
               
               
            } else {
                System.getProperty("line.separator")
                outFile.append("FAIL on number $line.Phone\r\n")

            }

Basically it's doing this:

For each line in the csv file, it prints out the phone number it's trying, and then I've defined an action to run the sipcli sip client.... to call that same number.  SipCli is a awesome command line sip tool for windows that can be used to find sip problems and issues.  It's very light and easy to use.  In this case I have it set to a 4 second timeout and i'm telling it to read the text, "This is a test..." when it makes the phone connection.

Assertions

The assertion of whether or not the phone call is valid is via the regex i'm doing in the if statement... If dialNum.text has success then we output to a file "Pass on number [DNIS]"
However, if the number fails to connect, we append to the same file "FAIL on number [DNIS]"

E-Mail

Finally, at the end, outside the runAsync closure, I do a call to email the results (that flat file) to a recipient, using the Grails mail plugin.  I'm passing the To value from the form... and as you may have seen, I'm forcing the tool to only work if an internal email is passed through.  The domain "mydomain.com" is a filler for the real domain I check. Since the tool uses an internal SMTP server, we can't send to outside emails... so I dont want the user to violate the SMTP capabilities.  Their email address entered (if from the right domain) is added to the To line below:
             sendMail {
                multipart true
                to "${emailTo}"
                from "SOMEONE@ADDRESS.COM"
                subject "Provisioning Report"
                body 'Please find the attached Provisioning Report...'
                attachBytes 'grails-app/test.txt','text/csv', new File('grails-app/test.txt').readBytes()
            }

That's it.  

Grails Project: Using Grails as a UI Wrapper to a CLI like SIPP

For many people this project is going to be Easy.  It's very simple.

The idea for this came from a situation at work.

I am a sole user of a command line SIP/VOIP tool called SIPP.  The tool is incredibly complex. It took me many weeks to get it up and running, but I know it well now and use it often. 

However, other members of the team could also benefit form the use of the tool, but some members of the office may not have a computer background, or SSH access to run the command line tool itself.

You may have command line tools you'd like to offload to others, but don't want them to make mistakes... so maybe using a GUI to wrap up the commands and constrain the options is useful for you as well.

To that end, I came up with an idea to play with a bit of Grails and get a Web Application stood up, that would power the command line tool itself.

The Setup (I wont cover these aspects):

First, in my case the tool (SIPP) needs to be installed
Second, Grails needs to be installed on the same environment

What I'll cover:

Building the Grails app
Making the calls to run the command line tool
Making the calls run in the background using Executor
Daemonize the app so it can run/stop and start on restart.

Pulling in server stats via Cacti will be covered in a different post.

Building the Grails app

This is pretty simple. There's not much difficulty in this task. 
So first things first... we need to know the command line tool parameters.  In my case I'm using SIPP.  The most common parameters I use are:
sipp -s [DNIS] [PROXY] -r [CALLS PER SECOND] -m [MAX CALLS] -sf [SCENARIO FILE] -d [DELAY OF THE CALL]

So I'll need to make a application that has a form with user selects for those items.  The PROXY in my case is going to be hard coded, as I don't want people accidentally sending traffic to the wrong IP/Host. 

In my case I only foresaw the need for one controller and a couple Views.  If you're not familiar with MVC, you might want to read up on it a bit.  Basically the Controller will handle logic, the View will be the display/rendering aspect of the app and the Model is the data handling.

So I'm going to make a super easy app that has one controller, and some views.

So lets start:

1. Make the grails app.  Go to a folder of your choice, and type grails create-app  At the prompt for a name, give it a name and hit enter.
2. It will now make a folder with the name of the Grails App that you just created.  Go ahead and enter the folder - cd [APP NAME]
3. In the app folder create a controller with the command grails create-controller At the prompt for a name, give it a name and hit enter.

Let's launch an IDE to edit this... I  use Spring's Grails Tool Suite.  I love Intellij, but the community (free) edition doesn't currently support Grails.  Only the paid ($500+) version does.

Import the project you just made.  In Grails Tool Suite it's File / Import / Project - Grails.

Once imported, lets edit the controller.

In the Project Tree in your IDE you'll see [web-app name]/controllers/[project-name]/[your-controller]

Go ahead and open that up in the IDE.

You'll have some code with a index action like
class SippcallController {
     def index() {
                    }
  }

I made a action that was more useful for what I'm doing.  So I have:
def callLogic() {
    def sipp = "sipp -s 18008888888 10.98.1.1 -r 1 -m 1 -sf uac.xml -d 1200".execute()
  }

Initially I just put in the actual command to dial a number via SIPP.  Then I tested it... it worked... so at that point I replaced all the parameters with values that will be coming from a user Form.  So the action became:

def callLogic() {
    def sipp = "sipp -s ${params.phoneNumber} 10.98.1.1 -r $(params.Rate} -m ${params.Max} -sf ${params.scenario} -d ${params.delay}".execute()

[view_darta:sipp.text]
  }

That last bit, [view_data:sipp} is a model I can access in a View or web page.  In a webpage of this app, I can just have ${view_data} and it will reference the output of the SIPP sip call itself.

Creating the Web Form for User Input

Now lets go ahead and edit the index.gsp file.  It's listed in the Project Explorer as: Project Name / Views /

Once that's open, you'll notice the basic sample page that Grails creates with each new application.  I removed all the Grails code within the Body Tags and replaced it with a form.... using g tags.

My form is like this ( the Grails tags are highlighted in green):
        <g:form name="callForm" controller="makeCall" action="callLogic">
        <p><h3>Phone number:</h3></p>
        <g:select name="phoneNumber" from = "${['12132830920', '12132830912', '12132830591', '13237549121'] }" value="12132830920" noSelection="['':'-Choose the phone number -']"/>
        <p><h3>Calls Per Second:</h3></p>
        <g:select name="cps" from="${1..30}" value="1"/>
        <p><h3>Max Calls:</h3></p>
        <g:select name="maxCount" from="${['1', '10', '50', '100', '500', '1000', '2000']}" value="1"/>
        <p><h3>How Long to Hold Audio Open (ms):</h3></p>
        <g:select name="delay" from="${['1200', '5000', '10000', '20000', '60000', '120000']}" value="1000"/>
          <p><h3>Scenario:</h3> </p>
          <g:select name="scenario" from ="${['uac.xml', 'codec_speex.xml', 'codec_g729.xml', 'carrier_sprint.xml'] }"
            value = "uac.xml" />
        <g:actionSubmit value="Place Call" action="callLogic"/>
        </g:form>

It's pretty easily readable. A few things to note:

• To handle multiple selections in a dropdown, you can use a function or just list them out like I did here.  A function is superior, but this was quick and easy to set up a test.  you make the g tag a select and you add from with "${['value1', 'value2']}" and so forth.  
• Like a regular form, you can set the default value with value="value1"  
• Make sure the form references your controller and action correctly.

 If done right, this should send the field data to your Controller and Action correctly... which would run the command line with the parameters supplied via this form. 

Creating an Output View

 

In my case I decided to make another page that would handle the output form the controller.  It's the same name as the controller.  So in the [project]/views/[subfolder]/callLogic.gsp page I will put some data from the SIPP tool....

I could add ${view_data}, and it would display the output from the SIPP test into the page itself.

In my case, as you can see in the screenshot, I'm pulling in server stats via Cacti.  It's too much to go into in this blog post, but I set up Cacti on the linux VM here that I'm using, to pull snmp data from the box I'm driving the SIPP load to.

When I first did this, the page loaded fine, but it would wait for SIPP to finish, then output the data to the view.

I now send the call to a background job using the Executor plugin for grails, and using the runAsync closure around the SIPP call in the controller.  This way once a user submits the form they load on the results page.  The data is still being collected and will need to be updated later onto the page (I haven't worked that part out yet.)

Installing Executor (to run background jobs)

In case you need this, here's how it works... you do a grails install-plugin executor
After it installs, you can then see the plugin listed in your IDE's project tree like so: project/plugins.  You can now use the methods the plugin gives access to.

In my case, I just wanted the runAsync method.  so back in my controller I added it like this:

def callLogic() {
    def sipp = "sipp -s ${params.phoneNumber} 10.98.1.1 -r $(params.Rate} -m ${params.Max} -sf ${params.scenario} -d ${params.delay}".execute()

runAsync{
[view_darta:sipp.text]
}
  }

This allows the output to be collected in the background. 

The part I haven't worked out yet, is getting a push or polling mechanism set up to get that data once it's ready.

Daemonizing It
I wanted to be able to start and stop this thing, so I set up a job in the /etc/init.d/ folder and called it sippgui  in that file is a series of commands, such as:

start() {
        cd /sippcall
        /sippcall/grails run-app -Dserver.port=8090 &
        sudo touch /var/lock/subsys/sippgui
        echo

}

You'll want to do something similar to stop it.

At that point it can be added to cron, or run manually with /etc/init.d/sippgui start|stop|restart, etc.

Btw. the -Dserver.port=8090 is how i'm setting the port it's using. 

Opening the Port

That's it... Oh WAIT one more thing...

If you can't get this App to load, you may need to open your firewall to allow traffic to this port (i.e. 8090.)  In my case I couldn't  use 8080, so I used a non standard port. I had to modify the IPTABLES to allow this. 

CENTOS uses IPTABLES so that's my solution. I won't go into details on it, just know that you'll need to open any non standard port and may need to google how to do that on whatever OS you're using. 

Setting up Raspberry Pi

This is a departure from the main crux of this blog.  But I wanted to gather some notes on my pitfalls I encountered with setting up a Raspberry Pi device.

Hardware and SD Card Set Up

I had the version 2 B 512 version of the Pi.
I used a PNY 16GB class 6 SD card
I used a solar/electric battery to power it
I used a usb keyboard
and a standard hdmi cable

I started this all off by downloading the official Raspberry "wheezy" distro (2013-05-25-wheezy-raspbian) from the direct download link.  However, I couldn't unzip it on windows... although the download was the correct size, windows' default unarcher gave me a "can't read" this file error.  However, 7zip would read it and uncompress it... I figured this was just a problem with Windows.

Butut when I used the img file that it produced to drop on the SD card... it corrupted the SD card.

Again - this was the direct download official link of the "wheezy" distro.  

At this point he SD card would read "18mb left of 56mb" yet this was a 16Gig card.  I freaked out a bit thinking the card was now damaged.  However I read that it only needs to be reformatted....

I tried to reformat using Windows' reformat tool (right click / format)... but the card would always read "56mb avail."

I in fact had to now download the official SD formatter... ugh.  Ok, Now that got me back to 16gigs.

I tried multiple downloads from that direct download link - all with the same problem!

Then I tried the torrent.

They recommend the torrent, but I hate using torrents.  I always end up with spyware from the torrent clients... anyway, I installed bit torrent and downloaded the distro via the torrent, and then uninstalled bit torrent afterwards.

I did get the torrent to download, it was about 10x faster then the direct link.  I also was able to unzip it fine without the error's encountered from the direct link.

I put the SD card back in the Pi, and plugged it back in... It booted up great.

Configuration

When it boots up successfully you'll see something resembling the old PC Bios menu.  This is how you'll configure your Raspberry Pi.

At the configuration screen, you'll want to:
1. set the thing to use the entire space on the SD card, it's the first option in this boot menu.
2. after that, if you want to boot to a desktop, go to that option, if you use a console only, skip it.
3. To turn on SSH (so you can remote into the box via SSH) you need to go to advanced settings and then choose SSH and then Enable.
4. If this is a public facing device, you might want to change the password as well - it's a option on the boot menu.
5. I also changed the hostname (it defaults to 'raspberrypi')

After that I chose to finish and reboot.

From here on out, you'll log in as your user "pi" with the password you picked... or use the default (raspberry.)

SSH

To SSH to this box, first get it's IP by running ifconfig from the command line.  You'll see "eth0" and in there a line inet addr:.  take that address.

From another box, using either terminal on mac, or Putty/supper putty on PC, you can just ssh over to:
ssh [ip]
login as user pi and input your pass.

Installing Software from the Command Line

If you're used to installers in Linux (like Centos' YUM) then you'll get the same thing here, but it's apt-get.  You'll sudo each command like this.  So: sudo apt-get iw tshark would install iw and tshark (for packet sniffing.)

Find the tools you want and they should give you the proper apt-get install commands.

Shutting Down the Raspberry Pi

Also another important thing.  The official site for Raspberry Pi, says to shut it down you just "unplug it" from what I've read this could damage the SD card.  So... you must run an official shutdown:

sudo shutdown -h now

From the UI, you click the red icon and choose shutdown.  Wait for the proper screen statement "system halted," and then remove the power cable.