Backups and Redundancy with Asterisk

One of the first things you learn after you purchase a new aquarium is that you're much better off with two tanks rather than one. This generally doesn't make the little Mrs. too happy but, after you explain that each dead fish represents a $100 bill, most wives usually come around. YMMV! You'll learn much the same thing with Asterisk®. It's not that new versions are that terrible. They're not. In fact, they're generally a quantum leap better and less buggy than the version you're replacing. It's just that programmers have this death wish to always "improve" everything. I've had lots of programmers work with me over the years, and the hardest thing to get across to them short of branding or tattooing the message directly onto their heads is this: "If It Ain't Broke, Don't Fix It!" The gotcha in Asterisk 1.2 was leaving out the sound card driver, chan_oss.so, which broke every company's paging system ... at least all those companies that now depend upon Asterisk. Digium® apparently has fixed the problem in the just released Asterisk 1.2.1, but it still ranks up there as a big mistake that should and could have been avoided. The silver lining in this tale is that it prompted a single individual, Tracy Carlton, to stay up an entire night trying to figure out what had come unglued. In the process, Tracy uncovered why the chann_oss.so driver has never worked with half-duplex sound cards, you know, the type that comes standard in virtually every PC on the planet. Guess what? Now it works with one simple tweak. You can read all about sound cards and paging in Tracy's post on SourceForge. Don't miss it! And, speaking of not missing it, the next time you come through our hometown of Atlanta, don't miss the new Georgia Aquarium (see insets). Known affectionately as Fish Depot because it was funded privately by Bernie Marcus of Home Depot fame, it's truly one of a kind and destined to become one of this country's national treasures. The Showstopper: an unbelievably friendly, 3,000 pound Beluga whale (inset below). And the mascot: a little orange fish called a garibaldi appropriately named ... Deepo.

What does all this have to do with Backups and Redundancy? Plenty. You'll save yourself and your customers a boatload of grief by thoroughly testing new Asterisk releases before you put them into production. Asterisk is such an exciting technology that it's always tempting to deploy the latest, greatest version immediately to take advantage of some new bell or whistle. The downside is what happened with paging in Asterisk 1.2. I'm no longer dumbfounded by developers that don't try everything before releasing an upgrade. There just isn't time, and usually it's not very exciting work either. Plus, it requires a substantial hardware investment. Your best defense is a sandbox (aka clunker) PC that can be used for all sorts of things. First, you can try out new releases. And second, for about $100 on Ebay, you can buy a great Asterisk@Home PC which can do all sorts of other things, too ... like automatic full system backups. If you're feeling rich, buy another Wal-Mart special for under $200, and you've got a great alternate server/backup/sandbox all rolled into one. Or, if you previously built our dual-system Bluetooth Proximity Detection System, then you're all set. And, yes, we're going to get software RAID 2 working with Asterisk@Home 2 one of these days for those that want a mirrored hard disk, but first things first. RAID doesn't offer the same protection as a good, frequent backup. Mirrored RAID drives immediately copy everything from one disk to another, mistakes and all. So, if you clobber a config file or driver by pressing the wrong button, a mirrored drive won't help you at all. It'll just give you a second copy of your original mess. So today we're going to show you a painless backup strategy that'll get you through almost any disaster except perhaps Katrina or a fire that totally destroys your home or office.

Overview. Our project for today is to build a bullet-proof backup system between your two Asterisk@Home systems. Yes, Asterisk@Home can be configured to make backups, but it puts them on the same hard disk as your system, not too helpful with a disk crash. That's not to say you shouldn't implement the Asterisk@Home backups. They're a great resource when you accidentally clobber something. It's easy: AMP->Setup->Backup and Restore->Add Backup Schedule. Give it a name and set everything to Yes. Then select Run Daily at Midnight. Click Submit Changes and then the red bar to reload Asterisk. Just don't forget to clean out some of the daily backups from time to time. They live in /var/lib/asterisk/backups under a directory name matching whatever you named your backup schedule.

The dual-PC backup system we're going to build today works well for almost any two Linux systems since Asterisk@Home really is a Linux system. It also works swimmingly on systems running BSD or MacOS X. If you need help getting either or both of your Asterisk systems set up, start here. And the software cost of the project: still $0. For those with a small business and one or more satellite offices, next week we'll tackle tying multiple Asterisk systems together for instant communications between your offices. Yet another freebie for those that deploy today's backup system. But we're getting ahead of ourselves. The drill for today is to use our previous SSH automatic login trick to enable us to do scheduled backups in the background on one or both of your systems. Lucky for us, CentOS includes the rsync utility which makes mirror image backups using SSH as the transport between two systems. Here's the step-by-step if you want to follow along.

Password-Free SSH Access. As with our Proximity Detection System, we want to use SSH for our backups to protect our sensitive data. After all, there's no reason that your two Asterisk@Home systems have to be in the same building. One can be across town or around the globe. This backup system will work exactly the same ... just slower. But to automate the process, we need to use crontab to schedule the backups. That means there will be no user interaction in the backup process. As you know, SSH typically prompts for a password when you connect to a remote resource. So here's the trick we introduced in our Bluetooth project. We'll do the same thing for backups. And, if you want to backup both of your systems, you'll need to perform this trick in both directions, i.e. from Server A to Server B and from Server B to Server A. For ease of explanation, we're going to refer to the servers as Asterisk Main and Asterisk Backup. You can extrapolate for your own systems from there. We're going to go through this example assuming you want to backup the data on Asterisk Main to Asterisk Backup. Once you complete the steps, there's no reason you can't reverse the process and also backup Asterisk Backup to Asterisk Main. We'll leave that for you to work out on your own. Just let us know if you get stumped.

To begin, log in to Asterisk Main as root. Then, from the command prompt, issue the following command: ssh-keygen -t rsa. Press the enter key three times. You should see something similar to the following. The file name and location in bold below is the information we need:

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
1d:3c:14:23:d8:7b:57:d2:cd:18:70:80:0f:9b:b5:92 root@asterisk1.local

Now copy the file in bold above to your Asterisk Backup server from Asterisk Main using SCP. The command should look like the following (except use the private IP address of your Asterisk Backup server instead of 192.168.0.104). Provide the root password to Asterisk Backup when prompted to do so.

scp /root/.ssh/id_rsa.pub root@192.168.0.104:/root/.ssh/authorized_keys

Once the file has been copied, you now should be able to log in to Asterisk Backup from Asterisk Main with SSH without being prompted for a password. Let's try it. Log in to Asterisk Backup from Asterisk Main with the following command using the actual IP address of Asterisk Backup, of course:

ssh root@192.168.0.104

You should be admitted without entering a password. If not, repeat the drill or read the complete article and find where you made a mistake. While you're logged in to Asterisk Backup, let's create the directory where your Asterisk Main system will be stored: mkdir /backup. We're assuming you have sufficient disk space on Asterisk Backup to store the complete contents of Asterisk Main. If you're not sure, issue the df command on both systems and be sure. Here's a sample from our system:

[root@asterisk1 backup]# df

Filesystem 1K-blocks Used  Available Use% Mounted on
/dev/hda2  27980732 4625788 21933596 18% /
/dev/hda1   101086   11077  84790    12% /boot
none         62612       0  62612     0% /dev/shm


The device we care about is /dev/hda2. That's where your backup data will reside. The Available column will tell you how much space is left on Asterisk Backup. When you run the same df command on Asterisk Main, the Used column for /dev/hda2 will tell you how much room you need to reserve on Asterisk Backup to hold the backup. Finally, if you should ever want to delete the /backup directory and all the backup data stored on Asterisk Backup, here's the command to do it: rm -rfd /backup. You get no warnings! Now log out of Asterisk Backup by typing exit. Then rerun the df utility on Asterisk Main to see how much space will be needed for the backup. Incidentally, all of the remaining steps in this tutorial will be performed on Asterisk Main.

Using rsync for Asterisk@Home Backups. There are a number of great tutorials on rsync. If you want to learn more than what's covered below, start here. We did. The actual command we're going to use to make our backups is shown below. It will make the first backup and then freshen it each time you run it in the future. We recommend running the command manually the first time to be sure everything is working as it should. When the rsync session completes (5-10 minutes on a new system with a 100 megabit Ethernet connection between the servers), issue the following command to check the rsync log for errors: tail /var/log/backup.log. If errors are reported, you can open the log with nano (nano /var/log/backup.log) to search for errors (Ctrl-W, failed to open). Then close nano (Ctrl-X, n). Typically, errors involve changes during the backup process, such as deleting a voicemail message while the backup is in process. Normally, running rsync again will solve any problems. Once you get a clean backup, move on to the Automating Backups step below.

rsync -avvz --delete --exclude "/proc/" --exclude "/mnt/" --exclude "[cC]ache" -e ssh / root@192.168.0.104:/backup/ > /var/log/backup.log

WARNING: If you cut-and-paste the code above, make sure you adjust the quotation marks which get mangled by the WordPress blog.
Both Linux and Asterisk like and want regular quotation marks in all commands or disaster awaits!

The only change you'll need to make in the command above is to replace the quotes and the 192.168.0.104 IP address with the actual IP address of your Asterisk Backup server. This will backup everything on Asterisk Main except cache files, the /mnt directory (which includes the contents of any mounted CD-ROM), and the /proc directory (which includes a bunch of stuff that would be unchanged if you did a clean Asterisk@Home install).

Restoring Your Asterisk@Home System. For future reference, to do a full system restore of Asterisk Main with a new hard disk, you would first reinstall Asterisk@Home from the same installation disk used to create Asterisk Main. Then you would restore the backup data from Asterisk Backup by logging into Asterisk Main as root and manually issuing the following commands:

cd /
rsync -avv -e ssh root@192.18.0.104:/backup/ . > /var/log/restore.log

Of course, we're assuming the IP address of your Asterisk Backup server is 192.168.0.104 in the above example so change it to properly reflect your Asterisk Backup's IP address. And don't forget the space and period and space in the above command. That's what tells rsync to restore your backup to the current folder location.

Automating Backups. Now that you have rsync working reliably, we want to automate the Linux backup process so that it runs each night without user intervention. This will assure that you never lose more than one day's data which usually isn't life threatening in the telephone business. If it is, you can adjust the crontab schedule to better meet your requirements. To begin, download the backup script here. Now log in as root and copy the script to /etc/asterisk on your Asterisk Main server. Reset the permissions: chmod 775 /etc/asterisk/backup.sh. Fix the ownership: chown asterisk:asterisk /etc/asterisk/backup.sh. And adjust the IP address in the file to match the IP address of your Asterisk Backup server: nano -w /etc/asterisk/backup.sh. Finally, add the script to your crontab to kick off at 2:01 a.m. each morning, save your update (Ctrl-X, Y, then enter) and you're all set to go.

export EDITOR=nano
crontab -e
01 02 * * * /etc/asterisk/backup.sh >/dev/null 2>&1


Coming Attractions. Yes, we're finally going to cover faxing later this week or early next week. And, the #1 requested feature for Asterisk, call presence monitoring with pickup button support, is now a reality. And soon we'll show you how to use it to mimic the key telephone instruments that corporate America just can't seem to live without. There also are new versions of both Asterisk (1.2.1) and Asterisk@Home (2.2) with more bug fixes so we'll have an updated HOW-TO guide for you shortly. Finally, as mentioned previously, we'll connect up our two new Asterisk servers to communicate with each other and start sharing extensions and dialplans as soon as Weekly UPDATE-itis comes to a close.

BroadVoice in the News. Well, it looks like our old pals at BroadVoice are back in the news ... at least with the Better Business Bureau. See this link where BroadVoice finishes in second place for most complaints out of 77 telephone providers over a three year period. And, better yet, BroadVoice would retain or perhaps better their ranking if the numbers were based upon just the past 12 months. In fact, all but one of the complaints filed against BroadVoice were filed within the past 12 months. And, yes, there is a silver lining to this story, too. BroadVoice for the most part gives BBB complaints the same great treatment that we've received when inquiring about a problem ... nada!