Home » Technology » Backups and Redundancy with Asterisk

The Most Versatile VoIP Provider: FREE PORTING

Backups and Redundancy with Asterisk

Georgia AquariumOne of the first things you learn after you purchase a new aquarium is that you're much better off with two tanks rather than one. This generally doesn't make the little Mrs. too happy but, after you explain that each dead fish represents a $100 bill, most wives usually come around. YMMV! You'll learn much the same thing with Asterisk®. It's not that new versions are that terrible. They're not. In fact, they're generally a quantum leap better and less buggy than the version you're replacing. It's just that programmers have this death wish to always "improve" everything. I've had lots of programmers work with me over the years, and the hardest thing to get across to them short of branding or tattooing the message directly onto their heads is this: "If It Ain't Broke, Don't Fix It!" The gotcha in Asterisk 1.2 was leaving out the sound card driver, chan_oss.so, which broke every company's paging system ... at least all those companies that now depend upon Asterisk. Digium® apparently has fixed the problem in the just released Asterisk 1.2.1, but it still ranks up there as a big mistake that should and could have been avoided. The silver lining in this tale is that it prompted a single individual, Tracy Carlton, to stay up an entire night trying to figure out what had come unglued. In the process, Tracy uncovered why the chann_oss.so driver has never worked with half-duplex sound cards, you know, the type that comes standard in virtually every PC on the planet. Guess what? Now it works with one simple tweak. You can read all about sound cards and paging in Tracy's post on SourceForge. Don't miss it! And, speaking of not missing it, the next time you come through our hometown of Atlanta, don't miss the new Georgia Aquarium (see insets). Known affectionately as Fish Depot because it was funded privately by Bernie Marcus of Home Depot fame, it's truly one of a kind and destined to become one of this country's national treasures. The Showstopper: an unbelievably friendly, 3,000 pound Beluga whale (inset below). And the mascot: a little orange fish called a garibaldi appropriately named ... Deepo.

Beluga WhaleWhat does all this have to do with Backups and Redundancy? Plenty. You'll save yourself and your customers a boatload of grief by thoroughly testing new Asterisk releases before you put them into production. Asterisk is such an exciting technology that it's always tempting to deploy the latest, greatest version immediately to take advantage of some new bell or whistle. The downside is what happened with paging in Asterisk 1.2. I'm no longer dumbfounded by developers that don't try everything before releasing an upgrade. There just isn't time, and usually it's not very exciting work either. Plus, it requires a substantial hardware investment. Your best defense is a sandbox (aka clunker) PC that can be used for all sorts of things. First, you can try out new releases. And second, for about $100 on Ebay, you can buy a great Asterisk@Home PC which can do all sorts of other things, too ... like automatic full system backups. If you're feeling rich, buy another Wal-Mart special for under $200, and you've got a great alternate server/backup/sandbox all rolled into one. Or, if you previously built our dual-system Bluetooth Proximity Detection System, then you're all set. And, yes, we're going to get software RAID 2 working with Asterisk@Home 2 one of these days for those that want a mirrored hard disk, but first things first. RAID doesn't offer the same protection as a good, frequent backup. Mirrored RAID drives immediately copy everything from one disk to another, mistakes and all. So, if you clobber a config file or driver by pressing the wrong button, a mirrored drive won't help you at all. It'll just give you a second copy of your original mess. So today we're going to show you a painless backup strategy that'll get you through almost any disaster except perhaps Katrina or a fire that totally destroys your home or office.


Overview. Our project for today is to build a bullet-proof backup system between your two Asterisk@Home systems. Yes, Asterisk@Home can be configured to make backups, but it puts them on the same hard disk as your system, not too helpful with a disk crash. That's not to say you shouldn't implement the Asterisk@Home backups. They're a great resource when you accidentally clobber something. It's easy: AMP->Setup->Backup and Restore->Add Backup Schedule. Give it a name and set everything to Yes. Then select Run Daily at Midnight. Click Submit Changes and then the red bar to reload Asterisk. Just don't forget to clean out some of the daily backups from time to time. They live in /var/lib/asterisk/backups under a directory name matching whatever you named your backup schedule.

The dual-PC backup system we're going to build today works well for almost any two Linux systems since Asterisk@Home really is a Linux system. It also works swimmingly on systems running BSD or MacOS X. If you need help getting either or both of your Asterisk systems set up, start here. And the software cost of the project: still $0. For those with a small business and one or more satellite offices, next week we'll tackle tying multiple Asterisk systems together for instant communications between your offices. Yet another freebie for those that deploy today's backup system. But we're getting ahead of ourselves. The drill for today is to use our previous SSH automatic login trick to enable us to do scheduled backups in the background on one or both of your systems. Lucky for us, CentOS includes the rsync utility which makes mirror image backups using SSH as the transport between two systems. Here's the step-by-step if you want to follow along.

Password-Free SSH Access. As with our Proximity Detection System, we want to use SSH for our backups to protect our sensitive data. After all, there's no reason that your two Asterisk@Home systems have to be in the same building. One can be across town or around the globe. This backup system will work exactly the same ... just slower. But to automate the process, we need to use crontab to schedule the backups. That means there will be no user interaction in the backup process. As you know, SSH typically prompts for a password when you connect to a remote resource. So here's the trick we introduced in our Bluetooth project. We'll do the same thing for backups. And, if you want to backup both of your systems, you'll need to perform this trick in both directions, i.e. from Server A to Server B and from Server B to Server A. For ease of explanation, we're going to refer to the servers as Asterisk Main and Asterisk Backup. You can extrapolate for your own systems from there. We're going to go through this example assuming you want to backup the data on Asterisk Main to Asterisk Backup. Once you complete the steps, there's no reason you can't reverse the process and also backup Asterisk Backup to Asterisk Main. We'll leave that for you to work out on your own. Just let us know if you get stumped.


To begin, log in to Asterisk Main as root. Then, from the command prompt, issue the following command: ssh-keygen -t rsa. Press the enter key three times. You should see something similar to the following. The file name and location in bold below is the information we need:

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
1d:3c:14:23:d8:7b:57:d2:cd:18:70:80:0f:9b:b5:92 root@asterisk1.local

Now copy the file in bold above to your Asterisk Backup server from Asterisk Main using SCP. The command should look like the following (except use the private IP address of your Asterisk Backup server instead of 192.168.0.104). Provide the root password to Asterisk Backup when prompted to do so.

scp /root/.ssh/id_rsa.pub root@192.168.0.104:/root/.ssh/authorized_keys

Once the file has been copied, you now should be able to log in to Asterisk Backup from Asterisk Main with SSH without being prompted for a password. Let's try it. Log in to Asterisk Backup from Asterisk Main with the following command using the actual IP address of Asterisk Backup, of course:

ssh root@192.168.0.104

You should be admitted without entering a password. If not, repeat the drill or read the complete article and find where you made a mistake. While you're logged in to Asterisk Backup, let's create the directory where your Asterisk Main system will be stored: mkdir /backup. We're assuming you have sufficient disk space on Asterisk Backup to store the complete contents of Asterisk Main. If you're not sure, issue the df command on both systems and be sure. Here's a sample from our system:

[root@asterisk1 backup]# df

Filesystem 1K-blocks Used  Available Use% Mounted on
/dev/hda2  27980732 4625788 21933596 18% /
/dev/hda1   101086   11077  84790    12% /boot
none         62612       0  62612     0% /dev/shm

The device we care about is /dev/hda2. That's where your backup data will reside. The Available column will tell you how much space is left on Asterisk Backup. When you run the same df command on Asterisk Main, the Used column for /dev/hda2 will tell you how much room you need to reserve on Asterisk Backup to hold the backup. Finally, if you should ever want to delete the /backup directory and all the backup data stored on Asterisk Backup, here's the command to do it: rm -rfd /backup. You get no warnings! Now log out of Asterisk Backup by typing exit. Then rerun the df utility on Asterisk Main to see how much space will be needed for the backup. Incidentally, all of the remaining steps in this tutorial will be performed on Asterisk Main.


Using rsync for Asterisk@Home Backups. There are a number of great tutorials on rsync. If you want to learn more than what's covered below, start here. We did. The actual command we're going to use to make our backups is shown below. It will make the first backup and then freshen it each time you run it in the future. We recommend running the command manually the first time to be sure everything is working as it should. When the rsync session completes (5-10 minutes on a new system with a 100 megabit Ethernet connection between the servers), issue the following command to check the rsync log for errors: tail /var/log/backup.log. If errors are reported, you can open the log with nano (nano /var/log/backup.log) to search for errors (Ctrl-W, failed to open). Then close nano (Ctrl-X, n). Typically, errors involve changes during the backup process, such as deleting a voicemail message while the backup is in process. Normally, running rsync again will solve any problems. Once you get a clean backup, move on to the Automating Backups step below.

rsync -avvz --delete --exclude "/proc/" --exclude "/mnt/" --exclude "[cC]ache" -e ssh / root@192.168.0.104:/backup/ > /var/log/backup.log

WARNING: If you cut-and-paste the code above, make sure you adjust the quotation marks which get mangled by the WordPress blog.
Both Linux and Asterisk like and want regular quotation marks in all commands or disaster awaits!


The only change you'll need to make in the command above is to replace the quotes and the 192.168.0.104 IP address with the actual IP address of your Asterisk Backup server. This will backup everything on Asterisk Main except cache files, the /mnt directory (which includes the contents of any mounted CD-ROM), and the /proc directory (which includes a bunch of stuff that would be unchanged if you did a clean Asterisk@Home install).

Restoring Your Asterisk@Home System. For future reference, to do a full system restore of Asterisk Main with a new hard disk, you would first reinstall Asterisk@Home from the same installation disk used to create Asterisk Main. Then you would restore the backup data from Asterisk Backup by logging into Asterisk Main as root and manually issuing the following commands:

cd /
rsync -avv -e ssh root@192.18.0.104:/backup/ . > /var/log/restore.log

Of course, we're assuming the IP address of your Asterisk Backup server is 192.168.0.104 in the above example so change it to properly reflect your Asterisk Backup's IP address. And don't forget the space and period and space in the above command. That's what tells rsync to restore your backup to the current folder location.


Automating Backups. Now that you have rsync working reliably, we want to automate the Linux backup process so that it runs each night without user intervention. This will assure that you never lose more than one day's data which usually isn't life threatening in the telephone business. If it is, you can adjust the crontab schedule to better meet your requirements. To begin, download the backup script here. Now log in as root and copy the script to /etc/asterisk on your Asterisk Main server. Reset the permissions: chmod 775 /etc/asterisk/backup.sh. Fix the ownership: chown asterisk:asterisk /etc/asterisk/backup.sh. And adjust the IP address in the file to match the IP address of your Asterisk Backup server: nano -w /etc/asterisk/backup.sh. Finally, add the script to your crontab to kick off at 2:01 a.m. each morning, save your update (Ctrl-X, Y, then enter) and you're all set to go.

export EDITOR=nano
crontab -e
01 02 * * * /etc/asterisk/backup.sh >/dev/null 2>&1

Coming Attractions. Yes, we're finally going to cover faxing later this week or early next week. And, the #1 requested feature for Asterisk, call presence monitoring with pickup button support, is now a reality. And soon we'll show you how to use it to mimic the key telephone instruments that corporate America just can't seem to live without. There also are new versions of both Asterisk (1.2.1) and Asterisk@Home (2.2) with more bug fixes so we'll have an updated HOW-TO guide for you shortly. Finally, as mentioned previously, we'll connect up our two new Asterisk servers to communicate with each other and start sharing extensions and dialplans as soon as Weekly UPDATE-itis comes to a close.

BroadVoice in the News. Well, it looks like our old pals at BroadVoice are back in the news ... at least with the Better Business Bureau. See this link where BroadVoice finishes in second place for most complaints out of 77 telephone providers over a three year period. And, better yet, BroadVoice would retain or perhaps better their ranking if the numbers were based upon just the past 12 months. In fact, all but one of the complaints filed against BroadVoice were filed within the past 12 months. And, yes, there is a silver lining to this story, too. BroadVoice for the most part gives BBB complaints the same great treatment that we've received when inquiring about a problem ... nada!


12 Comments

  1. I’m getting a load of errors (see below) running the backup script for the first time. I have to admit running this on Fedora rather than Centos, so maybe this is the problem.

    Any thoughts? Feel free to ignore, because I know you must get a million support questions!!

    Thanks once more for your amazing articles. I have learnt so much from them. Please please please keep them up!! :o)

    rsync: link_stat "/root/.delete" failed: No such file or directory (2)
    rsync: link_stat "/root/.exclude" failed: No such file or directory (2)
    rsync: link_stat "/root/.exclude" failed: No such file or directory (2)
    rsync: link_stat "/root/.exclude" failed: No such file or directory (2)
    rsync: link_stat "/root/[cC]ache" failed: No such file or directory (2)
    rsync: send_files failed to open "/proc/acpi/event": Device or resource busy (16)
    rsync: send_files failed to open "/proc/kcore": Operation not permitted (1)
    rsync: send_files failed to open "/proc/acpi/event": Device or resource busy (16)
    rsync: send_files failed to open "/proc/kcore": Operation not permitted (1)
    rsync: send_files failed to open "/sys/bus/pci/drivers/AEC62xx_IDE/new_id": Permission denied (13)
    rsync: send_files failed to open "/sys/bus/pci/drivers/ALI15x3_IDE/new_id": Permission denied (13)
    rsync: send_files failed to open "/sys/bus/pci/drivers/AMD_IDE/new_id": Permission denied (13)
    rsync: send_files failed to open "/sys/bus/pci/drivers/ATIIXP_IDE/new_id": Permission denied (13)
    Thanks – Jon

    [WM: Fedora probably is the problem. You can either ignore the errors understanding that these files aren’t getting backed up. Or you can work through the individual problems and try to eliminate them. The permissions problems probably can be fixed by making sure that root has rights to the affected files. I think I would exclude /proc/ just as we did with the other exclusions. The rest of the errors are harmless since the files don’t exist. Good luck!]

  2. Update on the Fedora problem. Where your script had single ‘-‘ I had to change it to ‘–‘ for example –exclude . I then excluded one more directory, and all worked perfectly.

    thanks again – Jon

  3. When I tried to enter the code above to run the rsync and copy it to the back-up server it failed with an error message:

    rsync: -delete: unknown option

    I’m not sure what I’m doing wrong, because I can’t tell exactly what I should be entering. Can you put a screen shot up perhaps?

    [WM: If you’re using a version of Linux other than CentOS, then you may need double hyphens. See comment above.]

  4. In comment 2 above, the writer did the exact same things both times. Perhaps the blog software mangled it too, but he wrote ‘-‘ instead of ‘-‘ where in both places he shows a single hyphen just as in your example. I feel as if I’m missing something really fundamental…or else you’re all pulling a really elaborate joke on me. I’m using Firefox 1.5 for Windows in case that has anything to do with it. I’ve tried looking at this page with different themes, different browsers, and on different OSs (Mac, Linux, and Windows XP) using IE, Firefox, and Opera. In all scenario’s it seems to have the same errors. What kind of adjustment needs to be made to the quotes? What are regular quotation marks? I know of only two types of quotation marks, single and double. The example above uses double, should I assume that we should use single quotations?

    [WM: Believe it or not, there are other quotation marks that don’t appear on the keyboard that face right and left and are typically used in newspapers and magazines. "Regular quotation mark" means the quotation mark character that usually is two keys to the right of the L key on PC keyboards and is created by holding down the shift key while pressing the single quote character. That’s the best I can do by way of explanation.]

  5. This this what you mean?

    rsync -avvz ?¢ÀÜ?°?Ǭ¢?¢‚Ǩ?°?ɂİ?Ǭ®?¢‚Ǩ?°?É‚Äû?ɬ?-delete?¢ÀÜ?°?Ǭ¢?¢‚Ǩ?°?ɂİ?Ǭ®?¢‚Ǩ?°?É‚Äû?ɬ?exclude ?¢ÀÜ?°?Ǭ¢?¢‚Ǩ?°?ɂİ?Ǭ®?¢‚Ä?ÀÜ?ɬ¨/proc/?¢ÀÜ?°?Ǭ¢?¢‚Ǩ?°?ɂİ?Ǭ®? ?¢ÀÜ?°?Ǭ¢?¢‚Ǩ?°?ɂİ?Ǭ®?¢‚Ǩ?°?É‚Äû?ɬ?exclude ?¢ÀÜ?°?Ǭ¢?¢‚Ǩ?°?ɂİ?Ǭ®?¢‚Ä?ÀÜ?ɬ¨/mnt/?¢ÀÜ?°?Ǭ¢?¢‚Ǩ?°?ɂİ?Ǭ®? ?¢ÀÜ?°?Ǭ¢?¢‚Ǩ?°?ɂİ?Ǭ®?¢‚Ǩ?°?É‚Äû?ɬ?exclude ?¢ÀÜ?°?Ǭ¢?¢‚Ǩ?°?ɂİ?Ǭ®?¢‚Ä?ÀÜ?ɬ¨[cC]ache?¢ÀÜ?°?Ǭ¢?¢‚Ǩ?°?ɂİ?Ǭ®? -e ssh / root@192.168.0.104:/backup/ > /var/log/backup.log

    [WM: I don’t know the syntax for the rsync command in any particular flavor of Linux other than CentOS/4 which is what the article was about. Once you undertake to change the recipe from what we’ve recommended, you are really, really on your own. There are a number of ways to figure out what the rsync syntax is for any OS. There’s rsync –help, there’s man rsync, there’s rsync -h, and there’s Google. Sorry I can’t be more helpful.]

  6. After getting this to work, say the main AAH 1.5 hard drive fails. Can I install AAH 2.2 on a new hard drive and use the restore to recover all the settings?

    [WM: Nice try, Mark, but no. You have to reinstall the original AAH 1.5 on a new hard drive and then restore your 1.5 backup.]

  7. It looks like you’ve fixed the problem with the WordPress blog software. Thank you!!!

    [WM: You’re not half as glad as I am.]

  8. It would seem you see little value in a RAID1, But my Asterisk runs my home & office and does not get touched for weeks/months on end, I have given up on CentOS supporting any RAID card out-of-the-box and need a soft mirror. I just want uptime, and generally I am not stupid enough to screw up my asterisk during a config change. You talk about a bullet-proof config, but poo-poo a mirror in such a designation? are you kidding? It’s the drives that fail most often on these "untouched" servers. Being a lost linux newbie (MCSE/CCNA, so I guess you would say I have gone to the dark side) I have been unable to figure this out myself. Please lay out some procedure for the linux handicapped. I am hoping to add a mirror to 3 existing servers (2 are IDE, 1 SCSI) 2 of these servers are remote, so a minor hard drive crash means no phones for 3 hours. Help.
    TimC

    [WM: I’m not opposed to RAID. You just have to understand what it’s good for and what it’s not. It isn’t a good backup. It does provide some fault tolerance. If you want to install RAID with Linux, you’ll need to build up a CentOS/4 box using the standard CentOS install, and then install Asterisk@Home from the tarball and then run the install script after reading the installation notes. There are some other gotcha’s so be sure to check SourceForge before you begin. We’re planning to do this for a customer soon, and I’ll write it up afterwards.]

  9. Regarding RAID1, I have been struggling with it for the past week now and after many installs, I can now offer the following solution:
    Install Summary:
    1. Install a@h v2.6 using the command ?¢‚Ǩ?ìlinux askmethod?¢‚Ǩ¬ù with raid1 & minimal package install; reboot;
    2. Follow the release notes:
    mkdir /var/aah_load (must use this directory)
    cp asteriskathome-1.5.tar.gz /var/aah_load
    cd /var/aah_load
    tar xvfz asteriskathome-1.5.tar.gz
    ./install.sh
    reboot
    3. Must now Asterisk from source:
    cd /usr/src/
    ./rebuildastsrc.sh
    reboot
    rebuild_zaptel
    reboot
    genzaptelconf ?¢‚Ǩ‚Äúv (-v is optional)
    reboot
    Note: At this point, if you have a Digium TDM11B, you should be able to dial out from ZAP/1 using ZAP/g0.
    4. install-pdf, netconfig, etc.

  10. Hello,

    How could I adjust this so that the second server is a hot backup trixbox server or a mirrored image of the first server? The new Aastra firmware allows for a second reg server and this would give me tha ability to have a little redundancy.

    Thanks for the great articles!

Comments are closed.