Incredible PBX Now Does Incredible Backups… and Restores

Along with many of you, we have wrestled with getting reliable backups of our Asterisk®-based PBXs since the Asterisk@Home days. Flawless backups, of course, are worthless unless there's an accompanying flawless restore to get you back in business. Therein lies the rub. The number of minefields we've discovered along Restoration Way is legendary. A quick list includes incompatible hardware, changing device drivers, incompatible file storage systems, and on and on.

Update: Just released Incredible Backup 2 for PIAF2 systems.

What's really disturbing about all of this is that lack of adequate backups is the single component, in our opinion, that has kept open source PBXs from being a true match for commercial systems. People can't live without their phone systems... even if they're old and out of date. So, regardless of age, there has to be a way to bring your system back from the dead, or it's of little use in a production environment.

When we set out to create The Incredible PBX, one of our primary design goals was to come up with a system architecture that would let you use this new system for a decade. Yes, a decade! Not six months, not next year, but ten years from now your Incredible PBX would still be humming along. One way was to totally insulate the system from the Internet. Another key ingredient was rock-solid dependability. Remember that black phone in your grandma's house. It wasn't designed for replacement every six months. Nor was its underlying phone system. As the old adage goes: "If it ain't broke, don't fix it!"

In order to reach these design goals, we not only needed a backup system but also a way to separate your critical data from the underlying hardware. Why? Because the hardware continues to change every six months. What this backup solution is not is a full disk backup. Every full system backup solution we've tried simply isn't reliable unless the hardware on the new system is virtually identical to the hardware on the old one, a most unlikely scenario two or more years down the road.

How It Works. The Incredible Backup and Restore works like this. You built a working Incredible PBX from a base PBX in a Flash install so we start there. To restore a system, you'll first reinstall PBX in a Flash on your new server. The actual version doesn't really matter so long as it works. And newer versions with the latest CentOS releases support newer hardware. This avoids most of the hardware pitfalls that usually accompany a failed restore process.

The next slippery slope was incompatible versions of FreePBX between your original system and your current server. We can always update Asterisk from source after the restore, but FreePBX was problematic because the structure of the MySQL database tables associated with different versions of FreePBX changes frequently. And your backup MySQL data might very well be in MySQL tables that don't match your original PBX in a Flash build. So Incredible Restore provides the option of first restoring the version of FreePBX that existed at the time you made your last backup.

Then there's the problem of incompatible network and email implementations. Incredible Restore provides options to let you choose whether to restore your old network and email settings. If your newly built PBX in a Flash server has functioning network and email connectivity, don't restore the old settings. Simple as that.

What we really care about is getting your data back including a functioning PBX. There's got to be a catch, right? For a pure VoIP PBX, everything should be fine. The gotcha is that there are hundreds of add-on cards to support all sorts of proprietary hardware as well as to access Ma Bell's PSTN network. You're on your own there. Just be sure you have copies of the software pieces needed to make your special hardware function again once we've completed the restore to your new server. The same goes for custom software such as Cepstral TTS and Amazon S3. The components necessary to reinstall these add-ons should still be in your /root directory after the restore so it's not really a big deal to put Humpty back together again. Our tutorial links are just above.

Before we get to the installation, we want to put in a plug for PogoPlug. Not only is this the best thing since sliced bread, but it doesn't cost much more. You add this $99 (if you hurry) device to your LAN at home, at your office, or at a friend's house. Then connect one to four USB hard drives, and you have your own Cloud Computing Solution that also happens to be absolutely perfect for Incredible Backups and Restores. In fact, the setup software can be installed as part of the restore process. And the software already is included with every Incredible PBX. Just insert your login credentials, and the PogoPlug disk drives (regardless of location) are transparently added in the /mnt/pogoplug directory tree.

It's GPL2! Last but not least, we've released both Incredible Backup and Incredible Restore as GPL2 open source modules. That means you not only can learn some bash scripting in your spare time but you also can embellish the scripts in any way you like to support your favorite add-ons. All we'd ask is that you upload a copy with your enhancements so that we can share your good deeds with the rest of the Asterisk community and incorporate your good ideas into the next release. Keep an eye on the comments to this article and the PIAF Forum for the most recent additions. Better yet, subscribe to the RSS Feed for Comments at the top of this page, and they'll be delivered to your door as they occur.

Overview. Here's the quick step-by-step to get things working:

1. Download the software onto Incredible PBX
2. Install your PogoPlug (optional)
3. Create a directory for backups
4. Enter directory location in IncredibleBackup script
5. Run IncredibleBackup to make backup
6. Purchase Machine #2 OR create new Proxmox KVM
7. Install latest PBX in a Flash
8. Run update-scripts and update-fixes
9. Download the software onto Machine #2
10. Create a directory to house backups AND
11. Copy backup tarballs to directory OR
12. Use PogoPlug and skip #10 and #11
13. Enter directory location in IncredibleRestore script
14. Run IncredibleRestore to restore backup

Using Incredible Backup. Installation couldn't be easier. On your Incredible PBX server, log in as root and issue the following commands:

cd /root
wget http://incrediblepbx.com/incredible.tar.gz
tar zxvf incredible.tar.gz

Once you decompress the tarball, you'll be left with two files: incrediblebackup and incrediblerestore. With both scripts, you'll need to edit them and insert the location of your backup directory. Before doing that, you need a dedicated backup directory which is not in the /root or /var/www directory trees. We don't need to tell you what a dumb idea it is to store your backups on the same machine you're backing up... so we won't. As noted, our recommendation is to use a PogoPlug and preferably at a location different from the site of your server. Whatever directory you choose, it needs to be accessible from your server. SAMBA also is available on PBX in a Flash systems to access other drives in your LAN, but it needs to be activated. Incredible PBX systems are totally insulated from the Internet by a hardware-based firewall so you're safe using SAMBA provided you trust other users on your LAN. Once the directory exists, edit the scripts and insert the location in backuploc: nano -w incrediblebackup. Save your change: Ctrl-X, Y, then Enter. Repeat process for incrediblerestore. To create an Incredible Backup, execute this command: /root/incrediblebackup. All of the backups are stored in compressed tarballs with a current time stamp, e.g. 1273067177.tgz. You can decipher the actual time of the backup with a command like this: date -d "@1273067177" --> Wed May 5 09:46:17 EDT 2010

REMINDERS: If you're using a PogoPlug, don't forget to run pogo-start.sh before running incrediblebackup.

If you wish to run incrediblebackup as a cron job, remember to comment out the following line in the script with a leading #:

read -p "To proceed at your own risk and agree to license, press Enter. Otherwise Ctrl-C."

Don't forget to also activate your PogoPlug as a cron job before the time that incrediblebackup is scheduled to run!

What To Back Up? As we mentioned previously, backups are the easy part. It's the restore process that causes premature aging. The best time to plan your restore strategy is before you need it! Always assume the worst case, i.e. that nothing is recoverable from your primary server. Then ask yourself whether the backup is capturing and saving in a safe location everything you'll need to put Humpty back together again. Currently, Incredible Backup captures the following files and directory trees:

Keep in mind that an Incredible Restore always begins with a functioning PBX in a Flash server. And you will have the option of restoring all Incredible PBX applications. With the exception of these applications, ask yourself whether the backup list above captures everything you've added to your server and is sufficient to meet your needs. With most Incredible PBX implementations, it should adequately restore an existing Incredible PBX together with your FreePBX customizations. But the beauty of open source software is that you can and should customize it to meet your specific needs. You can add any additional directories... so long as you do it and save the backup to some off-site location before your server dies. 😉

The other important question to ask yourself is what is your Incredible PBX as presently configured worth to you. If the answer is more than $200, perhaps the time is ripe to purchase a second system for emergencies and test your restore strategy in advance.

Using Incredible Restore. Let's get the cautionary notes out of the way up front. First, by using this software, you have agreed to assume all risks including the risk of losing all your data. Second, don't experiment with restores to your primary system. Third, in the most emphatic way we can, we encourage you to test a restore before D-Day arrives... but not on your live system! If it means borrowing a friend's old clunker for the afternoon, then by all means do so. If you can afford a second system, that's even better. If you have a virtual platform at the office, borrow a little space for the weekend and try a restore. Proxmox works and so does VMware and most other virtual platforms. We don't mean to be all doom and gloom about this, but unfortunately backups are all about doom and gloom. Now's the time to find out something didn't work quite right, not when you really, really need it.

The first step in using Incredible Restore is to install PBX in a Flash on the new server. We recommend you also run update-scripts and update-fixes once the PIAF install is complete. As with Incredible Backup, the next step in using Incredible Restore is to log into your new server and download the application:

cd /root
wget http://incrediblepbx.com/incredible.tar.gz
tar zxvf incredible.tar.gz

Unless you're using a backup tarball from external location supported by SAMBA or PogoPlug, Step #3 is to create a directory on your new server and copy the backup tarball to that directory. Step #4 is to configure the incrediblerestore script with the directory location of the backup tarball to be restored. Once you've saved the location, run the script: /root/incrediblerestore. You'll be given the following options to tailor how the restoration process should proceed:

1. Whether to enable PogoPlug functionality on the server
2. Whether to restore FreePBX application from the backup
3. Whether to restore Incredible PBX apps to new server
4. Whether to restore Network Settings from the backup
5. Whether to restore SendMail Setup from the backup
6. Whether to restore Asterisk binaries and source code
7. Whether to disable outbound SIP/IAX connectivity

1. Enabling PogoPlug. If you're using a PogoPlug for your backups, you'll be prompted whether to install the PogoPlug software as first option when you run the IncredibleRestore script. Choosing Y will load the necessary software. Then it's a simple matter of entering your login credentials in pogo-start.sh and running pogo-start.sh to activate the PogoPlug. Then just rerun the IncredibleRestore script to continue.

2. Restoring FreePBX Application. Unless you are absolutely certain that the version of FreePBX in your backup matches the version on your new server, choosing Y for this option is highly recommended. Otherwise, the structure of the FreePBX MySQL tables may differ and cause all sorts of difficult to diagnose problems.

3. Restoring Incredible PBX Applications. If your backup was made on an Incredible PBX server, then the Incredible PBX apps should be restored to your new server. We've made this optional only to accommodate those who may wish to tailor the scripts to support other Asterisk distributions.

4. Restoring Network Configuration. If you're recovering from a catastrophic failure and want to make certain that a static IP address is preserved when you restore your backup, then you obviously would want to restore your network configuration. If you're building a duplicate system to be kept off line or if you're moving your server to a virtual machine platform, then you probably do NOT want to restore the network configuration from your primary machine. A good rule of thumb probably goes like this. If network connectivity already is working on your new server, don't restore the network setup from your backup.

5. Restoring SendMail Setup. The only situation in which you would want to restore the SendMail setup from your primary server is if you have specially tailored SendMail on the primary server in order to send email. This typically would happen where an Internet service provider blocks outbound SMTP traffic, e.g. Comcast residential Internet service.

6. Restoring Asterisk Binaries and Source. This functionality is EXPERIMENTAL AND BARELY TESTED!! It only works (at all) with Asterisk implementations still using Zaptel, not DAHDI. Unless your primary server was running a version of Asterisk that differs from the default PBX in a Flash build, the correct answer to this prompt is N. Never use this option if you are restoring from a catastrophic failure. Instead, run update-source and update-fixes on the newly restored server. It's safer! We'll keep you posted on future developments.

7. Disabling Outbound SIP/IAX Connectivity. This option allows you to disable outbound SIP and IAX traffic on the new server. Typically, you would use this if the server on which the backup was made is still on line. The reason is to avoid having two identical servers compete for connections to SIP and IAX providers. If this option is chosen and you subsequently take your primary server off line, then you will need to enable SIP and IAX connectivity on the newly restored server before it can take over primary duties. To do this, log into your new server as root and issue the following commands:

cd /etc/sysconfig
cp iptables.sip iptables
service iptables restart

To again disable SIP and IAX outbound traffic, issue the following commands:

cd /etc/sysconfig
cp iptables.nosip iptables
service iptables restart

Feedback and Suggestions Encouraged. Incredible Backup and Incredible Restore are still very much works in progress. A number of folks on the PBX in a Flash Forums have assisted us in getting version 1.0 out the door today, but don't bet the farm on this software until you have carefully tested it using a redundant server! We will continue to improve/enhance the functionality for weeks and perhaps months to come. And, until the kinks are all worked out, we would strongly encourage you to download the latest and greatest version each time you make a backup or undertake to restore a backup to a new system. During this development period, we also would encourage you to make suggestions and to offer enhancements. After all, that's what open source is all about. Enjoy!

Need help with Asterisk? Visit the PBX in a Flash Forum.
Or Try the New, Free PBX in a Flash Conference Bridge.

whos.amung.us If you're wondering what your fellow man is reading on Nerd Vittles these days, wonder no more. Visit our new whos.amung.us statistical web site and check out what's happening. It's a terrific resource both for us and for you.

Special Thanks to Our Generous Sponsors

FULL DISCLOSURE: ClearlyIP, Skyetel, Vitelity, DigitalOcean, Vultr, VoIP.ms, 3CX, Sangoma, TelecomsXchange and VitalPBX have provided financial support to Nerd Vittles and our open source projects through advertising, referral revenue, and/or merchandise. As an Amazon Associate and Best Buy Affiliate, we also earn from qualifying purchases. We’ve chosen these providers not the other way around. Our decisions are based upon their corporate reputation and the quality of their offerings and pricing. Our recommendations regarding technology are reached without regard to financial compensation except in situations in which comparable products at comparable pricing are available from multiple sources. In this limited case, we support our sponsors because our sponsors support us.

BOGO Bonaza: Enjoy state-of-the-art VoIP service with a $10 credit and half-price SIP service on up to $500 of Skyetel trunking with free number porting when you fund your Skyetel account. No limits on number of simultaneous calls. Quadruple data center redundancy. $25 monthly minimum spend required. Tutorial and sign up details are here.

The lynchpin of Incredible PBX 2020 and beyond is ClearlyIP components which bring management of FreePBX modules and SIP phone integration to a level never before available with any other Asterisk distribution. And now you can configure and reconfigure your new Incredible PBX phones from the convenience of the Incredible PBX GUI.

VitalPBX is perhaps the fastest-growing PBX offering based upon Asterisk with an installed presence in more than 100 countries worldwide. VitalPBX has generously provided a customized White Label version of Incredible PBX tailored for use with all Incredible PBX and VitalPBX custom applications. Follow this link for a free test drive!
 

Special Thanks to Vitelity. Vitelity is now Voyant Communications and has halted new registrations for the time being. Our special thanks to Vitelity for their unwavering financial support over many years and to the many Nerd Vittles readers who continue to enjoy the benefits of their service offerings. We will keep everyone posted on further developments.
 

Some Recent Nerd Vittles Articles of Interest...