Incredible PBX Now Does Incredible Backups… and Restores

Along with many of you, we have wrestled with getting reliable backups of our Asterisk®-based PBXs since the Asterisk@Home days. Flawless backups, of course, are worthless unless there's an accompanying flawless restore to get you back in business. Therein lies the rub. The number of minefields we've discovered along Restoration Way is legendary. A quick list includes incompatible hardware, changing device drivers, incompatible file storage systems, and on and on.

Update: Just released Incredible Backup 2 for PIAF2 systems.

What's really disturbing about all of this is that lack of adequate backups is the single component, in our opinion, that has kept open source PBXs from being a true match for commercial systems. People can't live without their phone systems... even if they're old and out of date. So, regardless of age, there has to be a way to bring your system back from the dead, or it's of little use in a production environment.

When we set out to create The Incredible PBX, one of our primary design goals was to come up with a system architecture that would let you use this new system for a decade. Yes, a decade! Not six months, not next year, but ten years from now your Incredible PBX would still be humming along. One way was to totally insulate the system from the Internet. Another key ingredient was rock-solid dependability. Remember that black phone in your grandma's house. It wasn't designed for replacement every six months. Nor was its underlying phone system. As the old adage goes: "If it ain't broke, don't fix it!"

In order to reach these design goals, we not only needed a backup system but also a way to separate your critical data from the underlying hardware. Why? Because the hardware continues to change every six months. What this backup solution is not is a full disk backup. Every full system backup solution we've tried simply isn't reliable unless the hardware on the new system is virtually identical to the hardware on the old one, a most unlikely scenario two or more years down the road.

How It Works. The Incredible Backup and Restore works like this. You built a working Incredible PBX from a base PBX in a Flash install so we start there. To restore a system, you'll first reinstall PBX in a Flash on your new server. The actual version doesn't really matter so long as it works. And newer versions with the latest CentOS releases support newer hardware. This avoids most of the hardware pitfalls that usually accompany a failed restore process.

The next slippery slope was incompatible versions of FreePBX between your original system and your current server. We can always update Asterisk from source after the restore, but FreePBX was problematic because the structure of the MySQL database tables associated with different versions of FreePBX changes frequently. And your backup MySQL data might very well be in MySQL tables that don't match your original PBX in a Flash build. So Incredible Restore provides the option of first restoring the version of FreePBX that existed at the time you made your last backup.

Then there's the problem of incompatible network and email implementations. Incredible Restore provides options to let you choose whether to restore your old network and email settings. If your newly built PBX in a Flash server has functioning network and email connectivity, don't restore the old settings. Simple as that.

What we really care about is getting your data back including a functioning PBX. There's got to be a catch, right? For a pure VoIP PBX, everything should be fine. The gotcha is that there are hundreds of add-on cards to support all sorts of proprietary hardware as well as to access Ma Bell's PSTN network. You're on your own there. Just be sure you have copies of the software pieces needed to make your special hardware function again once we've completed the restore to your new server. The same goes for custom software such as Cepstral TTS and Amazon S3. The components necessary to reinstall these add-ons should still be in your /root directory after the restore so it's not really a big deal to put Humpty back together again. Our tutorial links are just above.

Before we get to the installation, we want to put in a plug for PogoPlug. Not only is this the best thing since sliced bread, but it doesn't cost much more. You add this $99 (if you hurry) device to your LAN at home, at your office, or at a friend's house. Then connect one to four USB hard drives, and you have your own Cloud Computing Solution that also happens to be absolutely perfect for Incredible Backups and Restores. In fact, the setup software can be installed as part of the restore process. And the software already is included with every Incredible PBX. Just insert your login credentials, and the PogoPlug disk drives (regardless of location) are transparently added in the /mnt/pogoplug directory tree.

It's GPL2! Last but not least, we've released both Incredible Backup and Incredible Restore as GPL2 open source modules. That means you not only can learn some bash scripting in your spare time but you also can embellish the scripts in any way you like to support your favorite add-ons. All we'd ask is that you upload a copy with your enhancements so that we can share your good deeds with the rest of the Asterisk community and incorporate your good ideas into the next release. Keep an eye on the comments to this article and the PIAF Forum for the most recent additions. Better yet, subscribe to the RSS Feed for Comments at the top of this page, and they'll be delivered to your door as they occur.

Overview. Here's the quick step-by-step to get things working:

1. Download the software onto Incredible PBX
2. Install your PogoPlug (optional)
3. Create a directory for backups
4. Enter directory location in IncredibleBackup script
5. Run IncredibleBackup to make backup
6. Purchase Machine #2 OR create new Proxmox KVM
7. Install latest PBX in a Flash
8. Run update-scripts and update-fixes
9. Download the software onto Machine #2
10. Create a directory to house backups AND
11. Copy backup tarballs to directory OR
12. Use PogoPlug and skip #10 and #11
13. Enter directory location in IncredibleRestore script
14. Run IncredibleRestore to restore backup

Using Incredible Backup. Installation couldn't be easier. On your Incredible PBX server, log in as root and issue the following commands:

cd /root
wget http://incrediblepbx.com/incredible.tar.gz
tar zxvf incredible.tar.gz

Once you decompress the tarball, you'll be left with two files: incrediblebackup and incrediblerestore. With both scripts, you'll need to edit them and insert the location of your backup directory. Before doing that, you need a dedicated backup directory which is not in the /root or /var/www directory trees. We don't need to tell you what a dumb idea it is to store your backups on the same machine you're backing up... so we won't. As noted, our recommendation is to use a PogoPlug and preferably at a location different from the site of your server. Whatever directory you choose, it needs to be accessible from your server. SAMBA also is available on PBX in a Flash systems to access other drives in your LAN, but it needs to be activated. Incredible PBX systems are totally insulated from the Internet by a hardware-based firewall so you're safe using SAMBA provided you trust other users on your LAN. Once the directory exists, edit the scripts and insert the location in backuploc: nano -w incrediblebackup. Save your change: Ctrl-X, Y, then Enter. Repeat process for incrediblerestore. To create an Incredible Backup, execute this command: /root/incrediblebackup. All of the backups are stored in compressed tarballs with a current time stamp, e.g. 1273067177.tgz. You can decipher the actual time of the backup with a command like this: date -d "@1273067177" --> Wed May 5 09:46:17 EDT 2010

REMINDERS: If you're using a PogoPlug, don't forget to run pogo-start.sh before running incrediblebackup.

If you wish to run incrediblebackup as a cron job, remember to comment out the following line in the script with a leading #:

read -p "To proceed at your own risk and agree to license, press Enter. Otherwise Ctrl-C."

Don't forget to also activate your PogoPlug as a cron job before the time that incrediblebackup is scheduled to run!

What To Back Up? As we mentioned previously, backups are the easy part. It's the restore process that causes premature aging. The best time to plan your restore strategy is before you need it! Always assume the worst case, i.e. that nothing is recoverable from your primary server. Then ask yourself whether the backup is capturing and saving in a safe location everything you'll need to put Humpty back together again. Currently, Incredible Backup captures the following files and directory trees:

/var/www/html /var/lib/asterisk /var/lib/mysql /root /etc/asterisk /tftpboot
/etc/pbx /etc/wanpipe /etc/sudoers /etc/odbc.ini /etc/odbcinst.ini
/var/lib/asterisk/sounds/tts /var/lib/asterisk/sounds/custom
/var/spool/asterisk /etc/amportal.conf /etc/wanpipe
/etc/hosts /etc/resolv.conf /etc/sysconfig/network-scripts/ifcfg* /etc/sysconfig/iptables /etc/sysconfig/network /etc/mail
/usr/local/bin /usr/local/sbin /usr/src and portions of /usr/sbin

Keep in mind that an Incredible Restore always begins with a functioning PBX in a Flash server. And you will have the option of restoring all Incredible PBX applications. With the exception of these applications, ask yourself whether the backup list above captures everything you've added to your server and is sufficient to meet your needs. With most Incredible PBX implementations, it should adequately restore an existing Incredible PBX together with your FreePBX customizations. But the beauty of open source software is that you can and should customize it to meet your specific needs. You can add any additional directories... so long as you do it and save the backup to some off-site location before your server dies. :wink:

The other important question to ask yourself is what is your Incredible PBX as presently configured worth to you. If the answer is more than $200, perhaps the time is ripe to purchase a second system for emergencies and test your restore strategy in advance.

Using Incredible Restore. Let's get the cautionary notes out of the way up front. First, by using this software, you have agreed to assume all risks including the risk of losing all your data. Second, don't experiment with restores to your primary system. Third, in the most emphatic way we can, we encourage you to test a restore before D-Day arrives... but not on your live system! If it means borrowing a friend's old clunker for the afternoon, then by all means do so. If you can afford a second system, that's even better. If you have a virtual platform at the office, borrow a little space for the weekend and try a restore. Proxmox works and so does VMware and most other virtual platforms. We don't mean to be all doom and gloom about this, but unfortunately backups are all about doom and gloom. Now's the time to find out something didn't work quite right, not when you really, really need it.

The first step in using Incredible Restore is to install PBX in a Flash on the new server. We recommend you also run update-scripts and update-fixes once the PIAF install is complete. As with Incredible Backup, the next step in using Incredible Restore is to log into your new server and download the application:

cd /root
wget http://incrediblepbx.com/incredible.tar.gz
tar zxvf incredible.tar.gz

Unless you're using a backup tarball from external location supported by SAMBA or PogoPlug, Step #3 is to create a directory on your new server and copy the backup tarball to that directory. Step #4 is to configure the incrediblerestore script with the directory location of the backup tarball to be restored. Once you've saved the location, run the script: /root/incrediblerestore. You'll be given the following options to tailor how the restoration process should proceed:

1. Whether to enable PogoPlug functionality on the server
2. Whether to restore FreePBX application from the backup
3. Whether to restore Incredible PBX apps to new server
4. Whether to restore Network Settings from the backup
5. Whether to restore SendMail Setup from the backup
6. Whether to restore Asterisk binaries and source code
7. Whether to disable outbound SIP/IAX connectivity

1. Enabling PogoPlug. If you're using a PogoPlug for your backups, you'll be prompted whether to install the PogoPlug software as first option when you run the IncredibleRestore script. Choosing Y will load the necessary software. Then it's a simple matter of entering your login credentials in pogo-start.sh and running pogo-start.sh to activate the PogoPlug. Then just rerun the IncredibleRestore script to continue.

2. Restoring FreePBX Application. Unless you are absolutely certain that the version of FreePBX in your backup matches the version on your new server, choosing Y for this option is highly recommended. Otherwise, the structure of the FreePBX MySQL tables may differ and cause all sorts of difficult to diagnose problems.

3. Restoring Incredible PBX Applications. If your backup was made on an Incredible PBX server, then the Incredible PBX apps should be restored to your new server. We've made this optional only to accommodate those who may wish to tailor the scripts to support other Asterisk distributions.

4. Restoring Network Configuration. If you're recovering from a catastrophic failure and want to make certain that a static IP address is preserved when you restore your backup, then you obviously would want to restore your network configuration. If you're building a duplicate system to be kept off line or if you're moving your server to a virtual machine platform, then you probably do NOT want to restore the network configuration from your primary machine. A good rule of thumb probably goes like this. If network connectivity already is working on your new server, don't restore the network setup from your backup.

5. Restoring SendMail Setup. The only situation in which you would want to restore the SendMail setup from your primary server is if you have specially tailored SendMail on the primary server in order to send email. This typically would happen where an Internet service provider blocks outbound SMTP traffic, e.g. Comcast residential Internet service.

6. Restoring Asterisk Binaries and Source. This functionality is EXPERIMENTAL AND BARELY TESTED!! It only works (at all) with Asterisk implementations still using Zaptel, not DAHDI. Unless your primary server was running a version of Asterisk that differs from the default PBX in a Flash build, the correct answer to this prompt is N. Never use this option if you are restoring from a catastrophic failure. Instead, run update-source and update-fixes on the newly restored server. It's safer! We'll keep you posted on future developments.

7. Disabling Outbound SIP/IAX Connectivity. This option allows you to disable outbound SIP and IAX traffic on the new server. Typically, you would use this if the server on which the backup was made is still on line. The reason is to avoid having two identical servers compete for connections to SIP and IAX providers. If this option is chosen and you subsequently take your primary server off line, then you will need to enable SIP and IAX connectivity on the newly restored server before it can take over primary duties. To do this, log into your new server as root and issue the following commands:

cd /etc/sysconfig
cp iptables.sip iptables
service iptables restart

To again disable SIP and IAX outbound traffic, issue the following commands:

cd /etc/sysconfig
cp iptables.nosip iptables
service iptables restart

Feedback and Suggestions Encouraged. Incredible Backup and Incredible Restore are still very much works in progress. A number of folks on the PBX in a Flash Forums have assisted us in getting version 1.0 out the door today, but don't bet the farm on this software until you have carefully tested it using a redundant server! We will continue to improve/enhance the functionality for weeks and perhaps months to come. And, until the kinks are all worked out, we would strongly encourage you to download the latest and greatest version each time you make a backup or undertake to restore a backup to a new system. During this development period, we also would encourage you to make suggestions and to offer enhancements. After all, that's what open source is all about. Enjoy!




Need help with Asterisk? Visit the PBX in a Flash Forum.
Or Try the New, Free PBX in a Flash Conference Bridge.


whos.amung.us If you're wondering what your fellow man is reading on Nerd Vittles these days, wonder no more. Visit our new whos.amung.us statistical web site and check out what's happening. It's a terrific resource both for us and for you.


 
New Vitelity Special. Vitelity has generously offered a new discount for PBX in a Flash users. You now can get an almost half-price DID and 60 free minutes from our special Vitelity sign-up link. If you're seeking the best flexibility in choosing an area code and phone number plus the lowest entry level pricing plus high quality calls, then Vitelity is the hands-down winner. Vitelity provides Tier A DID inbound service in over 3,000 rate centers throughout the US and Canada. And, when you use our special link to sign up, the Nerd Vittles and PBX in a Flash projects get a few shekels down the road while you get an incredible signup deal as well. The going rate for Vitelity's DID service is $7.95 a month which includes up to 4,000 incoming minutes on two simultaneous channels with terminations priced at 1.45¢ per minute. Not any more! For PBX in a Flash users, here's a deal you can't (and shouldn't) refuse! Sign up now, and you can purchase a Tier A DID with unlimited incoming calls for just $3.99 a month and you get a free hour of outbound calling to test out their call quality. To check availability of local numbers and tiers of service from Vitelity, click here. Do not use this link to order your DIDs, or you won't get the special pricing! After the free hour of outbound calling, Vitelity's rate is just 1.44¢ per minute for outbound calls in the U.S. There is a $35 prepay when you sign up. This covers future usage and any balance is fully refundable if you decide to discontinue service with Vitelity.
 


Some Recent Nerd Vittles Articles of Interest...

Be Sociable, Share!

11 Responses to “Incredible PBX Now Does Incredible Backups… and Restores”

  1. ward says:

    As luck would have it, PogoPlug devices have just gone on sale at buy.com for $99 with free shipping.

  2. ward says:

    For those that installed The Incredible PBX prior to this message, there were two missing directories which caused creation of some future reminders to fail. To fix it, log into your server as root and issue the following commands:

    cd /var/spool/asterisk
    mkdir reminders
    mkdir recurring
    chown asterisk:asterisk recurring
    chown asterisk:asterisk reminders

    Our apologies.

  3. Joshua Welch says:

    I get:
    tar: Removing leading `/’ from member names
    tar: /usr/sbin/fxotune: Cannot stat: No such file or directory
    tar: /usr/sbin/smsg: Cannot stat: No such file or directory
    tar: Error exit delayed from previous errors
    tar: Removing leading `/’ from member names
    tar: /usr/sbin/astribank*: Cannot stat: No such file or directory
    tar: /usr/sbin/dahdi*: Cannot stat: No such file or directory
    tar: /usr/sbin/lsdahdi: Cannot stat: No such file or directory
    tar: /usr/sbin/sethdlc: Cannot stat: No such file or directory
    tar: /usr/sbin/twinstar: Cannot stat: No such file or directory
    tar: Error exit delayed from previous errors
    tar: Removing leading `/’ from member names

    Is that okay?

    [WM: It's fine. Just means you're using Zaptel instead of DAHDI. But test it!]

  4. ward says:

    NEWS FLASH: Some refurbished Acer Aspire Revo machines are available from PC Connection for $159 with free shipping if you hurry.

  5. Alek says:

    Hi, I have a question, you had mentioned running PBX on a “$200 Acer Aspire Revo” before, today I saw SheevaPlug which is being sold for $99 with these specs:
    1.2 GHz operation
    512MB RAM
    Is it good enough to run a PBX on it?

    [WM: Won't run a full-featured Asterisk PBX with all FreePBX modules like ours. Sorry.]

  6. Alek says:

    Oh, thank you, is this because of the CPU?

    I think I should find one of those Aspire Revo, the site you posted on the previous comment doesn’t ship to Canada.

    [WM: Google is showing 45 sellers at the moment.]

  7. Paul says:

    Does this only work with the Incredible PBX distro?

    [WM: That's all we support. The basic design should work for any of the Asterisk/FreePBX aggregations provided you take into account any additional applications or "special features" that accompany your particular distro. It's open source so you can look through the code and determine for yourself whether it meets your needs.]

  8. Actually says:

    Asterisk/freepbx can run on the shivaplug

    http://www.plugpbx.org/

    [WM: Reread our comment. Never suggested otherwise. But it won't provide anything near the performance or functionality of the Incredible PBX.]

  9. Actually says:

    Care to give a list of specific modules known not to work on the shivaplug?

    [WM: We started looking for alternatives after a very brief look at the system constraints imposed by the ShivaPlug. It wasn't necessarily any particular module not working but the overall processor, storage, and memory constraints of the ShivaPlug that caused us to have concerns. By all means, prove us wrong. Load up all the FreePBX modules on a ShivaPlug and let us know how it goes when you're handling more than one or two simultaneous calls plus an IVR and voicemail for several extensions. :roll: ]

  10. ward says:

    /etc/wanpipe has been added to the backup list to support certain add-on cards.

  11. ward says:

    A major SIP security vulnerability was discovered in all versions of Asterisk today. You can read all about it here.

    We have developed a script for Asterisk 1.8.x which will quickly patch your system and eliminate the problem. Log into your server as root and issue the following commands:

    cd /root
    wget http://incrediblepbx.com/sipfix
    chmod +x sipfix
    ./sipfix

    Please apply this patch immediately to protect your server!

Leave a Reply

Ringbinder theme by Themocracy