JUST RELEASED: Visit the Incredible PBX WikiIt’s been a busy couple of days following the double whammy of two RAID drive failures on Wednesday morning. We wanted to provide an interim update on where things stand. We share a dedicated server with a couple of other folks. The server is managed and maintained by WestNIC out of New York but physically resides in a data center in Dallas. Even with state-of-the-art hardware, things go wrong. And a dual drive failure made things worse, much worse.
The server runs cPanel and includes weekly onsite backups to a separate server as well as monthly off-site backups to New York. At least that’s what we had been told. It turns out that cPanel apparently chokes on account backups larger than 10GB so our provider had set a size limit on the accounts that qualified for backups in order to assure that the backups were reliable. Not a bad idea… if only we had known. Then we would have made our own backups of some of the one-of-a-kind data such as the PIAF Forum. Woulda, coulda, shoulda unfortunately doesn’t help at this juncture.
Our sites include Nerd Vittles, PBX in a Flash (.org and .com), and Incredible PBX as well as some other smaller domains. Luckily, Nerd Vittles escaped the 10GB limitation although it turned out the weekly backup was damaged. But a restore from New York retrieved all of our content except for the last three articles. Thanks to Google, those three articles still were sitting in Google’s cache. So Nerd Vittles is almost back to normal except where there were links to content or images on pbxinaflash.com or pbxinaflash.org. By yesterday, we had Nerd Vittles fully operational once again. Incredible PBX was much the same story, and it’s once again among the living.
Both the .org and .com sites for PBX in a Flash didn’t fare as well. They both were larger than 10GB which meant there were no cPanel backups anywhere. Most of the static content on both sites is readily available elsewhere for restoration. Unfortunately, the .com site also hosts the PIAF Forum which now includes over 100,000 messages covering VoIP.everything for the past six or seven years. Luckily (we hope), the raw data in the form of static files still exists on the damaged drives. So today we will be moving the raw data for the .org site back into place on the new server. We’re keeping our fingers crossed that the test will go smoothly. That turned out to be 11GB of data.
The more problematic site is pbxinaflash.com. That involves 50GB of data. While most of it is static content that can easily be restored, the PIAF Forum is still a question mark because it includes very fluid MySQL tables running under XenForo. The MySQL databases are also static files under Linux; however, recovery of the data is unknown at this juncture because we don’t know the type of MySQL tables that were in use. If they’re MyISAM, you can basically shut off MySQL momentarily, copy the files over, and restart MySQL with no problems. If the tables use the newer innoDB format then things get more complicated. We think, although we haven’t been able to verify it yet, that we have a XenForo backup in place from several weeks ago. Loss of a few week’s postings would obviously be a godsend when one considers the other alternative. We’ll keep you posted on our progress.
Should you find any glaring issues with the Nerd Vittles site, please post a comment and let us know. We appreciate your understanding during this difficult time for all of us.
Originally published: Friday, May 17, 2013
Need help with Asterisk? Visit the new PBX in a Flash Dev Page & Community on Google+
[purehtml id=8]
Once you’ve joined, just click About and choose PIAF Forum Community to start participating.
whos.amung.us If you’re wondering what your fellow man is reading on Nerd Vittles these days, wonder no more. Visit our whos.amung.us statistical web site and check out what’s happening. It’s a terrific resource both for all of us.
Special Thanks to Our Generous Sponsors
FULL DISCLOSURE: ClearlyIP, Skyetel, Vitelity, DigitalOcean, Vultr, VoIP.ms, 3CX, Sangoma, TelecomsXchange and VitalPBX have provided financial support to Nerd Vittles and our open source projects through advertising, referral revenue, and/or merchandise. As an Amazon Associate and Best Buy Affiliate, we also earn from qualifying purchases. We’ve chosen these providers not the other way around. Our decisions are based upon their corporate reputation and the quality of their offerings and pricing. Our recommendations regarding technology are reached without regard to financial compensation except in situations in which comparable products at comparable pricing are available from multiple sources. In this limited case, we support our sponsors because our sponsors support us.
BOGO Bonaza: Enjoy state-of-the-art VoIP service with a $10 credit and half-price SIP service on up to $500 of Skyetel trunking with free number porting when you fund your Skyetel account. No limits on number of simultaneous calls. Quadruple data center redundancy. $25 monthly minimum spend required. Tutorial and sign up details are here.
The lynchpin of Incredible PBX 2020 and beyond is ClearlyIP components which bring management of FreePBX modules and SIP phone integration to a level never before available with any other Asterisk distribution. And now you can configure and reconfigure your new Incredible PBX phones from the convenience of the Incredible PBX GUI.
VitalPBX is perhaps the fastest-growing PBX offering based upon Asterisk with an installed presence in more than 100 countries worldwide. VitalPBX has generously provided a customized White Label version of Incredible PBX tailored for use with all Incredible PBX and VitalPBX custom applications. Follow this link for a free test drive!
Special Thanks to Vitelity. Vitelity is now Voyant Communications and has halted new registrations for the time being. Our special thanks to Vitelity for their unwavering financial support over many years and to the many Nerd Vittles readers who continue to enjoy the benefits of their service offerings. We will keep everyone posted on further developments.
Some Recent Nerd Vittles Articles of Interest…
Ward –
We’ve exchanged tweets and similar about this, but I’m more concerned about not being able to install PiaF during the outage. I have an interminable installation ongoing for a client with the bad luck also to have suffered a disk crash shortly before yours. The good news is we are still in test and acceptance, so the cord had not yet been cut to the analog PBX. I’m also still waiting for the production hardware to be delivered (ordered?). The design calls for a RAIDed system, mounted in the data room on a UPSed rack, with a hot spare, instead of a partially street-scored system mounted in an odd tower-with-a-handle case left over from my days as a VJ, tucked away in the back corner of whichever cubicle is empty this week, so presumably the ‘real’ PBX won’t drop and shatter at the first hiccup in power. However, as you know, even the most careful disaster avoidance plans can fail, and I can think of a number of scenarios where performing an installation from scratch might prove one’s best or only path to recovery. It’s been uncomfortable enough trying to explain why, after three days, I still haven’t been able to revive what is essentially an extraneous, unused system; I can’t imagine what it would be like if we were talking about a device carrying 100% of their telephony.
I know there have been cries of "get thee to the cloud!" — as if it were simply a forklift upgrade to cloud-ify something as complex as the PiaF discussion fora. On the other hand, though, it seems to me the cloud would make a perfect home for the gigabytes of code and executable that make up a PiaF payload. At the very least, there should be an inclusive offline loader available for each significant PiaF release.
For some reason, no one ever wants to hire me to work on something I know anything about, and this current project (which also had me looking into biometric access control methodologies and solar plant management systems) is no exception. Accordingly, for every hour that goes toward choosing and implementing a solution, I typically spend two questioning my sanity and second- [and third-] [and fourth-]guessing my choices. I looked into 32 different ITSPs before selecting one; I researched a ridiculous number of hosted and on-site PBX offerings (months later, Shoretel is *still* calling); and once I’d settled on the best approach to take, I evaluated every major, and quite a few of the minor, value-added packagings of Asterisk currently available. As a result, I can say with complete confidence the trifecta of Asterisk, FreePBX, and PiaF/Incredible PBX is far and away the best possible solution for this client. And I still do — this current installation glitch notwithstanding. Its just that, without the glitch, your story would be that much more compelling….
Thanks.
[WM: We’re testing the restored .org site as we speak. Once it’s working, installs will be fine. The ISO will switch to the .net site for files when it can’t find them on .com. You’re more than welcome to try a new install now. Feedback (good or bad) is most appreciated.]
Something Vann said struck a very clear chord with me. Instead of making the PIAF .ISOs little more than a build of CentOs with scripts to download and compile Asterisk, FreePBX, etc., it would be far more preferable to include everything in the initial "payload" so that we can get our PBX up and running even in the event that the mothership has been shot down. We’re going to have to download everything anyway, why not up front instead of later. It’s one of the reasons I prefer the VM builds, too.
I imagine it looses you the opportunity to monitor installations quite so closely, and to slip-stream bug fixes into already-released products, but consider the goodwill folks will feel at not being quite so blatantly tethered to your servers.
[WM: Blatant tethering is probably here to stay. The problem with the all-in-one approach is that you end up having to put out monthly ISOs, and we’re not going to get back in that business. Sorry.]
Alas, as of 12:15 PDT, still no PiaF-Green love. (I’m assuming the server outage is what’s responsible for that payload being reported as ‘unavailable,’ and not something stupid on my part.) But thanks for your prompt reply; now I know hopes imminent, I’ll keep checking!
[WM: May be that your DNS cache has not yet refreshed. Worked for me.]
I know this is small, but I tried to go to the special
Vitelity link to sign up and got a 404 error. Would
you happen to have that link handy?
When DSL Reports was down for several weeks last year, it was a brute reminder that things that we assume "are just there", come at a price. When it came back, I became a premium member the next day (after some 5 years of "freeloading"). I am looking forward to chipping in to PIAF as well (by subscribing to "paid updates" or in some more meaningful way).
I am sure I am not the only one feeling that way. When everything comes back (hopefully, with minimum damage) – maybe similar "premium" or "contributing" status on the board could be instantiated. As opposed to "guru" that is earned 🙂
[WM: Sad to say, you get what you pay for. We’re the only non-commercial group still in existence in the VoIP space. And the "contributions" can be counted on your two hands each year.]
if you want want a distant off site backup
I could likely spare a 60 gigs
I am in canada not to far to ship by disc if need be
nor am I so close as to suffer the same failings a local location could
Is there anything you are lacking that would smooth the restore process? I would not hesitate to launch a public appeal, particularly if donations of money would be useful.
[WM: Whiskey!]
Alas it does happen. Be assured we are madly rebuilding ALL of our software designs to add better fault tolerance and automated fall over. In future we won’t have all of our servers in the same basket and indeed we will be adding a new mirror or two that are located in different geo locations as well as by different providers. I suspect that the comment about a new class of user (premium) may be something to examine in the future to better support our mission.
As Ward alluded to while we have a few constant supporters of the project (Such as RentPBX yah! and OpenVox yah!) we need a steady source of funds to help prevent this from occurring again. I am constantly amazed how many commercial users rely on our *FREE* services but fail to send us a few bucks for every install or heck every other install.
I remember when we needed to replace the 2 primary development machines and 2 supporters stepped up and literally fedexed us a couple of machines. Tony from our "competition" sent us an uber server when we needed to replace our project/compiler server that generates ALL of the programs and ISO’s
I just finished upgrading to the latest version of Asterisk 11/1.8 and it is available now.
Tom
aka Darmock
I personally get tired of trying to find the funds to keep the lights on…….