Comments on: Lessons Learned: An Update on Our Server Outage

By: darmock

darmock — Sat, 18 May 2013 14:50:18 +0000

Alas it does happen. Be assured we are madly rebuilding ALL of our software designs to add better fault tolerance and automated fall over. In future we won’t have all of our servers in the same basket and indeed we will be adding a new mirror or two that are located in different geo locations as well as by different providers. I suspect that the comment about a new class of user (premium) may be something to examine in the future to better support our mission.

As Ward alluded to while we have a few constant supporters of the project (Such as RentPBX yah! and OpenVox yah!) we need a steady source of funds to help prevent this from occurring again. I am constantly amazed how many commercial users rely on our *FREE* services but fail to send us a few bucks for every install or heck every other install.

I remember when we needed to replace the 2 primary development machines and 2 supporters stepped up and literally fedexed us a couple of machines. Tony from our "competition" sent us an uber server when we needed to replace our project/compiler server that generates ALL of the programs and ISO’s

I just finished upgrading to the latest version of Asterisk 11/1.8 and it is available now.

Tom

aka Darmock

I personally get tired of trying to find the funds to keep the lights on…….

By: lgaetz

lgaetz — Sat, 18 May 2013 12:55:35 +0000

Is there anything you are lacking that would smooth the restore process? I would not hesitate to launch a public appeal, particularly if donations of money would be useful.

[WM: Whiskey!]

By: Rob

Rob — Sat, 18 May 2013 05:59:36 +0000

if you want want a distant off site backup
I could likely spare a 60 gigs
I am in canada not to far to ship by disc if need be
nor am I so close as to suffer the same failings a local location could

By: Felix

Felix — Fri, 17 May 2013 23:39:38 +0000

When DSL Reports was down for several weeks last year, it was a brute reminder that things that we assume "are just there", come at a price. When it came back, I became a premium member the next day (after some 5 years of "freeloading"). I am looking forward to chipping in to PIAF as well (by subscribing to "paid updates" or in some more meaningful way). I am sure I am not the only one feeling that way. When everything comes back (hopefully, with minimum damage) - maybe similar "premium" or "contributing" status on the board could be instantiated. As opposed to "guru" that is earned :) [WM: Sad to say, you get what you pay for. We're the only non-commercial group still in existence in the VoIP space. And the "contributions" can be counted on your two hands each year.]

By: WaldoTheMerciful

WaldoTheMerciful — Fri, 17 May 2013 20:05:47 +0000

I know this is small, but I tried to go to the special
Vitelity link to sign up and got a 404 error. Would
you happen to have that link handy?

By: Vann Hall

Vann Hall — Fri, 17 May 2013 19:21:19 +0000

Alas, as of 12:15 PDT, still no PiaF-Green love. (I’m assuming the server outage is what’s responsible for that payload being reported as ‘unavailable,’ and not something stupid on my part.) But thanks for your prompt reply; now I know hopes imminent, I’ll keep checking!

[WM: May be that your DNS cache has not yet refreshed. Worked for me.]

By: elfonblog

elfonblog — Fri, 17 May 2013 19:15:49 +0000

Something Vann said struck a very clear chord with me. Instead of making the PIAF .ISOs little more than a build of CentOs with scripts to download and compile Asterisk, FreePBX, etc., it would be far more preferable to include everything in the initial "payload" so that we can get our PBX up and running even in the event that the mothership has been shot down. We're going to have to download everything anyway, why not up front instead of later. It's one of the reasons I prefer the VM builds, too. I imagine it looses you the opportunity to monitor installations quite so closely, and to slip-stream bug fixes into already-released products, but consider the goodwill folks will feel at not being quite so blatantly tethered to your servers. [WM: Blatant tethering is probably here to stay. The problem with the all-in-one approach is that you end up having to put out monthly ISOs, and we're not going to get back in that business. Sorry.]

By: Vann Hall

Vann Hall — Fri, 17 May 2013 18:13:47 +0000

Ward –
We’ve exchanged tweets and similar about this, but I’m more concerned about not being able to install PiaF during the outage. I have an interminable installation ongoing for a client with the bad luck also to have suffered a disk crash shortly before yours. The good news is we are still in test and acceptance, so the cord had not yet been cut to the analog PBX. I’m also still waiting for the production hardware to be delivered (ordered?). The design calls for a RAIDed system, mounted in the data room on a UPSed rack, with a hot spare, instead of a partially street-scored system mounted in an odd tower-with-a-handle case left over from my days as a VJ, tucked away in the back corner of whichever cubicle is empty this week, so presumably the ‘real’ PBX won’t drop and shatter at the first hiccup in power. However, as you know, even the most careful disaster avoidance plans can fail, and I can think of a number of scenarios where performing an installation from scratch might prove one’s best or only path to recovery. It’s been uncomfortable enough trying to explain why, after three days, I still haven’t been able to revive what is essentially an extraneous, unused system; I can’t imagine what it would be like if we were talking about a device carrying 100% of their telephony.

I know there have been cries of "get thee to the cloud!" — as if it were simply a forklift upgrade to cloud-ify something as complex as the PiaF discussion fora. On the other hand, though, it seems to me the cloud would make a perfect home for the gigabytes of code and executable that make up a PiaF payload. At the very least, there should be an inclusive offline loader available for each significant PiaF release.

For some reason, no one ever wants to hire me to work on something I know anything about, and this current project (which also had me looking into biometric access control methodologies and solar plant management systems) is no exception. Accordingly, for every hour that goes toward choosing and implementing a solution, I typically spend two questioning my sanity and second- [and third-] [and fourth-]guessing my choices. I looked into 32 different ITSPs before selecting one; I researched a ridiculous number of hosted and on-site PBX offerings (months later, Shoretel is *still* calling); and once I’d settled on the best approach to take, I evaluated every major, and quite a few of the minor, value-added packagings of Asterisk currently available. As a result, I can say with complete confidence the trifecta of Asterisk, FreePBX, and PiaF/Incredible PBX is far and away the best possible solution for this client. And I still do — this current installation glitch notwithstanding. Its just that, without the glitch, your story would be that much more compelling….

Thanks.

[WM: We’re testing the restored .org site as we speak. Once it’s working, installs will be fine. The ISO will switch to the .net site for files when it can’t find them on .com. You’re more than welcome to try a new install now. Feedback (good or bad) is most appreciated.]