JUST RELEASED: Visit the Incredible PBX Wiki
The last few weeks have certainly reinforced the notion that one should never ASS-U-ME anything unless you’re willing to learn the hard way when things go south. We’ve also uncovered a new twist to the Golden Rule: "He who has the gold makes the rules." In the Digium®-centric Asterisk® world, it goes something like this. When life is good, we reserve the right to cash in on the proceeds. When things go wrong, the Asterisk community needs to do better testing. It’s a free product, and you get what you pay for.
We wish we could say that our suggestion that Digium eat its own dog food before releasing new Asterisk versions to the public was well received. Quite the contrary, and we probably should have learned several years ago about the tenor of responses one could expect when suggestions were made to change the Digium Way of doing things. In the previous case, we had suggested that altering dialplan syntax and punctuation between Asterisk versions was counter-productive because it broke almost every existing Asterisk application. That was sloughed off as being someone else’s problem since the Digium developers could not possibly anticipate all of the problems that would be caused by changing verbs and syntax in the dialplan.
Think of what would happen if you moved the location of the brake pedal on every new car, and you get some idea of the scope of the problem for Asterisk application developers, assuming you still can find the ones that wrote your company’s application.
Testing Methodology… NOT! With the release of Asterisk 1.8.4, we suddenly encountered a new can of worms. Virtually all Cisco SIP and Polycom TLS phones no longer worked. Keep in mind that this is the only "fully supported" (whatever that means) version of Asterisk that is still available. In the case of the Cisco phones, Digium managers claimed that they didn’t have every piece of equipment on the planet so it wasn’t their fault. In the case of Polycom, it turned out that Digium’s multi-million dollar headquarters reportedly is chock full of Polycom phones, but they’re all plugged into a commercial PBX that didn’t have the problems engineered into Asterisk 1.8.4.
That brings us to the Hobson’s Choice now facing existing and would-be Asterisk users. Wouldn’t you think that a company that profits enormously off hardware and software sales because of their "free" Asterisk product would have some rudimentary test lab in place with a dozen or two phones from the major VoIP manufacturers so that new releases could be checked out before the production-ready release is distributed? Well, apparently not. Kinda reminds us of an old Huntsville comment about the Apollo moon missions. Would you want to fly to the moon in a spacecraft built by the lowest bidder? For Huntsville’s Digium Corporation, the question might be phrased a little differently. Why would any organization want to stake its livelihood on an untested Asterisk PBX?
Does free really matter if your phones don’t work?1
As one of Asterisk’s primary cheerleaders for many, many years, this latest revelation that there is an almost complete lack of testing before production versions of Asterisk are released is disappointing to us not to mention incredibly short-sighted on Digium’s part. Since Digium appears unwilling to actually use their own product internally, we’d like to propose a dog food alternative.
First, instead of more leather chairs for the new Digium headquarters2, how about a 200 square foot test lab in the attic with a few $250 Atom-based PCs and a couple of under $1,000 Dell servers running Proxmox and VMware virtual machines with a couple dozen flavors of Asterisk. Then add a dozen SIP phones from the leading VoIP providers as well as a few of the leading ATAs. $5,000 would easily cover the total cost of the lab. How do we know? Well, the PBX in a Flash Dev Team (with no VC funding) has had a similar setup in two locations for years. We even do testing for outside organizations from time to time. 🙂
Make Lemonade Out of Lemons. Better yet, if we were king, the testing facility would be moved front and center to the first floor behind a glass showcase so that every visitor could see that Digium was just as serious about testing its products as it was about its revenue-generating training room and its foosball table. Click on Tom Keating’s photo of the Digium facility for the corporate tour. Testing is a matter of corporate pride in most organizations, not something to be ashamed of… unless you don’t happen to do much of it. Indeed, the comments we’ve received from Paul Belanger suggest that at least some of the Digium folks have their hearts in the right place about all of this. And, just because some Asterisk developers are not on the corporate payroll, the buck clearly stops with Digium, The Asterisk Company, to make certain that the Asterisk product is rock-solid reliable before it goes out the door.
Second, build a checklist of functions that must pass muster before any new Asterisk version is released. Ever heard of a Digium card that didn’t work with a new Asterisk release? Didn’t think so. We’re guessing this is something more than coincidence. The overall software reliability of Asterisk affects Digium’s bottom line just like hardware reliability even if the software product is touted as being free. Digium profits from Asterisk hardware sales, Asterisk consulting, Asterisk training, Asterisk conventions, Asterisk support, and numerous Asterisk software add-ons that cost money. If the reliability of Asterisk goes down the tubes, so goes the commercial side of Digium’s business as well.
Third, don’t depend solely upon software-driven tests in checking out new releases. Nothing beats a human at the controls for a day to give new software a proper workout. Make calls from every phone to every other phone on the same and on a different network to verify call quality and reliability. Then do the same thing using POTS phones connected to ATAs. When all of that works, move on to a short list of major Asterisk features to make sure they remain stable. Sounds simple, doesn’t it? It is. We do it regularly with no profit motive at all. Here’s our short list of two dozen deal-breakers, and our readers can probably suggest a couple dozen more. We’ll add them to the list as they arrive. If you don’t want to design a system for testing, then feel free to use The Incredible PBX with our compliments. All of these turnkey features are available out of the chute, and you can install it from a thumb drive on almost any hardware.
Text-to-Speech Apps
Conferencing
Music on Hold
Call Transfers
Call Forwarding
Call Waiting
Call Pickup
Call Recording
Call Parking
Do Not Disturb
Voicemail
Caller ID
IVR Samples
Faxing
Video Calls
Queues
Ring Groups
Zap Barge
Intercom
AGI Scripts
Google Talk/Jabber/Jingle
SIP Server Connectivity
IAX Server Connectivity
VPN Server Connectivity
Here’s hoping that we all get something positive back from Digium management this time around. Hopefully, they’ll realize before it’s too late that their future really does depend upon a reliable Asterisk product. And, no, we’re not going to print any response suggesting that users turn back to Asterisk 1.4 and 1.6.2 when Digium and the Asterisk developers are on record as being unwilling to address a bug such as the one that occurred in Asterisk 1.8.4 if instead it had arisen in either of the older versions of Asterisk that are barely on life support.
Every organization has defining moments. This is an important one for Digium. Take responsibility for the quality of your product! And, rather than focusing upon whether to call the next version of Asterisk 1.10 or 2.0, spend the necessary time and money to get the Asterisk 1.8 house in order. Otherwise, the VC-funded office building may belong to another fish in the growing sea of VoIP providers one day soon. It’s worth remembering that Digital Research of CP/M fame3 as well as WordStar, Ashton-Tate, Lotus, and WordPerfect all were household names and seemingly invincible software development houses once upon a time. History has a way of repeating itself. Wonder why?
Continue reading Part I, Part III, and Part IV…
Originally published: Wednesday, June 1, 2011
Changes in PBX in a Flash Distribution. In light of the events outlined in our recent Nerd Vittles article and the issues with Asterisk 1.8.4, the PIAF Dev Team has made some changes in our distribution methodology. As many of you know, PBX in a Flash is the only distribution that compiles Asterisk from source code during the install. This has provided us enormous flexibility to distribute new releases with the latest Asterisk code. Unfortunately, Asterisk 1.8 is still a work in progress to put it charitably. We also feel some responsibility to insulate our users from show-stopping Asterisk releases. Going forward, the plan is to reserve the PIAF-Purple default install for the most stable version of Asterisk 1.8. As of June 1, Asterisk 1.8.4.1 is the new PIAF-Purple default install. Other versions of Asterisk 1.8 (newer and older) will be available through a new configuration utility which now is incorporated into the PIAF 1.7.5.6.2 ISO.
Here’s how it works. Begin the install of a new PIAF system in the usual way by booting from your USB flash drive and pressing Enter to load the most current version of CentOS 5.6. When the CentOS install finishes, your system will reboot. Accept the license agreement, and choose the PIAF-Purple option to load the latest stable version of Asterisk 1.8. Or exit to the Linux CLI if you want a different version. Log into CentOS as root. Then issue a command like this: piafdl -p beta_1842 (loads Asterisk 1.8.4.2), piafdl -p beta_1841 (loads Asterisk 1.8.4.1), piafdl -p 184 (loads Asterisk 1.8.4), piafdl -p 1833 (loads Asterisk 1.8.3.3), or piafdl -p 1832 (loads Asterisk 1.8.3.2). If there should ever be an outage on one of the PBX in a Flash mirrors, you can optionally choose a different mirror for the payload download by adding piafdl -c for the .com site, piafdl -d for the .org site, or piafdl -e for the .net site. Then add the payload switch, e.g. piafdl -c -p beta_1842.
Bottom Line: If you use the piafdl utility to choose a particular version of Asterisk 1.8, you are making a conscious decision to accept the consequences of your particular choice. We would have preferred implementation of a testing methodology at Digium before distribution of new Asterisk releases; however, that doesn’t appear to be in the cards. So, as new Asterisk 1.8 releases hit the street, they will be made available through the piafdl utility until such time as our PIAF Pioneers independently establish their reliability.
Need help with Asterisk? Visit the PBX in a Flash Forum.
Or Try the New, Free PBX in a Flash Conference Bridge.
whos.amung.us If you’re wondering what your fellow man is reading on Nerd Vittles these days, wonder no more. Visit our new whos.amung.us statistical web site and check out what’s happening. It’s a terrific resource both for us and for you.
Special Thanks to Our Generous Sponsors
FULL DISCLOSURE: ClearlyIP, Skyetel, Vitelity, DigitalOcean, Vultr, VoIP.ms, 3CX, Sangoma, TelecomsXchange and VitalPBX have provided financial support to Nerd Vittles and our open source projects through advertising, referral revenue, and/or merchandise. As an Amazon Associate and Best Buy Affiliate, we also earn from qualifying purchases. We’ve chosen these providers not the other way around. Our decisions are based upon their corporate reputation and the quality of their offerings and pricing. Our recommendations regarding technology are reached without regard to financial compensation except in situations in which comparable products at comparable pricing are available from multiple sources. In this limited case, we support our sponsors because our sponsors support us.
BOGO Bonaza: Enjoy state-of-the-art VoIP service with a $10 credit and half-price SIP service on up to $500 of Skyetel trunking with free number porting when you fund your Skyetel account. No limits on number of simultaneous calls. Quadruple data center redundancy. $25 monthly minimum spend required. Tutorial and sign up details are here.
The lynchpin of Incredible PBX 2020 and beyond is ClearlyIP components which bring management of FreePBX modules and SIP phone integration to a level never before available with any other Asterisk distribution. And now you can configure and reconfigure your new Incredible PBX phones from the convenience of the Incredible PBX GUI.
VitalPBX is perhaps the fastest-growing PBX offering based upon Asterisk with an installed presence in more than 100 countries worldwide. VitalPBX has generously provided a customized White Label version of Incredible PBX tailored for use with all Incredible PBX and VitalPBX custom applications. Follow this link for a free test drive!
Special Thanks to Vitelity. Vitelity is now Voyant Communications and has halted new registrations for the time being. Our special thanks to Vitelity for their unwavering financial support over many years and to the many Nerd Vittles readers who continue to enjoy the benefits of their service offerings. We will keep everyone posted on further developments.
Some Recent Nerd Vittles Articles of Interest…
- There’s been a lively debate about all of this in the Comments to the original article and on the PIAF Forum and the FreePBX Forum, three eyeopeners you won’t want to miss. [↩]
- Digium HQ photo courtesy of Tom Keating. Click on the photo for a tour. [↩]
- Gary Kildall flew his own airplane, too. He reportedly was off on a flying adventure while Bill Gates was meeting with IBM to seal the DOS deal. The rest, as they say, is history. [↩]
I work in a test lab in the telecommunications industry and I am absolutely floored that Digium doesn’t even test the software in house! The software and hardware depend on each other and nobody’s going to buy Digium hardware cards if the software isn’t dependable. C’mon, Digium. This is unacceptable.
I’m sure the folks who designed our lab would be happy to consult with Digium on how to build a proper test environment. 🙂
I agree with several of your points.
Looks like they have released a fixed version on 1.8.4.1 http://www.asterisk.org/node/51639
[WM: You are correct, and the PIAF Dev Team and a number of PIAF Gurus have been independently verifying its reliability. As of 1 p.m. today and after thorough testing, we are pleased to advise that Asterisk 1.8.4.1 is the new default PIAF-Purple install. See the paragraph above the Pioneer with the Arrows for other available options.]
I would like to say how frustrating it has been to watch Digium keep posting smart ass comments all over the place instead of just saying "Sorry we broke everyone’s phones in the new release and here is what were doing to make sure it does not happen again".
Adam,
If you find a comment by a Digium employee that you feel is a smart-ass comment, please let me know. I can be reached via malcolmd digium com.
Cheers.
Morning,
Over the last few weeks many people have provided their feedback on the topic of testing, everything from frustration and disappointment to suggestions and solutions. My goal, in this post, is to try and take all the arguments (good or bad) and see what we can do as a community to move forward.
@Anonymous: Here is your chance to consult, we’ve been trying to get more people involved in #asterisk-testing (IRC channel on Freenode) to help talk about this issue, why not join up and share your experiences?
@DJ: Correct, in fact the fix for the Cisco phones was merged into branches/1.8 sometime ago, however the fix was not flagged to be included in 1.8.4, why? That’s what we need to resolve. Currently we rely a lot on the community to provide feedback to release candidates (RCs) with 1.8.4 we had a 3 month window for testing, however not one report of the Cisco regression was reported. As reported, Digium did not notice this however why did the community not notice too?
Adam: Sorry. I know that does not fix the issue though.
There are countless other suggestions in the previous threads, however I’ll not rehash them all. So as a blanket statement, here is a call to arms, your chance to get involved and fix the issues you have all been responding too.
Leif will be tagging 1.8.5-rc1 soon, aside of the tests run under the testsuite and unit tests, let us start defining some test cases / plans for people to run. What we need is step by step guides explaining what you need to do and the expected results. Additionally, we need to define what to do once a test fails.
Leif and I started brainstorming last week about the release process (https://wiki.asterisk.org/wiki/display/~lmadsen/Asterisk+Release+Process), while Malcolm has spec’d out a functionality test (https://wiki.asterisk.org/wiki/display/~mdavenport/Functional+Testing+Matrix). Feedback? Is this something you would be interested in getting involved with?
Again, if anybody has questions / comments or wants to get involved, feel free to find me in #asterisk-testing on Freenode. I’m happy to provide feedback or answer questions.
PB
@PB. I agree that "we" as the Asterisk community need to be more thorough when testing and reporting bugs/issues.
However Digium as the project "Manager/Moderator/Owner" should also take better steps regarding regression testing as this is not the first time something completely breaks in a "stable" version release.
I am not hammering the good people and their hard work, which I am very thankful for, but instead am agreeing that more care is needed in order to move forward with this project at the speed that the "world" expects it to grow.
As a community we already contribute a lot to the project in many different ways: Code, Reports, Tests, Marketing and advertising (yes, advertising Asterisk – or any other open/free software for that matter – as an enterprise grade, solid, scalable platform is not easy task ) and most of us do this on spare time as we need to work on other projects for a living.
So if Digium could be a bit more careful in releasing reliable, stable , regression tested asterisk versions as LTS it would be awesome to everyone.
Anyways… Thanks to all the who somehow one way or another help in the Asterisk project.
DJ
Thanks again Ward for another insightful article. I really enjoyed TMCNET’s review of Digium’s headquarters – I need a Collaboration Room like that!!
What people forget are that these are real-time communication systems and asterisk is a project sponsored by Digium – a software company, not an integration company – and the software they happen to release is distributed without warranty and perfection.
There’s a few things to consider here:
1) The operating system these systems are built on rarely have more than 5 or 6 years of a supported life cycle. To expect a system to last 10 years is unrealistic – 10 years ago, yes this was possible because you were installing/buying a product that had no moving parts, convection cooled and had modular replaceable parts from a proprietary vendor.
2) When upgrading a customer site, you should always have run through the process in a dev environment and not go about being a cowboy sys admin as I know there are many.
3) Read the changelog when considering an upgrade to a system. Many times the changelog incorporates changes that don’t warrant a system upgrade – you’re essentially doing work for no reason at all except that you enjoy recompiling code. For instance, the IAX channel may have updates along with the unistim channel, but if you’re customer is only using SIP – why upgrade?
The 1.8.4 fiasco was certainly an oversight on Digium’s part, and I’m quite surprised there’s a lack of internal testing with available hardware and a checklist of human testable conditions.
In modern times, user acceptance testing has become more common part of a project handover. Perhaps this is a task that needs some community work seeing as we have a software license that offers no warranty.
There are many many occasions when handset firmwares have their own idiosyncrasies – this alone would mean that a reseller/installer would have to have the business process of certifying their own products with version numbers – for their own peace of mind. The same could apply to switches and routers – in years gone by, how many switches and routers have needed a reboot for any number of unknown reasons.
In a VoIP phone call there are many points of failure – even on a LAN let alone a local phone call.
These days there are waaaay too many options in configuring devices – stick to a set of known, working configs instead of going for bleeding edge on live customer sites – you’re only creating negative energy for yourselves.
In the case of PIAF and other distros, it’s clever to have a bleeding edge, a testing, stable release program. You cannot rely on the upstream vendor because this is how they run their business – using free software licenses. It would be unreasonable to expect them to release a perfect product every release, software devs – as smart as some are, are still human.
I’m surprised Digium doesn’t do more thorough hands on testing as the sponsors of asterisk – perhaps this is something that needs addressing from the community as written in the comments.
Thanks for your time – from Australia!
CSTAChris
@DJ: Yes, I also agree, less regressions for Asterisk releases should be the goal. Additionally, getting more testers involved should be a goal too. I’m not disputing that ‘Digium as the project “Manager/Moderator/Owner” should also take better steps regarding regression testing’, in fact I’m trying to get the community to help define the steps. I don’t believe Digium should solely be responsible for that.
Let’s look at Fedora for example, they have a great process in-place for testing each release of Fedora. Sure the scale is much larger then Asterisk, however the underlining principles are the same. They have a series of requirements for each phase of testing; hundreds of tests. Each new release during the testing phase tests get more specific, bugs / regressions should be less and less. Since Fedora is sponsored by Red Hat does this mean Red Hat is responsible for all the testing? I’m sure they do help with the process however they are not the sole person responsible for it, the community is. Fedora has a fleet of testers, out testing the latest and greatest versions of Fedora. Why? What has Fedora done to attract all those testers? How can Asterisk do the same?
Chris Mylonas also makes some good points, regardless of the software being used, upgrade procedures should be tested in a development environment before moving into production. This adds another layer of testing to help catch issues that may affect your production systems.
There is just no excuse for not having a basic set of test procedures. When I was running the trixbox CE project we had a six page test plan that every team member had to run through for every build. The moment I left that testing process stopped and the quality of the releases dropped. A basic test plan of features and basic hardware testing is an absolute requirement.
Ward as always has quite a bit to say that is valuable, but the populist gouges about leather chairs and upscale offices really distract from what is important here. Huntsville isn’t as expensive as Silicon Valley. I’m certain this office space is a relative bargain, and the extra touches are appreciated to those who relocate there or visit on business. Sophisticated readers that aren’t going to miss that the real cost of testing isn’t 200 square feet and $5000 in equipment, but the salaries of those engaged in it.
[WM: You’re absolutely right on the chairs and the office space. It’s all quite beside the point. Not sure we agree on the need for testing. Producing a dependable product is what makes or breaks companies, and the associated cost of testing is just part of running a successful business. ]
@PB – Well… I believe that Red Hat puts a LOT of effort in testing RH Linux and as an effect the Fedora project has better product to "start" from.
But anyways, answering the question as to ‘how can we get more testers on board?’, the biggest difference between a distro project like Fedora or Ubuntu and the Asterisk project I believe to be the moving parts.
In order to setup and test an asterisk version a good amount of knowledge time and equipment is needed.
You need to compile code, configure dial plan, trunks, devices, applications and much more… while to test a distro all you gotta do is download an image and either run it on vm or boot through a CD or USB. The simplicity of the environment and it’s setup makes it attractive to test not only by developers bu also by the general public.
Maybe there should be a repository of actual workable dial plan an configuration (not only the samples and demo) that can be downloaded an used for testing, also would be interesting to have some sort of auto configure tool for testing that would make it easy to download, compile, install and test Asterisk.
Just my 2c.
Regards;
Oh.. how about a pre configured test VM image? chock full of test cases and dial plan goodies!
Download, run, test, report, wash, reuse until stable type o thing… 🙂
[WM: You’ve pretty well described what the Incredible PBX VM image is. Just install a PIAF virtual machine and the Incredible PBX VM edition, and you’re ready for testing. It exercises virtually every feature in Asterisk and FreePBX. Now all we need is some drivers. 😉 ]
@DJ: We can go back an forth all day 🙂 however I believe we are on the same page. Making it easier for people to get involve with testing or running a test should be the first step.
I’m currently working on a task this sprint to make it easier for people to write dialplan tests under the testsuite. Currently one needs to learn how the testsuite works, Python and maybe StarPy; a steep curve. Once finished, one would only need to create an Asterisk dialplan and drop it into a sub-directory. The testsuite would be responsible to everything else, creating the instances of Asterisk, installing the required files, originating calls, etc.
Additionally, I’d also like to decrease the amount of steps required to get the testsuite running on ones system. Right now it takes a fair bit of work, but not properly documented. Either by providing a VM or packaging the testsuite (RPM or DEB).
If you have not see, check out http://bamboo.asterisk.org and http://svn.asterisk.org/svn/testsuite for the source code.
[WM: We applaud everything that you are doing. Just a little surprised that it’s happening in June of 2011. But we’re delighted nonetheless. Our WAG would be that well over 75% of the Asterisk installations are probably CentOS, Fedora, or RedHat-based. So an RPM would be helpful to get the largest pool of volunteers for testing. Thanks, Paul.]
IN a previous life, I worked for DEC – testing was paramount there… As I recall, some software components (the Fortran compiler comes to mind) had over 100,000 discrete tests in their suite.
Testing wasn’t just "does it work". It was a) across all platforms (when you consider how many models in the VAX and Alpha Ranges, that got to be a big task – even though they were architecturally identical, it was still tested). b) across multiple backwards releases, or related software. c) Performance was also tested, right down at a code level (i.e. does this subroutine take longer to run, than previously. )
REsults were compared with existing data on all of the above criteria. ANd, of course, test engineers, as were tech writers, were engaged day one of the project.
It was the norm, it wasn’t difficult – much of it was automated (and we are talking near 20 years ago…) So there is little excuse for not ‘DOING IT RIGHT’
Umm…"What we need is step by step guides explaining what you need to do and the expected results." seems, to me, assbackward. Isn’t that something the project sponsor should be providing? Am I misinterpreting the personal pronouns? This drama is getting silly. freeSWITCH. Seriously.
That’s just crazy. I look after the regression test team at a large card transaction processing company and my system tests a million or two transactions every week, and a hundred thousand in detail every day.
All it takes is the will.
i