JUST RELEASED: Visit the Incredible PBX WikiYou may have read that a user discovered last week that current trixbox systems as recently as today include a remotely-configurable BOT, a software program that can execute certain commands locally once it receives its instructions. Reportedly, trixbox’s registry.pl "phones home" to Fonality via the Internet at 3:41 a.m. each morning to get a list of Linux commands to run. It then executes those Linux commands on your server while you’re sleeping. If the assertions of trixbox end users are true and we have no reason to believe otherwise, the existence of this remotely-configurable BOT had never been disclosed to unsuspecting users whether they were individuals or corporations. In fact, it doesn’t appear that even trixbox resellers were aware of the existence of the remotely-configurable BOT.
Let me hasten to add that Chris, Andrew, and Kerry have been good business partners of Nerd Vittles for years even though I’ve never personally met any of them. So I would never suspect that any one of them would use a tool like this for improper purposes. Our objection is more fundamental and goes to the existence of the tool itself and the failure to disclose it. Unfortunately, a remotely configurable BOT with root access privileges is a bit like giving someone a blank check… with your signature affixed. And it’s worse in this case because users had no notice that they were handing over the keys to their castle by installing and using trixbox. One can’t help wondering if Fonality management really grasps how dangerous such a system design is in this day and age. This isn’t about the commands that Fonality was executing. It’s about the commands that could be executed if this system were ever compromised. We have daily logs full of attempts to hack our systems using, you guessed it, remotely controlled BOTS.
We don’t for a minute believe that Chris Lyman and other senior management of Fonality knew about this in advance, but they certainly know now! The problem is that many programmers, in attempting to perfect the world’s finest software app, fail to consider what would happen if a tool like this one got into the wrong hands, for example the hands of a disgruntled employee. Unfortunately, just about every organization has at least one not-so-happy camper, and companies usually don’t know how dangerous such employees are until it’s too late. We obviously have no idea what safeguards Fonality may have put in place to monitor access and prevent abuse of this tool. For everyone’s sake in the Asterisk® community, we would hope LOADS OF THEM! A security breach at Fonality would basically hand over all of these trixbox systems for remote command execution as root. Or, if anyone’s DNS system is compromised, affected trixbox servers are now everyone’s worst nightmare. Hello!!!
As with many business decisions presented to organizations, the balancing act here is whether the benefits of collecting what have been represented to be marketing and usage statistics outweighed the risks if your absolute worst imaginable scenario came to pass. Merely revealing the existence of this tool made most folks shudder. And it’s still in operation. Remember, any Linux command or application could be executed with root privileges using this BOT. Take a look at the 25+ pages of comments on the trixbox forum, Google’s VoIP Users Conference, VOIPSEC, and now Slashdot if you have any doubts about the user reaction. Do we really think the crackers of the world can’t read? Is this what we want folks to remember when they hear about Asterisk?
Now imagine control of a tool like this getting into the wrong hands where someone could actually compromise the security of outside companies that knew nothing about its existence. All it took to execute commands on every newly-deployed trixbox server in the world was creation of a list of commands presumably stored on a server within the Fonality organization somewhere. Now you can appreciate how threatening a software design decision can be.
Having a hard-coded reporting mechanism that everyone is notified about up front was one thing, and that’s where this collection process began with trixbox 2. But it morphed into an open-ended, remotely-configurable BOT. And that is something quite different and downright dangerous. Suffice it to say, if we ever hope to seriously introduce Asterisk into the business community, there’s no room for BOTs in the equation, much less hidden ones. No business would knowingly tolerate an open-ended, remotely configurable BOT running on any server inside its corporate firewall, particularly one with the breadth of Linux applications at its disposal that one would normally find on trixbox systems.
This clever software should have been reviewed by senior management before it ever saw the light of day. The episode gives all of us a golden opportunity to stop and think about what we’re doing and what our fundamental obligations are to those who use our code. Hopefully, Fonality will turn this BOT off… permanently! The problem, of course, is that it’s hard to unring a bell. This BOT is already in the wild. Luckily there’s a very quick solution in this case. Here’s the command that should be added to tomorrow morning’s Fonality script: rm -f /var/adm/bin/registry.pl. We’ll all sleep better.
We hope everyone in the Asterisk development community will make a pledge to be open about the existence and scope of any future data collection processes associated with Asterisk offerings. Then users can make an informed choice on whether to use your software. A new trixbox forum member put it this way:
There is an understanding between users and developers. The understanding is often tacit but is nonetheless there. The understanding goes, "I will be executing something you wrote. I do not have the time/ability to check it all, but as professionals, I expect you to behave in a manner befitting that trust." –Minupla
We couldn’t have said it better. As for our own software, we want to be crystal clear: No Remotely-Configurable BOTs Ever! They have no legitimate purpose when weighed against the very substantial security risks they pose to all of us.
Full Disclosure. With the help of some very talented partners, Nerd Vittles now has an Asterisk-based PBX offering of its own, PBX in a Flash. It arguably "competes" against Fonality’s trixbox ce even though both offerings are free for the taking. Having written over 100 columns touting the beauties of trixbox, we felt some obligation to warn our users who may have upgraded to a more recent version of Fonality’s software. You may also want to review this article from Philippe Lindheimer, the lead developer of FreePBX.
Some Recent Nerd Vittles Articles of Interest…
I’ve been very worried since reading about this yesterday. Now one question I am sure a lot of people have been asking themselves is: "How do I tell my clients without losing their trust?"
My approach will be to call my clients and let them know that a security bug has been found, and that I will be having to go and patch their systems. But, since this is a top priority bug, there will be no charge to them…
I believe that, by being open, honest, and fair to my clients, they still will respect and keep trusting me. What do you think?
Not sure what’s up with your comment submital, it simply goes to a blank page so if you received multiple postings – i appologize, if you dont receive any of these, never mind!
I cannot understand how a competent engineer would architect a system in this way. Maybe they just used a design that was already in service elsewhere?
Does anyone know whether the Fonality proprietary PBX system phones home in the same way?
[WM: Good questions.]
Well, if you look at the registry script itself, basd on the server id, it contacts either proregistry.trixbox.com or update.fonality.com, but executes the results the exact same way.
This makes me think this is in fact the same mechanism used in the fonality PBX! I wonder if the Fonality user base know how vulnerable they have been!
Looks like my analysis is correct based on Kerry’s response. This is a much bigger problem than Fonality was willing to admit to.
I would have thought better of the NV staff than to take advantage of something like this, while it is a definite risk that they have, and they should have disclosed this, and people should be definitely allowed an opinion on it, but it certainly doesnt deserve this much bad publicity, it sounds like Enron, and yet nobody has been swindled, and they are putting forth an effort to fix it before anyone does. Some people dont understand that people buy PBXtra and Pro because they WANT someone to be able to control their system and fix it when there is a problem, they want those updates to just happen without any thought, they bought a hosted PBX on purpose, and people think they are concerned about the manufacturer having access to their phone system – like GM Onstar – unfortunately the free users didnt buy that same system, but got it with slightly less security – big problem, simple solution – tell everyone, and reverse it. dont like it – get PIAF, or CentPBX, or rubberpbx, or whatever – these people have lost nothing, and have a choice even if they want to use trixbox without the phone home mechanism, just remove it from cron and its gone, nobody is shuving it down their throat, and yet there are people that dont even use trixbox who have posted bashing remarks to various forums in an effort to give themselves self importance, a rush, or I dont know what gain they have by doing what they have done, remarks about sleeping with the project managers wife – people need to really calm down. Not that I agree on several things that fonality does, or their methods, but dont beat someone up for a so far harmless mistake.
I still support the Nerdvittles project as they have been free and have been ingenious and invaluable, but this support is like that of a politician, I will vote for the one that has done the least bad now instead of one that I like.
[WM: A lot of people took our advice in well over 100 Nerd Vittles articles and tried Asterisk@Home and then trixbox these past few years. We felt some obligation to warn folks that may have upgraded to ‘Heartbeat V3.0’ as it is now described. Call it what you will, but if it walks like a duck, and quacks like a duck, it’s still a duck. This went well beyond being a heartbeat. It was a remotely-configurable BOT that penetrated the security of any enterprise in which it was installed.
Having been responsible for the management of secure intranets for many, many years, I can tell you that this software qualified as a fundamental security breach. And people literally have written books about all of the problems inherent in the Fonality design of this BOT. The lack of notice or informed consent also was inexcusable.
For anyone that took our advice, tried trixbox, and then, on their own, upgraded to a current version of trixbox, I wanted to be sure (1) they knew about this and (2) they knew what our position on the appropriateness of the design and lack of notification was. If our article came across as bashing or kicking someone when they were down, that was certainly not the intent. And I can assure you the article was not motivated by some burning desire to give away more free software.
After reading all of Kerry’s responses, it appears the remotely-configurable BOT is still running today. What’s unclear is whether this was some new hairbrain design (as I originally thought) or something that was ported over from Fonality’s commercial products. That’s pretty scary, too! Providing reliable phone service is a noble goal but not at the expense of adding serious infrastructure vulnerability for your users IMHO.]
I hope this issue wont make Kerry or Andrew the fall guy for fonality
them and the product they produced will always get credit for what it is
and what they have done for the asterisk comunity I have to say I feel like I am jumping ship just before she starts to sink
Ward and the rest of the great people here I have to thank you so far I’m pretty happy with pbx in a flash
So, I can’t find any registry.pl script.
root’s crontab is empty, and asterisk’s crontab only lists /var/www/html/admin/modules/framework/bin/freepbx-cron-scheduler.php
Does this mean I have an early enough version that I am not affected? Are there other "phone home" mechanisms in use in the version I have (version 2.0.0)?
[WM: ‘Heartbeat V.3.0′ didn’t come out ’til later. You’ve still got ‘Heartbeat V.2.0.’ You’ll need to go to the trixbox forums to find out how to disable it. I don’t think they thought up the clever name until yesterday so don’t search for heartbeat.]
Just a heads up that a trixbox exploit now has been published and explained. If you have not patched your system, you would be well advised to do so and not wait for Fonality to release a fix. See also this entry on the CVE List.
The problem is not just Trixbox, Fonality’s commercial VoIP solution establishes a VPN connection, which could re-directed by poisoning the target’s DNS. Sure one can disable it, but unless Fonality needs to get into the system for troubleshooting, why bother to keep a VPN established? …that is unless they have some other purpose with YOUR phone system.
It’s about trust. And the Fonality crew has incinerated all of it.
First Asterisk@Home suddenly changed names to TrixBox. This raises concern in the user community. The reasons given at the time were that Digium held a trademark and that they couldn’t use Asterisk in the name any more.
Then it is discovered that magnanimous Fonality is "sponsoring" Asterisk@Home/suddenly-Trixbox. But, don’t worry they aren’t taking over or anything.
Then we find out that Fonality is selling a proprietary version of TrixBox and now offers an appliance. But, don’t worry, TrixBox is still freely available. Never mind that it is increasingly crippled by comparison to the proprietary version.
Then the new free version comes out and you have to register just to install modules and updates. But, don’t worry, we’re not using the information for anything and nothing has changed, we just want to get an idea of how many people are using our product. Trust us.
Now, it is discovered that a surreptitiously placed module is installed and is phoning home nightly and that module can and does execute code of Fonality’s choosing as a trojan or bot does. But yet again(!) Kerry and crew down play the whole thing http://www.trixbox.org/forums/trixbox-forums/open-discussion/trixbox-phones-home claiming that:
It’s not a big deal because Trixbox has always phoned home.
It only collect statistics.
You can trust us.
We understand that you are uspset about it but we are working as hard as we can to develop a fix but, that will take days at a minimum.
It was unintentional.
You were never at risk.
All of the BS that has flooded out of this project over the past year and a half will hopefully end here. The TrixBox project and its Fonality drones are no longer trust worthy. If they ever were.
"nobody is shuving it down their throat, " Mr. Hyde on comment #6 above.
That’s a wrong analogy. The analogy is like you were served a hamburger at a party but it has a parasite inside that it is going to live inside your body until you go to see a doctor, and only if the doctor finds out to tell you and treat you.
Sure nobody has "shuved" a hamburger down your throat, as it’s not polite at a party.
Give me a break. Your comment is equivalent to that we should never worry about this kind of security matter. You basically said that it is no big deal. Well, 99.9% of the people out there won’t agree with you.
And I’m not sure if you are hinting that the only reason there is a sucurity risk is only because the free users choose to take a free version vs. a commercial version. Are you kidding? The commercial version also has a controversial phone home "feature" as well and many people didn’t like that either.
And so far, I have not read about anything about sleeping with project manager’s wife or anything like that here. If you have a problem with such out of line stuff then take it to the place where it happened. Don’t take it here because I can’t read or even find out where, and you are lumping all the negatives here only to equalize the basic issue (fallacy).
You said "just turn it off, it’s no big deal". May be it is not being turned off because users didn’t find out about it. May be b/c Fonality didn’t inform the users?
If I were the person who is responsible for deploying something with a remote-BOT inside, I’d be very worried about my future career, even though it was not my own fault per se.
Mr. Hyde, you can’t be more serious? Gee
I have a slightly different perspective on all this. Yes, it was a pretty boneheaded blunder. But it took them only six days to fix it, or at least it was six days from the first post on the Trixbox forums to the fix:
http://www.trixbox.org/audit-tool-fix-being-pushed-out-tonight
The post titled "trixbox CE audit tool official statement and "fixes"" includes an apology, details about what happened, and the fix to the Heartbeat script:
http://www.trixbox.org/forums/trixbox-forums/open-discussion/regarding-trixbox-trojan
From "it’s not a problem" to "oops" and then "all fixed" in six days is nearly warp speed. How about giving them a little credit for listening, and then taking care of the problem? A more typical response is to stonewall, evade, and then release both attack lawyers and oily PR persons.
I haven’t seen anything about a privacy policy, which seems to be the remaining bit of unfinished business.
Well, it’s been about one month, and well over one year since I asked them to post their privacy policy. As of today, I still don’t see one. I guess in terms of "warp speed" that would be -8 or so.
They said that we could disable it. Really? I don’t understand how we could disable it without even knowing of it’s existence.