Zero Day Vulnerability Protection and More: Introducing Cover Your Asterisk

It’s been a difficult couple of weeks for the Linux® and Asterisk® communities with the back-to-back disclosures of the BASH Shellshock bug and then the FreePBX® Asterisk Recording Interface (ARI) bug a few days later. Both of these vulnerabilities have been circulating in the wild for years. We won’t repeat Wikipedia’s Zero Day Attack analysis other than to note that what makes these particular bugs so scary is not only the fact that both went undetected and unpatched for years but also that the attack vectors for both bugs were so simple. Anyone with a web server exposed to the Internet that was running any flavor of Linux or any Asterisk server with the FreePBX GUI was fair game for a seriously compromised server.

For those with shared servers in a hosted environment running under cPanel, your web platform typically runs with the equivalent of root privileges which means that any web intrusion inherits the same server privileges that you as the administrator had. This is similar to the way FreePBX runs with Asterisk. The same user account used for web access controls all of the Asterisk assets on your server. While it’s convenient, it’s also dangerous whenever there’s a web vulnerability because the entire Asterisk platform has exposure.

We always chuckle when one of the anonymous forum trolls launches a tirade claiming that these alerts are nothing more than Monday morning quarterbacking disguised as Chicken Little. What’s more amazing is that anyone would take the comments of an anonymous poster seriously especially on a matter involving server security. It’s one thing to label folks as alarmists for suggesting that the sky is falling when it isn’t. It’s quite another to launch these anonymous personal attacks even when there is documented evidence that the Internet sky was indeed caving in. Kinda reminds us of the global warming naysayers when the polar ice caps are melting beneath their feet.

According to the naysayers, we’re all doomed when it comes to cyberterrorism so why fight it. Here’s why. While reacting to security vulnerabilities has always been a defensive game of cat and mouse, that doesn’t mean you shouldn’t proactively do what you can to patch serious security holes in your servers. The alternative is to give cybercriminals a blank check to launch bots from your server that generate spam or participate in large-scale zombie attacks on our most trusted resources whether they’re DNS root servers, utility infrastructure and our electric grid, banking assets, and even national security resources. So let’s circle back and address what you can do to assure that you’re part of the solution rather than part of the problem.

For purposes of this discussion, our focus today is Asterisk server security. And the number one thing you can do to insulate your server from these vulnerabilities is to make certain that your web server is not exposed to Internet access by the general public. Neither Asterisk nor FreePBX requires public web server access to manage your server. In fact, neither Asterisk nor FreePBX requires any public access to your server to properly perform all required telecommunications functions. And the second paragraph above explains why this is especially dangerous with servers running both Asterisk and FreePBX.

So why do people still publicly expose their web servers and UDP ports 5060 and 10000-20000 to the Internet? As much as we hate to say it, it’s because it’s always been done that way. It’s also because there are a handful of SIP providers that still require UDP 5060 access to make and receive calls. Most do not! And even for those that do require UDP 5060 access, their requirements can be satisfied with a properly configured firewall that supports whitelisting of "safe" IP addresses for limited access. Incredible PBX comes preconfigured with a locked down WhiteList. The same can be added to PBX in a Flash by installing Travelin’ Man 3. We hope the other aggregations will follow suit. It’s long overdue.

Public web server access often is because there are more than a few (lazy) VoIP providers that install systems in a way that makes it easy for them to manage remote sites. Of course, a VPN would provide secure access to the same resources but that’s a little more work on the deployment end. With NeoRouter VPN, it’s a 5-minute job!

There also are companies with remote users or traveling salesmen that claim their servers must be open to the Internet to keep the company running. First, it’s hard to imagine a company whose salespeople don’t have cellphones that require no link to home base. Second, there are numerous solutions for safe connectivity with a home office: VPNs, FQDNs with dynamic DNS support, Port Knocker, and Travelin’ Man 4 to name just a few of the ones we previously have recommended. With the exception of the lazy VoIP installer, you will note that none of the above scenarios ever require web access to a VoIP server. So the rationale for public exposure of an Asterisk web server is all but non-existent.

The bottom line is that, if your server is not and has never been accessible from the Internet by typing its IP address into a public web browser and assuming your root password has not been compromised, then the BASH and ARI vulnerabilities are purely an academic discussion from your vantage point. Should you apply the patches anyway? Absolutely. Will your server be compromised if you don’t? Probably not… at least not from these two vulnerabilities.

That brings us to our topic for today. Having said all of the above, how do you really know if your server has been compromised by some zero day attack vector that none of us yet know about? After all, there are tens of thousands of applications installed on a typical Linux server. And a zero day vulnerability could be hiding almost anywhere.

First, a few words about what Cover Your Asterisk is not. This application won’t detect previously compromised servers! Wearing a condom the day after your wild night on the town isn’t all that helpful. If your server has been running as a public web server for the last 5 years, then our best advice is to start with a fresh install to a new, secured server. Then manually copy the settings (not the files!) from your old server to the new platform. Now you’re ready to protect your server.

Second, more than a few words about the VoIP environment in which we find ourselves. If you’re running any of the so-called Asterisk aggregations including PBX in a Flash, Incredible PBX, AsteriskNOW, FreePBX Distro, or Elastix, then your server includes some flavor of the FreePBX GUI, a web-based application to manage and configure Asterisk. As part of the FreePBX GUI setup, you give FreePBX 2.11 and beyond an expansive set of privileges on your server. These include read, write, and delete access to all of your web assets, all of your VoIP-related MySQL database assets, and all of your Asterisk assets. You also grant FreePBX rights to inventory and monitor critical pieces of information about your server so that you can be informed about pertinent FreePBX updates. We don’t see this as a bad thing. But, even with the incredibly talented FreePBX development team, this application design can be dangerous for a number of reasons not the least of which is the events of the past week. Consider for a moment a scenario in which a disgruntled employee or a web vulnerability allows somebody to modify a critical Asterisk configuration file such as manager.conf which controls access to the Asterisk Manager Interface, or to adjust MySQL’s admin.ampusers table which controls web access to the FreePBX GUI, or even to insert a malicious module into FreePBX which "looks and feels" like part of FreePBX. When you don’t know what you’re looking for, detecting subtle changes can be extremely difficult even for the most talented people in the business. For everyone else, it’s next to impossible. This is especially true when the changes aren’t noticeable in the standard day-to-day operation of your server. That was what led us to conclude that an additional detection mechanism was essential to highlight hidden changes made to any of the critical components that make up the Asterisk platform. Thus was born Cover Your Asterisk.

The Elastix folks apparently weren’t comfortable with this arrangement and forked FreePBX years ago and moved to a self-managed environment. The drawback has been their pace of releasing updates and patches, and that apparently applies to the unaddressed ARI bug as well.

The remaining aggregations all function as we’ve described. Before we delve into Cover Your Asterisk, here’s a little known tip. On the output side, FreePBX is basically a code-generator for Asterisk. Once you’ve configured your server using the FreePBX GUI, there is no Asterisk-FreePBX linkage of which we’re aware that requires your web server to remain operational. That turns out to be a good thing. What this means is you can shut down Apache and still have a fully functional Asterisk server with all of the functionality of your FreePBX-designed configuration. Given the times in which we live, that may not be such a bad idea.

So what does Cover Your Asterisk do? What we’ve sought to do with this GPL2 application is to take a snapshot of your most valuable Asterisk and FreePBX assets and then create checksums of all the individual components. This includes the /etc/asterisk, /var/www/html/admin, and /var/lib/asterisk/agi-bin directories as well as the Asterisk DB and MySQL’s asterisk database. Periodically, you then run another script which compares your current setup to the previous snapshot and identifies the changes for further examination. Once you are satisfied that any reported changes are legitimate, you then take a new snapshot of your server and periodically check it to make certain no unexpected modifications have crept into your system. A duplicate of these production assets is always maintained in a separate directory structure (/etc/asterisk.snapshot) accessible only by root. It can easily be converted into a gzipped tarball: tar -cvzf cya.tar.gz /etc/asterisk.snapshot. Then simply store the tarball off site for a rainy day emergency… when the sky falls once again.

Because this application was designed for production servers, its testing and scope have been limited to the Asterisk 11 and FreePBX 2.11 platform. For our installed base, that translates into PIAF-Green with FreePBX 2.11 and all flavors of Incredible PBX 11 running atop CentOS, Scientific Linux, Ubuntu 14, Debian, and Raspbian platforms on both Intel and ARM hardware including the Raspberry Pi, BeagleBone Black, CuBox, and PogoPlug.

Log into your Asterisk 11 server as root and issue the following commands to install the Cover Your Asterisk software:

cd /root
wget http://incrediblepbx.com/cover-your-Asterisk.tar.gz
tar zxvf cover-your-Asterisk.tar.gz
rm -f cover-your-Asterisk.tar.gz

To take the original snapshot of your server, run: /root/protect-your-ASSets.sh

To check your current setup against the snapshot, run: /root/check-your-ASSets.sh

To compare a file with its snapshot, run: diff /dirpath/filename /etc/asterisk.snapshot/dirpath/filename

To restore a snapshot file to your current Asterisk configuration, run these commands:

cp -p /etc/asterisk.snapshot/etc/asterisk/filename /etc/asterisk/filename
amportal restart

For Raspberry Pi and BeagleBone Black users, change the MySQL root password in both scripts:

sed -i 's|passw0rd|raspberry|' /root/protect-your-ASSets.sh
sed -i 's|passw0rd|raspberry|' /root/check-your-ASSets.sh

Finally, let us close with several recommendations. First, before making changes to your server with FreePBX, always run check-your-ASSets.sh, correct any detected problems, and then run protect-your-ASSets.sh to create a new snapshot of your server. After making any changes with the FreePBX GUI, run check-your-ASSets.sh again to verify that the changes you sought to make were, in fact, the changes that actually were made to your server. Then finish up by taking a new snapshot. These scripts take less than 30 seconds to run on a typical server so this is not a cumbersome process.

Before you restore any snapshot file or if you are puzzled by any changes you see listed after running check-your-ASSets.sh, we strongly recommend that you first seek advice from the gurus on the PIAF Forum. They can help you identify the severity of the problem, if any, and recommend an appropriate course of action for correction of the problem.

Finally, a cautionary note. Cover Your Asterisk is still a project in development. This means there will be changes/improvements as the coming weeks go by. One wrinkle with updates is your previous snapshots will have to be checked before you update. And then the newest protect-your-ASSets.sh script will need to be run following the update. To keep track of future releases and what’s included, visit this development thread on the PIAF Forum. Enjoy!

Originally published: Monday, October 6, 2014

Need help with Asterisk? Visit the PBX in a Flash Forum.

Special Thanks to Our Generous Sponsors

FULL DISCLOSURE: ClearlyIP, Skyetel, Vitelity, DigitalOcean, Vultr, VoIP.ms, 3CX, Sangoma, TelecomsXchange and VitalPBX have provided financial support to Nerd Vittles and our open source projects through advertising, referral revenue, and/or merchandise. As an Amazon Associate and Best Buy Affiliate, we also earn from qualifying purchases. We’ve chosen these providers not the other way around. Our decisions are based upon their corporate reputation and the quality of their offerings and pricing. Our recommendations regarding technology are reached without regard to financial compensation except in situations in which comparable products at comparable pricing are available from multiple sources. In this limited case, we support our sponsors because our sponsors support us.

BOGO Bonaza: Enjoy state-of-the-art VoIP service with a $10 credit and half-price SIP service on up to $500 of Skyetel trunking with free number porting when you fund your Skyetel account. No limits on number of simultaneous calls. Quadruple data center redundancy. $25 monthly minimum spend required. Tutorial and sign up details are here.

The lynchpin of Incredible PBX 2020 and beyond is ClearlyIP components which bring management of FreePBX modules and SIP phone integration to a level never before available with any other Asterisk distribution. And now you can configure and reconfigure your new Incredible PBX phones from the convenience of the Incredible PBX GUI.

VitalPBX is perhaps the fastest-growing PBX offering based upon Asterisk with an installed presence in more than 100 countries worldwide. VitalPBX has generously provided a customized White Label version of Incredible PBX tailored for use with all Incredible PBX and VitalPBX custom applications. Follow this link for a free test drive!
 

Special Thanks to Vitelity. Vitelity is now Voyant Communications and has halted new registrations for the time being. Our special thanks to Vitelity for their unwavering financial support over many years and to the many Nerd Vittles readers who continue to enjoy the benefits of their service offerings. We will keep everyone posted on further developments.
 

Some Recent Nerd Vittles Articles of Interest…