Grandstream has done with Asterisk what Samsung and others did with Android. You basically take a freely available, open source toolkit and transform it into a terrific piece of turnkey hardware with tremendous savings in development costs. While it’s great for consumers, to us it highlights what is wrong with the GPL2 license which lets companies do this in the first place. These for-profit companies give almost nothing back to the open source community. Remember, it’s not their toolkit which took talented (and uncompensated) developers hundreds of man-years to construct. In Samsung’s case, they built closed source smartphones and tablets. With the Grandstream UCM6100 series, you get closed source PBXs. What’s wrong with this picture? Lots! You’re taking someone else’s work product, embellishing it to make a profit, and returning nothing to the open source community that made your open source product possible in the first place. Don’t get us wrong! We love Samsung’s smartphones and tablets. We’ve owned at least a half dozen of them. And Grandstream’s UCM6100 is an incredibly useful appliance for home offices as well as small and large organizations. We can think of a thousand use cases for the UCM6100 in the corporate and government workplace. If done right, it could easily have replaced the $200,000 PBX that supported 100+ employees in one of my former organizations. We also should note that Grandstream isn’t the first company to attempt this feat with Asterisk. Read Tom Keating’s excellent article for the history. And don’t forget the AA50 for a few cents more.
What is disappointing is that all of these products would be so much better and so much safer if the companies would open source their code and encourage community development to finish the job they started.1 No individual and few companies could match the hardware development platform that Samsung and Grandstream have managed to put together. In Grandstream’s case, you can buy the UCM6102 at retail for $264! It includes two FXS ports for devices such as fax machines and two FXO ports for interconnecting your Ma Bell PSTN trunks to a one-pound SIP powerhouse. That $264 buys you an incredibly attractive piece of hardware with an LCD that tells you everything about your PBX at the click of a button. And there are small LEDs to display the status of the LAN, WAN, USB, SD card, Phone, Fax, and both Telco lines. The device can sit under your phone on your desk in a SOHO office, or it can be wall-mounted in the closet of a bank’s branch office. Models are also available with 4 FXO ports (pictured above) as well as 8 and 16 FXO ports. One of these could meet the needs of almost any organization, regardless of size. Amazing hardware technology, really!
The web-based software user interface (UI) is no less impressive. FreePBX® has been our development partner on open source Asterisk® projects for the better part of a decade. To say they’ve made Asterisk what it is today is an understatement. Asterisk is a toolkit. FreePBX makes it a useful PBX for millions of users around the globe. Having said all of that, competition makes the world go ’round. And Grandstream has built an impressive UI for the UCM6100 devices. What is more amazing is to compare the performance of the Grandstream device to our own Incredible PBX for the Raspberry Pi which runs with Asterisk and FreePBX on a virtually identical processor with the same memory constraints as the UCM6100 devices. Night and day is the only way to sum it up. The Grandstream PBX literally runs circles around the Raspberry Pi in hardware and UI performance. In fact, you would never know the Grandstream PBX wasn’t running on a quad-core processor with several gigs of RAM if you were judging by performance. And there’s even a little fan that comes on about once an hour as if to remind you that there’s a real computer under the covers.
After receiving our UCM6102 late last week, we put it through its paces. We set up extensions and trunks and ring groups and outbound routes and inbound routes. We tested voicemail. We configured an IVR. We uploaded custom voice prompts. We tried out the Parking Lot and Call Forwarding and Conferencing. It all worked swimmingly, and configuration took only minutes with the web-based UI which was quite intuitive given its similarity to older releases of FreePBX such as 2.8 and 2.9.
But, in the words of Geoffrey Chaucer, “All good things must come to an end.” Our next mission was to interconnect the UCM PBX with one of our existing PBX in a Flash servers. After all, the real utility of a turnkey PBX appliance like this would be to support a branch office with no technical staff in residence. This would allow a bank or a hospital or a real estate company to interconnect sites with extensions at each site that could transparently connect to each other. For example, dialing 5000-5099 would ring phones in the main headquarters while dialing 5300-5399 would ring phones in branch office #3. For this to work in the Asterisk environment, we need password-protected trunks on each Asterisk server that interconnect the PBXs to each other to form a meshed network. It’s not difficult, and we’ve explained how to do it in previous Nerd Vittles articles using PBX in a Flash as well as Incredible PBX for the Raspberry Pi.
Trunk to Trunk Server Connections. As the screenshot above shows, connecting a trunk from the Grandstream PBX to our Asterisk server was a breeze using both SIP and IAX trunks. But attempts to connect a trunk from the Asterisk server to the Grandstream PBX using both SIP and IAX failed with password errors. When we alerted the Grandstream development team, suffice it to say they were confused. Did we mean we wanted to connect a remote Asterisk server to an extension on the UCM6100? That was the first hint that all was not well in Asterisk Land. It became readily apparent that the developers were quite adept at mimicking the functionality of FreePBX to create a powerful PBX. But they lacked an in depth understanding of some of the Asterisk fundamentals. While the Grandstream development team was incredibly responsive, it reinforces why open sourcing their code would provide huge benefits not only to others but also to their own project. It gets worse, unfortunately, much worse.
To make a long story short, it doesn’t appear that safely interconnecting trunks between Asterisk servers and the Grandstream devices is available at least at this juncture. What is possible and what the Grandstream developers documented is the ability to create a trunk on a remote Asterisk server that registers to an extension on the Grandstream PBX. But this still did not enable users on remote Asterisk servers to call extensions on the Grandstream PBX unless the Allow Guest Calls option was enabled in the device’s SIP settings. That didn’t make a lot of sense to us if, in fact, the remote Asterisk server was actually registered to the Grandstream PBX. So we changed the password on the extension to make sure the registration would fail. And, yes, you still could make calls to the Grandstream PBX extensions so long as Allow Guest Calls was enabled. Did we mention? It gets worse, much worse.
IVR Vulnerability. Remember that IVR setup we mentioned? By default, it sits on extension 7000 on the Grandstream PBX. We called it from an extension on the remote Asterisk server, and it worked as expected even without a valid SIP registration so long as Allow Guest Calls was enabled. You probably can guess what our next test was. We disabled Allow Guest calls and attempted to call an extension on the Grandstream PBX. It rang busy as it should. We then dialed extension 7000, and guess what? The call went through. Whoa! Remember, SIP guest calls had been disabled, and there was no SIP registration because of a password mismatch. In short, anybody from anywhere that knew the public IP address of our Grandstream PBX could now connect to any IVR on the device just by knowing that the IVRs begin with extension 7000. It’s a classic dial plan mistake of letting external calls bleed into privileges which should be reserved for internal users. For security and other reasons, it’s also why FreePBX does not assign extension numbers to IVRs. But there’s more.
Stealth AutoAttendant Gone Bad. As you can see from the IVR Setup screen shown above, two of the options available when setting up an IVR are to enable calls to Extensions and to Trunks. Many administrators as well as casual users that barely understand what they’re doing probably would enable these features believing the options would be restricted to local use by the default guest call restriction. Wrong! What it means in terms of this security lapse is that now any anonymous caller with your IP address can dial into your Grandstream PBX and, while the IVR announcement on the default IVR extension (7000) is playing, the anonymous caller can dial any Extension or any long distance call supported by the Grandstream PBX trunk configuration so long as these options were enabled in the IVR. In Nerd Vittles parlance, think of it as a remake of our Stealth AutoAttendant with Public DISA Connectivity… for the world!
FXO/PSTN Warning. In discussing this with Tony Lewis of Schmooze and FreePBX fame, he reminded me that we’re talking about a PBX that’s been designed for business use with FXO ports and PSTN trunks. So, while the SIP vulnerability at least required that someone know the IP address of your PBX, once you connect PSTN lines to the Grandstream PBX and answer incoming calls with an IVR on the system, all bets are off. Anonymous bad guys now can place PSTN calls to any published phone number for your server that happens to connect to an IVR. These calls then can be used as the springboard to place outbound calls to anywhere the PBX trunk setup permits. Get out your checkbook!
Syslog Configuration. We have another concern with the device as well. The default syslog setup sends information to log.ipvideotalk.com which is a server registered to Grandstream Networks in Los Angeles. With a closed platform, you have no way to decipher what is actually being sent without putting Wireshark on the line and monitoring it. While we are not suggesting that Grandstream has anything but the best of intentions, we think it’s a better practice to allow folks to opt in to monitoring systems, particularly ones that provide as much confidential information as the Asterisk syslog setup.
Other Security Issues. Having owned the device for only a few days, we obviously have not tested all of the potential attack vectors. There are other anomalies in the dial plan code which we really can’t quite figure out without seeing the actual code. We were going to try to document an equally serious issue with the trunk peering, but your head would probably explode just trying to wrap your head around the problem. Ours did! Suffice it to say, with a single outbound route to a registered trunk that has failed to register, all outbound calls initiated by internal and external callers should always fail. They don’t! We’re also unclear whether the appliance provides SSH access for the root user. In any case, you aren’t provided the password. That could potentially be a problem if, in fact, a root account is enabled on the appliance. Finally, we should note that, according to the GPL materials published by Grandstream, this appliance is running Asterisk 18.104.22.168. Twenty-five versions of Asterisk 1.8 have been released since that offering appeared eight months ago. Some of those updates patched serious security vulnerabilities in the Asterisk 1.8 code.
Until Grandstream addresses some of these security issues, you are well advised to only operate a Grandstream PBX behind a secure, hardware-based firewall with no Internet port exposure. We would caution against connecting PSTN trunks to the device at this juncture. If you’re feeling lucky, a possible option for the time being would be to disable IVRs and especially the extension and trunk dialing options. That alternative unfortunately defeats the real purpose of buying these devices.
I Have A Dream. Not to beat a dead horse, but discoveries like this reinforce the need for companies such as Grandstream to revisit their design strategy and give serious consideration to open sourcing their code. After all, Grandstream is primarily a hardware company, and they could sell a gazillion of these appliances if the platform were open. We’ve hurriedly compiled a list of features that currently are missing which could be added almost overnight if this were an open source project. The PBX in a Flash development team would be at the front of the line to assist!
- No text-to-speech functionality
- No speech-to-text functionality
- No (intended) DISA functionality (but data is collected in syslog??)
- No ability to load custom dialplan code
- No AGI/PHP script support
- No Google Voice support for free calling in U.S. and Canada (add it for $30 like this)
- No SIP/IAX trunk registrations from remote Asterisk servers
- No incoming calls except via anonymous SIP or PSTN (nixes interoffice setups for extensions)
- No traditional fax support except using fax machine on FXS port (T.38 is supported)
- No access to Asterisk CLI for debugging or otherwise
- Crippled SSH access (basic config info, set/get variable, upgrade, reboot, reformat)
- No VPN support
- No SIP security with Internet exposure
- No Fail2Ban support
- No WhiteList security to lock down the server
Recommendations. In closing, we don’t mean to suggest that security vulnerabilities never occur in open source code, but open source does guarantee that hundreds if not thousands of developers would be reviewing the code rather than a handful of people that may not fully appreciate all of the nuances of Asterisk. And each time a discovery like this occurs that has the potential of costing unsuspecting companies thousands of dollars in unanticipated phone bills, it gives Asterisk an undeserved black eye. Issuing a patch unfortunately won’t cure this problem for most purchasers because most purchasers never upgrade firmware on appliances.
We hope Grandstream will either pull the devices from the marketplace until the default firmware is fixed or place a big orange warning sticker on the boxes warning purchasers to upgrade the firmware and explaining the consequences of not doing so. Better yet, do the right thing and open source the platform and the code so that others can benefit from Grandstream’s development work on what still could be an incredibly useful and amazing device.
July 31 Update: After an exchange of emails with Grandstream, we have a better understanding of their call routing methodology that we want to pass along. It should be noted that the security holes we documented still exist, but there are mechanisms in place to stop the bleeding… if you know how to use them. Grandstream relies upon a set of Privilege Levels for extensions and IVRs as well as inbound and outbound routes. These include Internal, Local, National, and International. Only Extensions and IVRs with matching or higher privileges can use Inbound and Outbound Routes of a matching or lower privilege level. Read that again! It’s important. For example, if an extension has Internal privileges (the default), then that Extension can only access Outbound Routes designated as Internal. Calls to other numbers will fail. Unfortunately, all routes default to Internal, and this security mechanism is barely documented in the User Manual. Unlike FreePBX which uses Outbound Routes to connote calls leaving your server, Outbound Routes in Grandstream parlance are a set of dialplan rules for every call. Stated differently, to have a secure system, you need to create an Outbound Route for every possible type of external AND internal call. The same holds for Inbound Routes. Here’s an example of how to safely configure Trunks and Extensions between the Grandstream PBX and a remote Asterisk server so that extension-to-extension calls can be made between the two offices while insulating your IVRs from the long distance free for all that we documented in the original article.
Unfortunately, the IVR setup is still buggy and hence vulnerable. As the chart at the end of this article makes clear, there presently is no way to configure an IVR in such a way that remote callers cannot make long distance trunk calls while local extensions can. The only options presently available are either to disable the Dial Trunk option or to set the IVR Privileges lower than the Privileges setting for your outbound trunks. Do NOT rely upon a separate IVR for local users with the Dial Trunk option enabled thinking you’re safe. You’re not! Our original article above explains the possible consequences.
Remote Asterisk Server Setup Using FreePBX. On our remote server, we want to create two Trunks and an Outbound Route. One trunk will be used to set up an outbound registration to an Extension on the Grandstream PBX. We’ll use this trunk to place calls to Grandstream PBX extensions, IVRs, and conference rooms. The other trunk will be used to authenticate an inbound registration from the Grandstream PBX. The Grandstream PBX extensions will use this trunk (with registration from the Grandstream PBX) to initiate calls to extensions registered on our remote server. The outbound route will be used to route calls using the outbound registration trunk to Grandstream PBX extensions, IVRs, and conference rooms.
Here is the outbound registration trunk to extension 5001 on the Grandstream PBX (192.168.0.120 in our example):
Here is the inbound registration trunk to authenticate the Grandstream PBX matching trunk:
Here is the outbound route that allows extensions on the remote server to call Grandstream extensions, IVRs, and Conference Rooms:
You would also want to create an Inbound Route for 5001 that sends incoming calls from dialing 5001 on a Grandstream PBX extension to a particular destination on your remote server. Otherwise, the calls would be processed using the FreePBX default inbound route if you happen to have one. In our setups, we typically point the default inbound route to an IVR or a receptionist’s extension.
Grandstream PBX Setup to Connect to Remote Asterisk Server. To make all of this work securely, we need to create an Extension to handle the inbound registration from the remote Asterisk server so that users on the remote server can call extensions, IVRs, and conference rooms on the Grandstream PBX. And we need a SIP trunk that will register to the remote Asterisk server so that Grandstream PBX users can call extensions on the remote Asterisk server. Then we need Inbound and Outbound Routes to lock things down. We’re using 192.168.0.181 as the IP address of the remote Asterisk server in this example. The key point in securing the Grandstream PBX is to assign the proper permissions to the Grandstream Extension and IVRs that will be used with remote server connections. Then elevate permissions where necessary on the Inbound and Outbound Routes to make sure only our truly local extensions can make calls using Grandstream long distance and PSTN trunks. Don’t confuse local extensions with Local permissions. A local extension is an extension that registers to the Grandstream PBX. Local permissions is a security level that means a particular resource can only do things with other matching Internal or Local resources and with no resources that have been assigned a higher permission level. Internal permissions means a resource can only do things with other Internal resources. Clear as mud? We know. Hang in there until we’re finished.
First, create extension 5001 that will be used by the remote Asterisk server to register with the Grandstream PBX:
Next, create a SIP Trunk that will register to the remote Asterisk server at 192.168.0.181. We’ve used 1234 as the password in our examples so plug that in for the time being. You obviously would want something more secure than that! You’ll note that you don’t assign a Permission level to a Trunk. That is handled in the Inbound and Outbound Routes which tie particular routes to designated trunks. So Trunks inherit their permissions based upon a matching route. We suspect this may be the root cause of the security holes that we’ve documented. If there is no specified route for a particular type of call, Grandstream is doing something internally to make a determination on whether to allow the call or not. In some cases, that determination just happened to be wrong.
For truly local users, i.e. extensions directly connected to the Grandstream PBX, you need to elevate the Permissions for those extensions to reflect the types of calls you want them to be able to make. Typical permission for these extensions would be National or International. The same holds true for IVRs. Elevate IVR permissions to restrict usage to your intended audience. Keep in mind that we’re treating calls to extension 5001 on the remote Asterisk server as Internal. That’s the bottom rung in the security ladder which means every local extension and IVR will be able to place calls to that extension. If this isn’t what you want, then you’ll need to elevate the 5001 extension permissions accordingly. For example, you may only want Grandstream PBX extensions with Local call permissions to be able to call extensions on the remote PBX. In this case, you would want to change the 5001 extension permission level to Local.
Let’s tackle the Inbound Routes next since this was the cause of the inability to connect to local Grandstream extensions from the remote server. If you’re using the default Grandstream setup, then you’ll need Inbound Routes for both _50XX extensions and _70XX IVRs to permit remote callers to connect with Grandstream PBX extensions and IVRs with Local permissions only. This means that even if they connect to the 7000 IVR, they will not be able to make long distance calls on your nickel even if Trunk dialing is enabled.
The Inbound Route rule for Extensions should look like this:
The Inbound Route rule for your IVRs should look like this:
The key point to keep in mind with Inbound Route IVR permissions is to keep the permission level LOWER than whatever permission level you assign to the Outbound Route for placing calls that cost you money, typically National and International.
Now let’s set up the Outbound Route to restrict outbound calls to 10-digit numbers for extensions, IVRs, and Inbound Routes to those with at least National permissions. Keep in mind you may need additional outbound routes with Local permissions for certain 10-digit numbers if your local calling area happens to include free calling to multiple area codes, e.g. Atlanta.
Depending upon your setup, you may need additional dialplan rules and outbound routes to handle 11-digit numbers which should be routed out through a PSTN trunk, e.g. 1NXXNXXXXXX. And because of the security hole, be sure to add a catch-all for international calls that requires International permissions. The dial string XXXXXXXXXXX. will catch everything not included in the NXXNXXXXXX and 1NXXNXXXXXX outbound rules.
Finally, you’ll need an Outbound Route that allows local callers on the Grandstream PBX to connect to extensions on the remote PBX. You typically would assign Internal or Local permissions to this route which would look something like the following depending upon the extension configuration on your remote PBX:
A Word of Caution on IVRs: In the Grandstream security model, IVRs have their own Privilege levels. At least at this juncture, that Privilege level can “promote” the permissions of a call that began at a lesser privilege level. For example, if your Inbound Route for 7XXX calls is assigned Local privileges and the 7000 IVR is assigned National privileges, an incoming call to 7000 from a remote PBX will “inherit” the National privileges of the IVR. This obviously should never be possible. Either the 7000 IVR should generate Congestion and not answer the call at all where the Inbound Route has lesser privileges than the IVR. Or, at the very least, those options in the IVR (including stealth extension and trunk dialing) that require National or International privileges should generate Congestion and disconnect the call. For the time being, ALWAYS set the Privilege level of an IVR to the lowest permission threshold to protect your server and wallet from the consequences of placing unintended toll calls. Here’s a little chart we put together to document the impact of merely changing the Privilege setting for the 7000 IVR:
Other Tips and Tricks. Here are a few other suggestions to expand the functionality of your Grandstream PBX:
Deals of the Week. There are a couple of amazing deals still on the street, but you’d better hurry. First, for new customers, Sangoma is offering a board of your choice from a very impressive list at 75% off. For details, see this thread on the PIAF Forum. Second, a new company called Copy.com is offering 20GB of free cloud storage with no restrictions on file size uploads (which are all too common with other free offers). Copy.com has free sync apps for Windows, Macs, and Linux systems. To take advantage of the offer, just click on our referral link here. We get 5GB of extra storage, too, which will help avoid another PIAF Forum disaster.
Originally published: Tuesday, July 30, 2013
Need help with Asterisk? Visit the PBX in a Flash Forum.
Don’t miss the first-ever FreePBX World on August 27-28 at the Mandalay Bay in Las Vegas. For complete details, see this post on the FreePBX blog.
We are pleased to once again be able to offer Nerd Vittles’ readers a 20% discount on registration to attend this year’s 10th Anniversary AstriCon in Atlanta. Here’s the Nerd Vittles Discount Code: AC13NERD.
New Vitelity Special. Vitelity has generously offered a new discount for PBX in a Flash users. You now can get an almost half-price DID from our special Vitelity sign-up link. If you’re seeking the best flexibility in choosing an area code and phone number plus the lowest entry level pricing plus high quality calls, then Vitelity is the hands-down winner. Vitelity provides Tier A DID inbound service in over 3,000 rate centers throughout the US and Canada. And, when you use our special link to sign up, the Nerd Vittles and PBX in a Flash projects get a few shekels down the road while you get an incredible signup deal as well. The going rate for Vitelity’s DID service is $7.95 a month which includes up to 4,000 incoming minutes on two simultaneous channels with terminations priced at 1.45¢ per minute. Not any more! For PBX in a Flash users, here’s a deal you can’t (and shouldn’t) refuse! Sign up now, and you can purchase a Tier A DID with unlimited incoming calls for just $3.99 a month. To check availability of local numbers and tiers of service from Vitelity, click here. Do not use this link to order your DIDs, or you won’t get the special pricing! Vitelity’s rate is just 1.44¢ per minute for outbound calls in the U.S. There is a $35 prepay when you sign up. This covers future usage and any balance is fully refundable if you decide to discontinue service with Vitelity.
Some Recent Nerd Vittles Articles of Interest…