Asterisk Server Troubleshooting: Finding and Fixing Bugs & Gremlins in Your PBX

Ashton-Tate of dBASE® fame used to call them anomalies. Sheer arrogance kept them from ever quite admitting there was an actual bug in their software. We don’t claim to be quite so perfect. If you use any software for very long, you’re going to encounter bugs. But not all reported problems turn out to be bugs. Many turn out to be errors generated by users that don’t quite know what they’re doing because they never bothered to RTFM. Whether they’re bugs or self-inflicted wounds, the result is still pretty much the same. The code doesn’t work as advertised.

As part of our Back to School series, today we’re going to introduce you to a methodology to keep your Asterisk® server running smoothly… warts and all. That process begins by your clicking the Getting Started Guide at the top of the Nerd Vittles page, identifying your particular platform, and reading the applicable tutorial from front to back. About nine times out of ten, that will tell you whether you’ve encountered a bug or just a feature that you haven’t quite mastered.

Unlike the other Asterisk aggregations, Incredible PBX™ includes an Automatic Update Utility. It gets run whenever you log into your server as root and is there primarily to address security issues. Depending upon the severity of the bug, we address some non-security related bugs as well. The problem from our perspective is that we’re dealing with a moving target with almost a dozen different versions and platforms. On each of those platforms, there are literally hundreds of software applications that are maintained and "improved" by various developers around the world. Sometimes things break. The other fact of life is there are only so many hours in the day and about 95% of all users of open source software never contribute a dime toward any open source project. Translation: We depend upon volunteers rather than paid staff to report and fix our bugs and those of other projects upon which we depend.

We’re happy to provide the latest software with the latest bug fixes at no cost. The rest is pretty much up to you. If the bugs in your current version become intolerable, then install a newer release and chances are that most of the problems will have been resolved. Yes, you will have to manually reconfigure your extensions and trunks and routes, but everything else will pretty much be the same. That’s the only real cost of using free software.

We want to identify the most common reported problems with Incredible PBX today and show you how to fix some of these issues yourself. After all, this is supposed to be a learning experience. And learning to fix things for yourself or at least knowing where to turn to get the answers is the best thing you can do to assure you have a stable platform for years to come. There are two terrific sources of information to keep your system current and stable. The first is the RSS Feeds for Asterisk, FreePBX®, and PBX in a Flash/Incredible PBX. The second is the Forums for these three platforms: Asterisk, FreePBX, and PBX in a Flash/Incredible PBX, especially the Bug Thread. If you’re going to use open source software, then you owe it to yourself and your server to check all of these sources at least once a week and address any new issues that have been identified. The downside of not heeding this advice is you are exposing your server to potential attacks that may compromise not only your server but the servers of others as well once your system has been transmogrified into a Zombie.

1. Login Failures: Linux CLI, Incredible PBX GUI, Web Apps, AvantFax, WebMin

There are four different passwords that cause problems for new users. On the Incredible PBX platform, these get set in different ways. The confusion typically arises when the user attempts to access a server resource and a password prompt appears. Fixing the problem depends upon which password is being requested. Be advised that more than two attempts to guess a password may get you locked out of your server for several hours because of Fail2Ban. Then you have two problems to contend with rather than one.

Linux CLI Root Password. Regardless of the Linux platform, the root password gets set when you install the operating system. If you can’t log in as root from the server console, chances are pretty good that you’ve forgotten the password or typed it incorrectly. Fixing this problem is major surgery and often you will be better served by reinstalling your server. If you’d prefer to reset the password, then follow the steps in this Linux Gazette article. While it doesn’t apply to Incredible PBX builds, you may encounter a failed root login using SSH or Putty if the server has been configured to deny root SSH access or to require an SSH key to log in. This can be remedied by logging in from the console and reconfiguring the login parameters for either the CentOS/Scientific Linux or Ubuntu/Debian/Raspbian OS.

Incredible PBX GUI’s admin Password. If you’re using a browser to access the Incredible PBX GUI and you’ve chosen the GUI Administration option from the Incredible GUI Main Menu (shown immediately above), then you arrived at a screen that looks something like what’s shown above. Clicking on Incredible PBX Administration will generate a prompt requesting your username and password. There can be any number of account names to access various Incredible PBX GUI resources, but there will always be an admin account. Reset this password by logging into your server as root and running the script: /root/admin-pw-change

If you’ve gotten fancy and added a password to the Incredible PBX Main Menu (shown at the top of this Top 10 List) using Admin -> Menu Configuration, then it won’t be too long until you forget what it was. You’ll need it to get back into the Admin options on the Main Menu. To retrieve the password, display the contents of the following file and look at the entry between the first two commas:

Incredible PBX User Control Panel Access. Also shown above is an icon to access the User Control Panel (UCP). This typically is used to allow end-users to manage one or more extensions on your PBX. Account names and passwords for UCP access are created in the Incredible PBX GUI by choosing: Admin -> User Management -> Add New User

Incredible PBX Web Application Access. From the Incredible PBX Main Menu and/or from the Maintenance tab in Incredible PBX GUI, users and administrators can gain access to a number of Incredible PBX web applications including AsteriDex, Telephone Reminders, phpMyAdmin, AvantFax, and others. All of these web applications require Apache login credentials consisting of a username and password. In the case of AvantFax, you will also need an AvantFax username and password that is requested after your Apache credentials have been provided. All of these applications can be accessed using the admin Apache account. To set the admin password: htpasswd -b /etc/pbx/wwwpasswd admin newpassword
Separate end-user accounts also can be created for applications such as AsteriDex and Telephone Reminders. To set up each of these accounts, use the following syntax: htpasswd -b /etc/pbx/wwwpasswd acctname acctpassword
The admin account and password are required to access phpMyAdmin and other administrator applications.

AvantFax Web Admin Access. As noted above, you first must enter any valid Apache web credentials to access AvantFax from the Incredible PBX Main Menu or from the AvantFax tab within the Incredible PBX GUI. After successfully entering your web credentials, you will be prompted for an AvantFax username and password. The admin user account for AvantFax can be set by logging into your server as root and issuing the command: /root/avantfax-pw-change

Once you gain admin access to AvantFax, you can create additional user accounts and passwords by clicking the Dashboard icon shown above and choosing: Menu -> New User

WebMin GUI Access. WebMin is a tool for use by skilled Linux administrators only. You can seriously and irreparably damage your PBX by making changes within WebMin. You’ve been warned. To access WebMin on Incredible PBX servers, click on the icon in the Main Administrator Menu. Or you can access WebMin using https://ServerIPaddress:9001. The username must be root, and the password is your Linux root password.

2. Emails/Voicemails Don’t Get Sent/Delivered

Identifying the Problem. 99% of the problems with delivery of emails with voicemail attachments from your server have little to do with your server. They are the result of one of two things. Either your email client has placed the incoming email messages in its SPAM or JUNK folder, or your Internet Service Provider (ISP) is blocking downstream SMTP mail servers from sending email. The ISPs claim this is a security precaution to reduce SPAM generated from compromised servers.

Testing Email Delivery. Try this: echo "test" | mail -s testmessage yourname@emailserver.com

Fixing the SPAM problem. If you find the test email message in your SPAM or JUNK folder, there are two ways to go about fixing the problem. The simplest is to mark the sending email address (whatever it happens to be in the email message) as NOT SPAM in your email client. For Gmail, simply create a filter for the email sender and specify "Never send it to SPAM" and "Mark it important." The alternative is to assign a Fully-Qualified Domain Name (FQDN) to your server. This could be done using a Dynamic DNS Server such as dyndns.org. Once you’ve set up your FQDN, this thread on the PIAF Forum will walk you through assignment of the FQDN to your server.

Fixing ISP Blocking of Downstream SMTP Traffic. If your ISP happens to be one of those that blocks emails from downstream SMTP servers (e.g. Comcast), you will first need the FQDN of your ISP’s SMTP gateway. We will reconfigure SendMail to use your ISP’s mail gateway as the SmartHost for your server and send emails out using their SMTP gateway instead of yours. This usually means they will tack on some hidden information to the email messages so that they can identify the sender if a SPAM problem is reported. Once you have the name of your ISP’s SMTP gateway, log in as root and edit /etc/mail/sendmail.cf. Search for DS and append your ISP’s SMTP FQDN with no space, e.g. DSsmtp.comcast.net
Save the file and restart SendMail: service sendmail restart.

3. Asterisk Won’t Start/Restart

Identifying the Problem. This part is easy. Restart Asterisk and see what happens: amportal restart

The Fix. This part isn’t. Keep in mind that, on FreePBX-enabled Asterisk servers, FreePBX actually starts up Asterisk as part of its initialization process. Unfortunately, the error messages are cryptic. The causes can be just as obscure. If you’ve recently made changes to an Asterisk config file in /etc/asterisk, start there. The next thing to test is whether MySQL is functioning properly. Without MySQL, FreePBX won’t start. Here’s the simple test: /etc/init.d/mysqld restart

If MySQL fails to start, have you changed your passwords for MySQL? If so, change the root password back to passw0rd. Then test access from the Linux CLI: mysql -u root -ppassw0rd. Next, find your asteriskuser password in /etc/freepbx.conf. Now test MySQL access again using the password you deciphered: mysql -u asteriskuser -p

The other critical piece for a successful startup is PHP. If it isn’t functioning properly, Asterisk with FreePBX won’t start. From the Linux CLI, issue this command and look for errors: echo "< ?php phpinfo(); ?>" | php

If all else fails, Google is your friend.

4. IPtables Firewall Testing

There’s one component of Incredible PBX that separates the men from the boys. That’s a secure and functioning firewall with a WhiteList of those authorized any type of access to your server. Obviously, if it’s not functioning, you’re not secure. Running the status command from the Linux CLI should tell you whether IPtables is working properly. To be doubly sure, issue the following command: iptables -nL. If the result looks like this, you’re missing the IPtables config file and need to head to the PIAF Forum for some help:

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Next, issue the restart command and fix any reported errors: iptables-restart

Finally, there’s one more configuration setting that should be checked. Because the Travelin’ Man 3 WhiteList feature allows you to enter either IP addresses or FQDNs, we need to be sure IPtables is started up after DNS services are enabled on your server, or IPtables startup will fail. By default on many servers, IPtables startup occurs first. That means a manual configuration change is required to be sure IPtables startup is successful. Issue the following command to display the custom startup for your CentOS server: cat /etc/rc.d/rc.local. On Ubuntu/Debian/Raspbian platforms, issue the command: cat /etc/rc.local. The results should include: /usr/local/sbin/iptables-restart. If not, add it before exit 0.

5. Fail2Ban Log Scanner Testing

Fail2Ban is a log scanner that searches for certain text strings which indicate failed attempts to access your server. It is NOT failsafe because these text strings change from time to time and because your server must have sufficient horsepower to scan complete logs before the bad guys find a hole in your security. That often is difficult if the bad guys are using high-powered servers such as Amazon EC2. Nevertheless, it’s another layer of security that is worth having, and we need to make sure it’s functioning properly. Whenever IPtables is restarted, Fail2Ban needs to be restarted. This should be handled in the iptables-restart script. To be sure, issue the following command: grep fail2ban /usr/local/sbin/iptables-restart. It should return the following result: service fail2ban restart. If not, add it to the script by issuing the following command after logging into your server as root:

To verify that Fail2Ban is functioning, issue the command: iptables -nL

The final lines of the listing should look something like this:

Chain fail2ban-ASTERISK (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0

Chain fail2ban-asterisk-udp (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0


6. HylaFax/AvantFax in Limbo.

For those using Incredible Fax, there are two main problems: not being able to log in to AvantFax and the dreaded "IAXmodem please wait" message once you get there. On the CentOS platform, the most important fix is making sure you’re using the latest Incredible Fax install script. If you’ve already encountered the problem on CentOS servers, just download the new script, tar zxvf incrediblefax11-centos*, and run the installer again. We know: you’re not supposed to. If that doesn’t address your problem and you’ve already reset your admin password as documented in section 1, see this PIAF Forum thread for some helpful hints.

7. Asterisk Server Log Rotation

Identifying the Problem. Some Incredible PBX servers were missing the script required to rotate the Asterisk logs which means your /var/log/asterisk/full file continues to grow. To test whether your server is missing the log rotator, run this command: if [ ! -f /etc/logrotate.d/asterisk ]; then echo "Missing"; fi

The Fix. If the test above reports Missing, then issue the following commands as root to fix the problem:
cd /etc/logrotate.d
wget http://incrediblepbx.com/asterisk-logrotate.tar.gz
tar zxvf asterisk-logrotate*
rm -f asterisk-logrotate*


8. Detecting Trunk Failures

One of our most requested utilities is a script to notify administrators when a trunk goes off-line. Just issue the following commands to install it. Then edit /root/trunkcheck.sh and insert your email address. Works for SIP, IAX, and GV trunks.
cd /root
wget http://incrediblepbx.com/trunkcheck.tar.gz
tar zxvf trunkcheck.tar.gz
rm -f trunkcheck.tar.gz
echo "5 * * * * root /root/trunkcheck.sh > /dev/null 2>&1" >> /etc/crontab
nano -w /root/trunkcheck.sh


9. Detecting Whether Asterisk Is Running As Root

Don’t ask us why but on some servers Asterisk ends up running as the root user rather than the asterisk user. Given the current design of FreePBX which assigns almost all privileges to the asterisk user anyway, it’s more an academic problem than a real one. If an intruder gains asterisk user access to your server, your system is toast whether the intruder has root privileges or not.

Identifying the Problem. To test whether the main Asterisk program is running as root, just issue the following command: ps aux | grep sbin/asterisk. If the first column of the first entry in the list shows root, then apply the fix.

The Fix. Issue the following commands to reset the Asterisk application to run as the asterisk user:
amportal kill
chown -R asterisk:asterisk /var/run/asterisk
sed -i '/END INIT INFO/a AST_USER="asterisk"\nAST_GROUP="asterisk"' /etc/init.d/asterisk
amportal restart


10. Inability to Activate Repository Tabs in Module Admin

Identifying the Problem. Some users have reported a problem activating the various repository tabs within the GUI’s Module Admin component.

The Fix. Issue the following commands from the Linux CLI to correct the problem:

amportal a ma uninstall digium_phones
gui-fix

Special thanks to Lorne Gaetz and Andrew Nagy for the fix.

Originally published: Wednesday, September 2, 2015

Support Issues. With any application as sophisticated as this one, you’re bound to have questions. Blog comments are a terrible place to handle support issues although we welcome general comments about our articles and software. If you have particular support issues, we encourage you to get actively involved in the PBX in a Flash Forums. It’s the best Asterisk tech support site in the business, and it’s all free! Please have a look and post your support questions there. Unlike some forums, ours is extremely friendly and is supported by literally hundreds of Asterisk gurus and thousands of users just like you. You won’t have to wait long for an answer to your question.

Need help with Asterisk? Visit the PBX in a Flash Forum.

Special Thanks to Our Generous Sponsors

FULL DISCLOSURE: ClearlyIP, Skyetel, Vitelity, DigitalOcean, Vultr, VoIP.ms, 3CX, Sangoma, TelecomsXchange and VitalPBX have provided financial support to Nerd Vittles and our open source projects through advertising, referral revenue, and/or merchandise. As an Amazon Associate and Best Buy Affiliate, we also earn from qualifying purchases. We’ve chosen these providers not the other way around. Our decisions are based upon their corporate reputation and the quality of their offerings and pricing. Our recommendations regarding technology are reached without regard to financial compensation except in situations in which comparable products at comparable pricing are available from multiple sources. In this limited case, we support our sponsors because our sponsors support us.

BOGO Bonaza: Enjoy state-of-the-art VoIP service with a $10 credit and half-price SIP service on up to $500 of Skyetel trunking with free number porting when you fund your Skyetel account. No limits on number of simultaneous calls. Quadruple data center redundancy. $25 monthly minimum spend required. Tutorial and sign up details are here.

The lynchpin of Incredible PBX 2020 and beyond is ClearlyIP components which bring management of FreePBX modules and SIP phone integration to a level never before available with any other Asterisk distribution. And now you can configure and reconfigure your new Incredible PBX phones from the convenience of the Incredible PBX GUI.

VitalPBX is perhaps the fastest-growing PBX offering based upon Asterisk with an installed presence in more than 100 countries worldwide. VitalPBX has generously provided a customized White Label version of Incredible PBX tailored for use with all Incredible PBX and VitalPBX custom applications. Follow this link for a free test drive!
 

Special Thanks to Vitelity. Vitelity is now Voyant Communications and has halted new registrations for the time being. Our special thanks to Vitelity for their unwavering financial support over many years and to the many Nerd Vittles readers who continue to enjoy the benefits of their service offerings. We will keep everyone posted on further developments.
 

Some Recent Nerd Vittles Articles of Interest…