You always want to get the most out of your investment. Push the limits of what your server can do to get the most out of it. Asterisk gives you a great way to get a lot of calls through a single server, but sometimes people wonder why they’re hitting a limit that’s lower than others get. Sometimes it’s just a matter of bandwidth. Sometimes it’s inefficient codec usage. Overuse of an application with a flaw (such as MeetMe) can cause performance issues. Very often, it’s the Asterisk call recordings that are to blame. If you’re very, very careful, there are ways to do better with your recordings. If you’re not careful, disaster and disarray await.
Call Recordings are a recent favourite topic for this blog. Whether it’s mixing the recordings, or moving your recordings to Amazon, recordings are incredibly important to the call center, so we’re always thinking about them. Unfortunately, recordings can be a big performance hit on your system. There are a few reasons for that.
Asterisk writes your call recordings, by default, to the directory /var/spool/asterisk/monitor which is usually on a hard disk on the local machine. If you’re using a Cloud-based call center,such as something hosted on Amazon, you may have attached storage that works very similar to a hard drive. Unfortunately, the hard drive is usually pretty slow compared to the rest of the system. During the call, Asterisk may be recording each leg of the call as its own audio file. As the call continues, data is appended to each file. Once the call is over, Asterisk will mix the separate files into one recording, and delete the originals. Every time Asterisk has to hit the disk, things slow down.
A popular way to avoid the problems with disks and recordings is to use a memory-based drive. Often called RAM drives, there are a few ways to configure one, depending on the operating system and version. Setup and use is not terribly difficult, and the immediate performance gains can be quite good. So what do you need to set up a RAM drive in the first place?
- Adequate RAM. If your Asterisk server has 2 GB of RAM, a RAM disk is probably not for you. Asterisk will only use so much memory (usually less than 4GB), so if you’ve got more than 8 GB and no other memory-intensive services running, it’s usually safe to dedicate a few GB to a RAM disk.
- The ability to edit system config files to create the drive and mount it in the right spot.
- A location to move the files from afterwards.
- Serious file monitoring, and scripts that can watch for issues.
The first two are required to get the RAM drive running and in use. The latter two are required to actually have recordings later on.
Moving Your Recordings
A RAM disk is temporary and small. Sure, 8 GB may seem like a lot of space, but when you’ve got enough concurrent calls to require a RAM disk, it only holds a small part of your call volume for the day. You have to move those recordings to a larger location. You also need to make sure that recordings aren’t on the RAM disk when the server is rebooted. When the power goes off, the RAM disk is gone.
You’ll need to make sure that the next location for these recordings is big enough to hold them all. If you’re moving recordings to Amazon S3 or other remote storage, you also have to make sure that they’re put in a temporary holding area (or can be) if Amazon becomes unavailable, or other issues arise. Even if you’re using local network storage, enough things can go wrong that you have to have a plan B for files.
Monitoring and Scripting
When using a RAM disk, there are problems that can arise that may not be obvious right away, but will cause you to lose recordings if you don’t plan for them. These issues include:
- Asterisk crashing or being restarted. If Asterisk stops for any reason, any calls in progress stop (of course) and the recordings don’t get mixed. If your move recordings script is only looking for mixed recordings, you’ll miss these orphaned files. You need to make sure they get moved to permanent storage as soon as possible in order to avoid loss.
- Server Reboot. Sometimes you need to reboot your server (in the correct manner). You’ll need to make sure that there’s a cleanup script set to run on shutdown that moves everything from the RAM disk to permanent storage.
- Long Recordings. Sometimes a call may end, but Asterisk doesn’t receive proper notification and clean up all the channels. This happens most often when a call is in a MeetMe or some other application, and the channel is being recorded. It can also happen as a bug in a specific version of Asterisk. Despite the lack of actual audio, the raw channel recordings will continue to grow in length. After a few days, these can really impact available space. A script that looks for and kills these channels can save you a lot of headaches.
The Ultimate Risk
Let’s say you’ve set up your scripts to move files, check processes, find orphaned recordings, and handle cleanup on shutdown. There’s still one case that you haven’t handled: losing power to the server. There’s nothing you can really do to guarantee safety of recordings in this case. Using a UPS that sends a signal to the operating system can help. But we’ve seen cases where a tech at the colo simply turns off the power bar that the servers were plugged in to (?!!!). Immediate loss of power. Somebody could unplug the server. Or trip over a cord. If something happened like that, are you able to withstand the loss of recordings of in-progress calls? If so, then a RAM disk may be a good option for you. If not, then don’t even try.