I’m currently looking into possible failover strategies. Of course, you should always try to make sure you scan your error_log files for possible problems first and get rid of PHP bugs and what not before you move on to ugly workarounds


First Things First: APC and PHP Causing Trouble

PHP has some pretty ugly bugs. You will eventually run into some problems with PHP, especially if you run PHP alongside APC. For example a corruption issue or other nasty surprises caused by PHP will show up eventually. PHP can quickly cause your server to overload.

Best practice tells us that you should first update and re-compile PHP, but sometimes the issue is another. For some reasons, my PHP.ini file was completely reset and my apc configuration was completely removed. This eventually caused various glibc errors, including invalid pointer errors and double free or corruption issues.

Make Sure APC Is Configured Correctly And Has Enough Memory

This is a good configuration to begin with. However, you need to tweak it depending on your server and resources. Set apc.enable_cli to 0 if you want to avoid trouble, it’s only for dev servers.

apc.enabled=1
apc.shm_segments=1
apc.shm_size=2G
apc.ttl=3600
apc.user_ttl=7200
apc.gc_ttl=3600
apc.max_file_size=2M
apc.stat=1
apc.enable_cli=0

Using Control Groups

One failover strategy can be to use control groups to limit resource usage

Restart Apache If Server Load Hits A Certain Number

One ugly workaround to keep your server from overloading and crashing is to have a cronjob that will automatically shut down certain services whenever your server load is higher than X.

Proc Load Average.png

Let’s see how this works.

mkdir /root/scripts
cd /root/scripts
pico restart.sh

Copy and paste this:


#!/bin/sh
check=`cat /proc/loadavg | sed 's/\./ /' | awk '{print $1}'`
if [ $check -gt 10 ]
#load average 5 min
then
/etc/init.d/httpd restart
fi

Next we’re adding a cronjob:

export EDITOR="pico"
crontab -e

Now copy and paste this into the crontab file: */5 * * * * root /root/scripts/restart.sh >/dev/null 2>&1

DNS Failover

If you have a backup server you can use that server for emergencies – simply prepare a 1:1 copy and synchronize the servers using rsync every day

Now follow this tutorial to set up a failover using DnsMadeEasy:
DNS Failover Guide