ClonePanel

The first version to be made publicly-available – if you’re interested in being a guinea pig alpha tester please be sure to let me know what you think – you can drop me an e-mail or add a comment here.

Thanks,
Chris

Latest version should always be at: clonepanel.com/latest/

This version is at: clonepanel.com/downloads/clonepanel_0.30.tar.gz

Please see the installation instructions.

Main system

ClonePanel is written for *nix systems, and uses several of their standard features (most notably rsync and hard links). I use Centos 4 but other linux distributions should run with little or no modification; some tweaking may be required for *bsd systems. In principle it should also run on MS Windows using Cygwin (untested so far, but I think rsync should work and apparently hard links are supported on NTFS file systems). If you find changes are needed please let me know what you do so I can improve the program for everyone.

Edit: With some small reservations it does seem to work on Windows using Cygwin (more details).

The main installation to run backups, monitoring and dns control requires a low-powered computer on a DSL connection with a static IP address*. If you prefer to run this system in a datacenter then a small VPS would be suitable. I suggest having disk space available at least 2 x that which you want to back-up.

For security I recommend NOT running any other services on the same system (web server, mail server, ftp server etc.). Access keys stored on the ClonePanel system would allow anyone with access to it to also access all connected systems.

As an example, a P3-500 with 128Mb memory on a 512/128k ADSL connection works without any problems and runs at very low load. So dig out that old PC, download the latest Centos distribution, choose a minimal install and you’re ready to go!

Web Hosts

ClonePanel backup and monitoring works with hosting ranging from dedicated server down to shared hosting plans – really the only requirement is that the host permits user-level SSH (secure shell) access. Many hosts will do this on request, even if they don’t advertise it, particularly if you explain that it is for taking backups using rsync. CPanel hosts usually use “jailshell” – a slightly limited version of the standard bash shell – this works just as well.

I’m listing some hosts I’ve used with brief reviews in the hosting category.

If you want to use ClonePanel to control DNS records (edit zone files) then you will need WHM access to a CPanel server (a reseller account) or a VPS / dedicated server that allows you to change the nameserver setup.

* Strictly speaking a static IP address is required only because it’s used as an additional security feature in the remote-access setup (the key will only work from a single IP address). If you really want to run on a dynamic IP then a small change to setup_remote_key will permit this for you.

The latest version of the program will be found at clonepanel.com/latest/

Download using wget (or by any other method) and extract the files:

By default the archive unpacks into a directory “cp0.30″ (for the 0.30 version). You may want to rename this or create a symbolic link to it – I use a symlink called “cp”.

Change to the scripts directory and run the “install” script.

The install script prompts for some basic information (on initial install you can probably accept the defaults) and writes them to a config file. Then it runs another script (“./host add”) to store information about the machine it’s running on. You need to enter accurate information at these prompts:

Server numeric IP address – On a dsl connection, this should be the static IP given to your connection (if you don’t know this, visit whatismyip.com) . On a VPS or dedicated server this should be the main IP assigned to your server – the one used by the server when it connects to others.

Server A record – The IP address you use to access this computer. On a dsl connection this will be the address on the local network, on a VPS or dedicated server again the main server IP.

Nameserver IP address – Normally this will be blank, but if you are running a nameserver on this computer then enter its assigned IP address here.

Template for home directory – Most linux servers use the form /home/username for user directories. Unless you have a very unusual system just hit Enter to accept the default.
Monitor this host – Since this is the backup system, and will be monitoring other hosts, you normally don’t want it to be monitored itself – press n

Upload monitor data to this host – Since this is the backup system, and will be monitoring other hosts, it will have the monitoring data already – press n

The ClonePanel system is now set up, but before it can do anything useful you need to enter information about the host systems it will connect to and the accounts on them.

This is done by entering the commands "./server add" and "./account add", and following the prompts.

Here we’ve specified a host server (it may be a shared or reseller account, VPS or dedicated server). Since we answered "y" to the monitoring question there was some more information needed – the IP, domain name and path to the monitor program. More about monitoring later.

At this point if you have other hosts (for example, if you want to mirror a web site onto a second server) you can define them in the same way. Or if you prefer this can be done later.

Next we set up an account for backup / mirroring. The account must be hosted on one of the hosts already defined.

Important warning: This setup creates a remote-access key for each account, allowing login without password. If anyone were to get access to your ClonePanel account they would also be able to read and write to all the remote accounts set up.

So, if you still want to go ahead (!) enter "./account add" and follow the prompts:

If all goes as it should (indicated by the messages in green above) the account will be set up and ready to take backups. If not, first check that you have SSH access to the server you’re setting up and that you’re using the correct username and password. Then check the error code against the list in scripts/error_codes. If you have problems please post them here so that others can also see the answers.

If your web sites are all static HTML then you can skip this section, but modern sites often have at least some database-driven content and many are totally dynamic. For such sites backing up the files would be futile – most of the important data is in a database.

Here we set up ClonePanel with the information needed to backup all databases.

Important – ClonePanel stores a username and password able to access each database. CPanel permits you to use the main account username and password, or even the reseller password to access databases – YOU MUST NOT USE THESE HERE. ALWAYS CREATE A DATABASE USER TO ACCESS THE DATABASE(S) AND ENTER THAT INFORMATION HERE.

This is because the information collected here will be stored on the remote account, because that’s where the database will be dumped or restored. As such the way the database login is stored is no more of a risk than using any application’s standard config file – which also must never contain the account password for the same reason. A future version of ClonePanel could perhaps use remote access to MySQL via a secure tunnel.

If you’re following this in sequence then it will come as no surprise that the command to add a database is "./database add".

After adding a database you must sync the ClonePanel data with the remote account(s). This can be done manually with "./sync_remote" or by answering "y" when prompted. If you want to add more than one database then you only need to sync data after defining the last one.

At this point (finally!) the basic setup is done and we’re ready to back-up the web site.

When synchronizing content across two or more servers one of the trickier aspects is handling user updates – forum posts, comments etc. If updates are accepted on all servers then merging the data would be a regular headache.

Instead, ClonePanel requires that each server be given a role – slave or master – and updates made on the master are copied to the slave. If any changes were made on the slave they would be overwritten and lost at the next sync. So it’s essential that the master server should handle all user updates and the slave must always be read-only.

WordPress handles this perfectly through the two setup options: WordPress address and Blog address.

The blog address is used throughout the site for links to posts, categories and pages – this can be blank to ensure that a visitor browsing the site (master or slave) will stay on the same site / server he first arrived at.

The WordPress address is used in links to admin functions (Register, Login, Dashboard) and in the action of forms – all the things that change the site in any way. So here we use the fully-qualified uri of the master server and ensure that all changes are made only there.

This site itself is an example. It’s available at clonepanel.com and clonepanel.net – one hosted in Australia the other in California (you may notice a slight speed difference, depending on your own location). You can browse either site without problems but as soon as you try to register, log-on or leave a comment you will find yourself on clonepanel.com – the master server.

The next step is to introduce failover – if the master server goes offline, how can the slave take over? That’s more difficult, and will be the subject of many future posts…

I was updating the old web site to the latest version of Xaraya (1.1) and getting frustrated with some of the changes that demand modification of existing themes.

I’ve also been working on a couple of WordPress sites recently and loved the tiny code-base and the simplicity of integrating my own php code into a WordPress site.

Don’t get me wrong – I like Xaraya but it’s a huge beast! And if I’m going to be developing and maintaining other WordPress sites it makes more sense for me to concentrate on that rather than learning how to do the same things through Xaraya…

So I deleted Xaraya, downloaded WordPress and a neat theme from themes.wordpress.net and a couple of hours later here it is – the new WordPress-powered ClonePanel.

ClonePanel is a software toolkit for backup and synchronization, monitoring and dns control with the ultimate aim of making it possible to operate a web site on two or more separate hosting accounts, giving you:

Fully redundant DNS service

Use the nameservers of both hosting services to provide DNS information about your domain.

Fully redundant mail service

Use one account as your primary mail server, the other as your secondary. Any mail collected on the secondary is transferred at regular intervals to the primary.

Your own multiple snapshot-type backups

Regularly downloading full backups from a hosting control panel is tedious and wasteful. Using the standard linux system tools (notably rsync, ssh, gzip and hard links) you can:

• Download over a secure connection with data compression
• Transfer only what’s changed (only the modified part of changed files)
• Keep multiple copies of your complete web site using disk space that’s only the total size of the site plus changed files.

Hot-spare server

A spare hosting account complete with your web sites, kept up to date and ready to go whenever it’s needed.

DNS Fail-over or round-robin load sharing

Fail-over means that your primary server will handle all requests under normal circumstances, but if it fails DNS is switched automatically to the backup. Round-robin uses multiple A records to the main web site so that visitors are shared between the servers at all times. You will need to decide which is suitable for your site, and what changes are needed to implement them.