Installation on hosts without SSH access

August 23rd, 2007

I’ve been hearing good things about HSphere as a platform for a reseller account – clustered services, fully integrated billing and ticketing and the ability to resell both Windows and Linux hosting (not that I really want to offer Windows but it could be useful at some stage). But there’s a catch: I’ve yet to find a HSphere provider who’s willing to offer SSH access. This is rarely a problem for CPanel or DirectAdmin – most hosts using these panels seem to offer it on request, presumably having accepted that some users have a legitimate need for it and that any additional security risks are minimal (above the risks associated with allowing users to run PHP, perl and shell scripts via CGI or cron). But perhaps the situation is different for HSphere – I read somewhere that SSH is used for internal communications within the cluster and allowing user SSH could compromise this. In any event, the reality seems to be, if you want SSH access don’t choose an HSphere reseller account.

Now I’ve always considered SSH access a prerequisite for ClonePanel. The backup server uses an SSH connection to transfer files using rsync. But could it be done any other way? I thought this could be an interesting challenge (masochist that I am) so I signed up for a reseller account with a popular HSphere host and set out to discover what was possible.

Of course there are always other ways to do backup and restore. Most common is to tar and zip all files into a single backup, transfer that using ftp and unzip and untar at the other end. But I really want to avoid this if I can – rsync is just so much more efficient and I don’t want these regular backups to cause any perceptible load on the servers. So I started looking at how to use rsync without being able to make an SSH connection to the hosting server. If I can’t connect in, what about having the server connect out? I can run shell scripts through cron, the system gives me access to the usual linux / bash commands including ssh and rsync, so what’s to stop me connecting from the hosting server to my backup VPS?

Well, as it turns out the first thing stopping me is a firewall! They have it sewn up pretty tight, including outgoing connections on several high-numbered ports that I tried – it makes sense from the security point of view (don’t open up anything unless you have to) but it makes things tricky for me. But it turns out that there are at least two ports open, 80 and 443, allowing scripts on the server to connect to external web sites – this seems like a requirement for any hosting server where the clients may need to fetch information from the web or communicate with a payment gateway.

So if I set up the VPS to run sshd on one of these ports, the hosting server can connect to it. Of course this means that it’s not possible to run a regular webserver on the VPS (on the standard ports) but I don’t want it to be web-accessible anyway. So I set up the ssh private key and known_hosts file on the hosting server (normally ssh will update known_hosts with the other server’s RSA key on the first interactive connection, after prompting to ask whether you want to connect, but in this case it must be done manually – there is no interactive connection!) and after a few attempts managed a simple ‘ls’ command on the backup server.

So it is possible: an outbound SSH connection from a hosting server where inbound SSH isn’t allowed. The use of a non-standard port is a little inconvenient but not a major problem. However we are talking about passwordless remote access to the backup server from a web-server… This is something I’ve always tried to avoid – the webserver is relatively insecure (ie. accessible to anyone who wants to connect to it, and running scripts which, even if kept updated, just might contain a zero-day exploit). This is a tough call – something I’d prefer not to do but against that I really would like to extend the functionality of ClonePanel to hosting servers without SSH access.

My compromise is this: a separate install of ClonePanel under a different user, with the new install used only for accounts connecting to this hosting server. Still not ideal perhaps but at least all other accounts are kept totally separate by file permissions on each user’s home directory. In the worst-case scenario where the hosting server gets hacked the attacker could use this connection to the backup VPS to read anything from the user’s filesystem, but this is mostly what’s already available to him in the web-directories of the compromised server, and the open-source ClonePanel system itself. There is one potential issue: the private keys allowing access to the corresponding accounts on other servers, but since these keys are restricted to use from the ClonePanel system’s IP address even that doesn’t allow the attack to go any further.

So that’s how it works. All that remains is to set up a simple rsync transfer script on the hosting server similar to the sync script on the ClonePanel system and run it at intervals with a cron job. Here’s what I ended up with:

#!/bin/sh
# This program is intended to sync a slave server to the backup
# where SSH connections to the slave are not allowed.
# The program runs on the slave server (via cron)
# and connects from the slave to the backup server to sync the files.

# After syncing the databases are restored.

# Several constant values need to be filled in below.
# ClonePanel - Manages duplicate accounts on two or more webservers,
# including snapshot backups, monitoring and failover dns.
# Copyright (C)2006 Chris Cheers, Internet Lynx.
# Contact chris[at]clonepanel[dot]com.
# Internet Lynx, PO Box 7117, Mannering Park, NSW 2259, Australia
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.

exec 2>&1
# Display errors

set -u
# Be strict about variable declaration

unset PATH
#avoid use of $PATH - limit script to system commands we choose

#start of constant definitions
RSYNC=/usr/bin/rsync
SSH=/usr/bin/ssh
MYSQL=/usr/bin/mysql
#most likely won't need to change these

REMOTEUSER=username
REMOTEHOST=backup.clonepanel.example.com
REMOTEPORT=443
REMOTEWEB=path/to/public_html
REMOTEDB=path/to/cp0.30_db
#details for backup server - change as required

AUTH=/hsphere/local/home/username/.ssh/private.key
#private key file giving access for REMOTEUSER to REMOTEHOST

HOMEDB=/hsphere/local/home/username/cp_db
HOMEWEB=/hsphere/local/home/username/example.com
#directories for local server - change as required

EXCLUDES=excludes.txt
#name of file containing filenames to exclude - one per line
#filenames of all files modified to suit this server

NUM_DATABASES=2
DATABASES=(username_db1 username_db2)
DATABASE_USERS=(username_dbuser1 username_dbuser2)
DATABASE_PASSWORDS=(secret1 secret2)
DATABASE_HOSTS=(dbhost1 dbhost2)
DATABASE_FILES=(username_db1 username_db2)

#end of constant definitions (stop editing here!)

#sync from backups

$RSYNC -avz --delete --exclude-from="$EXCLUDES" -e "$SSH -p $REMOTEPORT -i $AUTH" $REMOTEUSER@$REMOTEHOST:$REMOTEWEB/* $HOMEWEB/

#sync web files

$RSYNC -avz -e "$SSH -p $REMOTEPORT -i $AUTH" $REMOTEUSER@$REMOTEHOST:$REMOTEDB/* $HOMEDB/

#sync database dump files

#restore databases
for (( i = 0 ; i < $NUM_DATABASES ; i++ ))
do
  db=${DATABASES[$i]}
  db_user=${DATABASE_USERS[$i]}
  db_pass=${DATABASE_PASSWORDS[$i]}
  db_host=${DATABASE_HOSTS[$i]}
  db_file=${DATABASE_FILES[$i]}
  $MYSQL -u$db_user -h$db_host -p$db_pass $db <$HOMEDB/$db_file.sql
done

Conclusion: It seems to be possible, with some minor compromises. With more development the process could be better integrated into ClonePanel – perhaps using a similar technique to transfer a directory of special scripts which could then be run by the ClonePanel system as cgi.

More to follow as I find the time…

CPAN modules to load

August 13th, 2006

Based on a standard Centos 4.3 distribution, the backup and sync functions of ClonePanel work immediately. However for manipulation of zone files and connecting to hosts to update nameservers using the WHM remote access key the following perl modules also need to be installed:

  • Net::DNS
  • Net::DNS::ZoneFile::Fast
  • HTML::TreeBuilder

The best way to do this, ensuring that all dependencies are satisfied, is to use CPAN – as root, type cpan and follow the prompts. Once cpan itself is installed and you have the cpan> prompt, the command “install modulename” will take care of everything!

Edit: This also applies to a Cygwin install, although in this case I had some tests fail that required a “force install Net::DNS::SEC” (a dependency of Net::DNS::ZoneFile::Fast).

0. Prerequisites

July 28th, 2006

Main system

ClonePanel is written for *nix systems, and uses several of their standard features (most notably rsync and hard links). I use Centos 4 but other linux distributions should run with little or no modification; some tweaking may be required for *bsd systems. In principle it should also run on MS Windows using Cygwin (untested so far, but I think rsync should work and apparently hard links are supported on NTFS file systems). If you find changes are needed please let me know what you do so I can improve the program for everyone.

Edit: With some small reservations it does seem to work on Windows using Cygwin (more details).

The main installation to run backups, monitoring and dns control requires a low-powered computer on a DSL connection with a static IP address*. If you prefer to run this system in a datacenter then a small VPS would be suitable. I suggest having disk space available at least 2 x that which you want to back-up.

For security I recommend NOT running any other services on the same system (web server, mail server, ftp server etc.). Access keys stored on the ClonePanel system would allow anyone with access to it to also access all connected systems.

As an example, a P3-500 with 128Mb memory on a 512/128k ADSL connection works without any problems and runs at very low load. So dig out that old PC, download the latest Centos distribution, choose a minimal install and you’re ready to go!

Web Hosts

ClonePanel backup and monitoring works with hosting ranging from dedicated server down to shared hosting plans – really the only requirement is that the host permits user-level SSH (secure shell) access. Many hosts will do this on request, even if they don’t advertise it, particularly if you explain that it is for taking backups using rsync. CPanel hosts usually use “jailshell” – a slightly limited version of the standard bash shell – this works just as well.

I’m listing some hosts I’ve used with brief reviews in the hosting category.

If you want to use ClonePanel to control DNS records (edit zone files) then you will need WHM access to a CPanel server (a reseller account) or a VPS / dedicated server that allows you to change the nameserver setup.

* Strictly speaking a static IP address is required only because it’s used as an additional security feature in the remote-access setup (the key will only work from a single IP address). If you really want to run on a dynamic IP then a small change to setup_remote_key will permit this for you.

1. Download and extract

July 28th, 2006

2. Run install script

July 28th, 2006

3. Set up hosts

July 28th, 2006

4. Set up accounts

July 28th, 2006

5. Set up databases

July 28th, 2006