Redundancy, high-availability, backups and disaster recovery

Datacentres

Precedence has two datacentres in Cambridge, UK; primary and secondary. They are joined by a dedicated private fibre link. Specifications of the primary datacentre are covered here.

We also have datacentre space available to us in London (with a dedicated private link to Cambridge), plus as an IBM business partner, we can provision bare metal servers in any of their global datacentres.

Each virtual machine will have a home datacentre where it usually runs. This is obviously also true for physical dedicated servers.

Internet connectivity

Each Precedence datacentre has a minimum of one Internet feed, plus the ability to access the links at the other datacentre over the private link. This means that four Internet connections are available for use across both sites. The Internet connections are provided through different uptream providers, so that if one provider has problems the other connections will not be affected.

Precedence manages its own ranges of IP addresses; Precedence is a RIPE member and LIR. It manages its own AS number (AS209113). The owned IP addresses can be accessed from both datacentres and are managed with BGP (Border Gateway Protocol). This means that if an Internet connection goes down, the traffic will be automatically re-routed via another connection or even via a different datacentre.

Private LANs

Customers' private LANs are replicated across the sites. This means that customer resources can be running at both datacentres simultaneously and will consider themselves to be on the same Layer 2 network.

Hosting redundancy and high-availability

High-availability status showing N+2 capability

Customers' virtual machines are run on highly-available servers and storage. The network, host health and storage links are continuously monitored and, if a host fails, virtual machines will automatically restarted on another available host. Hypervisors run in a minimum of N+1 (N+2 where possible) meaning that one or two physical server failures can be tolerated.

All physical hypervisor hosts are linked via multiple paths to back-end storage (exceptions to this may include hosts dedicated to a single customer). The storage is managed with active/passive redundant controllers with all storage components having dual-port connections. This means that storage maintenance and upgrades can be performed with no downtime.

Data backups

Data on managed virtual machines (including any MySQL databases) are backed up using our integrated Remote Safeguard backup system. This backs up changes in your data nightly and keeps 30 days of changes. The standard retention period of 30 days is included at zero cost (longer periods can be chosen at extra cost). This data backup service is also included for managed physical co-located servers.

The Remote Safeguard backup system holds the main copy of the data at our primary data centre. This is the copy that you may access yourself using rsync-over-ssh, scp or sftp. The backed-up data is replicated nightly to our secondary data to storage that is not connected to or accessible from the Internet.

Unmanaged virtual machines are not backed up in this fashion (as we have no access to their storage). A Remote Safeguard contract may be purchased for this purpose (can be shared by multiple VMs).

Virtual machine snapshots and backups

Snapshots of virtual machines (both managed and unmanaged) are taken weekly and are replicated to the other datacentre. The backups are staggered throughout the week to spread the load (so could be up to 3 days apart). If you have virtual machines that should have snapshots taken close to each other in time, please contact Precedence support to arrange this otherwise they wlil be assigned randomly. Two weekly snapshots are included in the hosting cost (i.e. you may go back up to two weeks). Further or more frequent copies are available as an optional extra.

A snapshot copy of a VM can be cloned and then started. Iif the original VM is still running, the copy would need to be connected to a separate private customer LAN before starting (to avoid a name and/or IP address clash). Alternatively, its virtual hard disc(s) could be attached to another existing virtual machine allowing the backed-up data to be easily accessed, compared and restored. These are manual operations that may involve raising a ticket with Precedence support.

The snapshot copy can be run in its home datacentre using the backup storage at the other datacentre (this will have disk throughput and IOPs penalty, but will allow it to be started quickly). The running copy can be migrated back to its home datacentre while it is running with no further downtime.

Disaster Recovery

The snapshots of virtual machines (as described above) are designed to run in their home datacentre (even though the data resides in the alternative datacentre). In the event of the home datacentre being entirely unavailable, they would need to be linked to the alternative datacentre. This would not be immediate. Moreover, as the snapshots are generally taken weekly, any data they contain may be up to a week out of date (but there may also be nightly backups from Remote Safeguard).

Disaster Recovery uses frequent or continual replication to maintain a second runnable copy of a virtual machine in the alternative datacentre. Pricing is available upon request.

Please note that Disaster Recovery and backups are different things. Disaster Recovery allows the current state of your machines to be accessed quickly in multiple locations. If the data is corrupt, replication will ensure it will be corrupt at all locations. Backups allow recovery of previous versions of the data, even if this is not instantaneous.