Backups and restore

How it works#

Every night, LibrePortal takes a snapshot of each app: a copy of all its files and its database at that moment. It also takes one of its own settings, shown as Configs. The snapshots go to one or more backup locations, which can be a folder, a USB disk, another server or cloud storage.

Underneath, LibrePortal uses restic, a well-established backup program. That gives you three things without any setup:

• Everything is encrypted before it leaves the server, so a backup on a USB stick or in the cloud is useless to anyone without the password.
• Only changes are stored. A file that hasn't changed since last night isn't stored again, so thirty nightly snapshots take up far less than thirty copies.
• Every snapshot is checked. After each backup, LibrePortal restores the new snapshot into a scratch folder to prove it can actually be read back.

You'll find it all under Admin → Backups, and each app's own snapshots are on its Backups tab.

The password you must not lose#

Each backup location has its own repository password, which LibrePortal generates when the location is set up. It's what encrypts the snapshots, and it's the only thing that can decrypt them.

So save a copy somewhere other than the server, today:

Do it again whenever you add a location. The Configs snapshot also carries every location's password, so when you rebuild a server you only need to remember the one for the location you restore from; the rest come back with it.

Where your backups go#

A new install has one location, Local disk, which is the /libreportal-backups folder on the server (or wherever you pointed --backups-dir when you installed). That protects you from a bad update or a deleted file. It does not protect you from the server's disk failing, or the house burning down, because the backups are on the same machine.

So add a second location that isn't on the server. On the Locations tab, press Add location and pick a type:

New locations start switched off, so you can fill in the details before anything is written. Turn Enabled on when you're ready. From then on, every backup goes to every enabled location.

USB and external disks#

Use a disk formatted for Linux (ext4, xfs or btrfs). FAT, exFAT and NTFS can't keep the file ownership the backups need, and LibrePortal warns if you try.

If the disk is sometimes unplugged, switch on the location's Require Mounted Drive setting. LibrePortal then refuses to back up when the disk isn't there. It isn't in the Locations editor, so set it from a terminal, with the location's number in place of <n> (backup location list shows the numbers):

sudo -u libreportal libreportal config update 'CFG_BACKUP_LOC_<n>_REQUIRE_MOUNT=true'

Ransomware-safe locations#

A location can be made Append-only (it's under the location's advanced settings). LibrePortal can then add new snapshots but never delete old ones, so even someone who got into your server couldn't wipe your history. Old snapshots aren't tidied away on those locations; you do that yourself on the storage side.

Backups from all your servers in one place#

The Backup Server is a small program for a second machine, such as a cheap VPS, whose only job is to keep an encrypted, off-site copy of your backups that your installs can't change or delete. The full guide is The Backup Server; this is the short version.

If you run more than one LibrePortal, you can have them all send their backups to one place. That box runs the Backup Server app, which receives the backups, keeps each server's in its own protected space, and can copy them on to somewhere off-site. It's useful when one machine has a big disk, or sits in a different building, and you'd rather manage one place than several.

Installing the Backup Server#

On the machine that will receive the backups, run:

curl -fsSL https://get.libreportal.org/librebackup | bash

That installs the Backup Server and a librebackup command for managing it from a terminal. It updates itself every day at 12:30; you can change that, or update by hand, in Settings → Updates.

Connecting an install to a Backup Server#

The easiest way to link an install is with a one-time code. On the install, open Admin → Backups → Locations and press Connect to a Backup Server. It shows an address and a code. Then, on the Backup Server, go to Sources → Add a source → LibrePortal, paste the address and code, and confirm the fingerprint matches. The install is paired and the location is set up for you. The code works once.

What a Backup Server can reach#

A Backup Server is only ever allowed to read your backups. The account it uses is locked down to that one job: its keys are restricted, root owns the key files, the account has no extra groups, and the server's SSH rules allow nothing else. Each key only works from the address it connects from, and a Backup Server on your admin VPN is refused. It can't be tricked into reading outside the backed-up folder through links, and the WebUI's login secrets and logs are left out of everything it pulls.

When you pair two installs so one can pull apps from the other, the owner decides which apps. A paired install can pull only the apps you've allowed, and by default that's none.

sudo -u libreportal libreportal peer allow <peer> <app>
sudo -u libreportal libreportal peer disallow <peer> <app>     # or --all
sudo -u libreportal libreportal peer allowed [peer]

Peers that were already pulling before 0.9.11 keep working.

When backups run, and what's in them#

By default backups are Automatic: every night at 05:00, LibrePortal backs up every app that has Enable Backups switched on, plus the Configs snapshot. The dashboard shows when the next one is due. You can change the time on the Configuration tab (App Backup Schedule), or choose Manual, in which case nothing is backed up until you do it yourself.

Most apps have backups switched on when you install them; a few don't. Each app's page in this guide says which under "At a glance", and you can change it on the app's Config tab.

Configs is LibrePortal's own settings: your domains, your WebUI login, your backup locations and their passwords, everything you chose in the setup wizard. The dashboard says it plainly: back up Configs first, because it's needed to restore the rest. The WebUI itself isn't backed up as an app, because everything it needs is in Configs.

Does the app stop while it's backed up?#

Only when it has to. Apps whose databases LibrePortal knows how to copy safely while they run are backed up live, with no downtime. The rest are stopped, snapshotted and started again, so they're briefly unavailable in the middle of the night. That's the Automatic backup strategy, and it's the right choice for nearly everyone. You can override it per app with Backup Strategy on the app's Config tab.

How long snapshots are kept#

On the Configuration tab, Backup style decides how far back you can go:

Old snapshots are cleared out after each backup, to the plan the style sets. Three things are always kept, whatever the numbers say: the newest complete snapshot, the last one proved good by a read-back, and the snapshot an update or version step saved to roll back to, while it's still recent. That last one is Keep Rollback Points (days) on the Configuration tab, 30 days to start with, so you can always undo an update that only turned out wrong a week later.

A snapshot that failed its check counts for nothing, and if the newest snapshot is incomplete, or the last read-back failed, nothing is cleared out at all that night: a bad backup can never push a good one out. Every snapshot removed is noted in the backup log, saying which, when it was taken and from where.

Backing up by hand#

Before you do something risky to an app, take a snapshot first. Open the app, go to its Backups tab, and press Backup now.

To back up everything at once, use Backup all apps on the Backups tab of the backup centre, or from the command line:

sudo -u libreportal libreportal backup all

Getting an app back#

Say an update went wrong, or you deleted something inside an app and want yesterday's version.

LibrePortal checks the snapshot is really there and readable before it touches the app, then stops the app, clears its folder, puts the files and the database back, and checks each one went back whole before it starts the app again. If any part can't be restored, it stops and says "Restore of app FAILED", rather than starting the app on an empty folder and reporting success. You can watch each step on the Tasks page.

Moving to a new server#

The same backups can rebuild a whole server: after a dead disk, or when you move to a bigger machine. For this to work, the backups need to be somewhere the new machine can reach, like a USB disk you can plug into it or an SFTP or cloud location, and you need that location's password.

Backups before every update#

Whenever LibrePortal updates an app, automatically or because you pressed Update, it takes a snapshot first, to every enabled location, and checks it. If the update fails, it restores that snapshot and puts the previous version back. If the snapshot can't be taken, the update doesn't happen at all.

That means an app with no working backup location won't update. Keep at least one location enabled. Updates has more.

Exports aren't backups#

An app's Export button makes a single .lpapp file you can import on another LibrePortal. It's handy for moving one app between servers, but it isn't a backup: it has no history, isn't encrypted, and isn't made on a schedule.

When a backup fails#

A failed backup shows on the dashboard, and the previous snapshots are untouched. If you've set up notifications, LibrePortal also sends you a message: by default it sends one for every failure and nothing else.

A backup fails honestly. If an app's database can't be copied cleanly, the whole backup for that app fails and no snapshot is kept, rather than storing one with an empty or half-written database that would look fine until the day you needed it. Each database copy is checked as it's made, and again when the snapshot is read back.

A snapshot that was written but couldn't be read back when it was checked is marked Last backup could not be read back. Treat that one seriously: check the location's disk or connection, then take a fresh backup with Backup now.

You can also ask LibrePortal to read back a sample of what's stored and check it's intact:

sudo -u libreportal libreportal backup verify

From the command line#

Everything above works from a terminal on the server too.

Run each one as sudo -u libreportal libreportal <command>.