Prometheus

What it's good for#

Prometheus is the part of a monitoring setup that nobody looks at. Every 15 seconds it asks your server and your apps how they're doing (how much memory and processor they use, how full the disks are, how much traffic goes in and out) and writes the answers down. Keep that up for a few weeks and you can answer questions like "when did the disk start filling up?" or "which app ate all the memory on Tuesday night?"

It's paired with Grafana, which turns those numbers into dashboards. Grafana won't install until Prometheus is there, because it would have nothing to show. If you only want to know whether each app is up, Uptime Kuma is the simpler tool; Prometheus and Grafana are for how well things are running, over time.

What it collects#

Out of the box, with nothing to switch on, Prometheus collects three things:

• The server itself: processor, memory, disks and network. A small companion called node-exporter reads these.
• Every container: how much processor, memory and network each part of each app uses. Another companion, cAdvisor, reads these.
• Prometheus itself, so you can tell if it's struggling.

Adding your apps#

Some apps can report their own, more detailed numbers too: Traefik's requests and errors, for example, or AdGuard Home's and Unbound's DNS queries. Each of those has an Export metrics to Grafana switch in its settings, which can be turned on once Prometheus and Grafana are both installed.

Turn it on for an app and LibrePortal does the rest: it tells Prometheus to collect from that app and gives Grafana a dashboard for it. Turn it off and both go away again. Prometheus's Works with row says how many apps it's collecting from, and each app's own row says whether its metrics are being collected.

Looking at the data#

Grafana is the place to look. Prometheus's own Web Interface is handy for checking things work, though. Its Status → Targets page lists everything it collects from and whether each one answered the last time it asked. A target marked down usually means that app is stopped.

Good to know#

• It's private. Prometheus has no login, and its page shows every number it collects about your server and apps. So from 0.4.7 it installs Private, reachable only from your trusted places. Grafana reaches it over LibrePortal's internal network, so it keeps working whatever you choose here.
• Installed with 0.4.6 or earlier? Then it started Public, and an update doesn't change that. Open Prometheus's Network tab, choose Private under Who can reach it, and press Apply.
• It keeps 15 days. That's Prometheus's own default, and LibrePortal doesn't change it. Older data is dropped as new data comes in, so its disk use levels off rather than growing for ever.
• The companions look at the whole machine. node-exporter reads the host's filesystems and cAdvisor reads Docker's, both read-only. That's how they see the server rather than just their own container.
• Updates are automatic, and it's included in backups.