Self-hosted Linux and Docker infrastructure

A personal homelab running a set of containerised services on hardened Ubuntu. It is both the infrastructure behind the services I use daily and a place to test configurations, hardening, and recovery procedures before those lessons reach client work.

Encrypted storage with ZFS

Data lives on ZFS datasets encrypted at rest. A dedicated boot-time service imports the pools, loads the encryption keys, and mounts the datasets before anything that depends on them is allowed to start — Docker, sync services, and the containers themselves. Snapshots and zfs send / receive replication handle backup and point-in-time recovery without file-level copying.

Server and network

Hardening includes key-based SSH only, fail2ban with custom filters for each self-hosted application, firewall rules, and unattended security updates. Services run under restricted users where the software allows it.

Caddy sits in front of every service, discovers containers by label, and issues and renews TLS certificates automatically. All HTTP is redirected to HTTPS, and each service is reachable under its own subdomain without exposing container ports.

Containers and configuration

Services are isolated in Docker containers and managed with Compose. Data volumes are kept separate from container images, so an upgrade does not touch persistent data.

The systemd units, helper scripts, fail2ban rules, and Compose files that hold the host together live in their own git repository. Stock package-shipped configuration is deliberately not tracked, so the repository stays a record of what is custom rather than a copy of the operating system.

Backups

A systemd timer snapshots the encrypted dataset, replicates incrementally to a mirror pool, and prunes old snapshots on a retention schedule. Each step logs its outcome. A backup that fails silently is worse than one that does not exist, so the run is designed to make failure visible.

The lesson that changed the mounting service

Adding standard systemd hardening to the unit that mounts the encrypted datasets looked safe and reported success. Because that hardening runs the unit in a private mount namespace, the mounts were only visible inside the namespace and vanished when the process exited. Everything downstream saw an empty directory.

The fix was straightforward; the safeguard is the useful part. The mounting service now writes a sentinel file at the root of each dataset it mounts, and every dependent service checks for that file before starting instead of trusting that another unit’s success means the resource is visible. A longer account of the failure is in the notes.