diff --git a/container/site/content/posts/2026-07-07-home-ops/index.md b/container/site/content/posts/2026-07-07-home-ops/index.md new file mode 100644 index 0000000..64ff180 --- /dev/null +++ b/container/site/content/posts/2026-07-07-home-ops/index.md @@ -0,0 +1,135 @@ ++++ +categories = ["build","software"] +date = 2026-07-07T10:00:00-05:00 +description = "" +draft = false +slug = "2026-07-07-home-ops" +title = "🏗️ Home Ops Upgrades" +author = "nicholas" ++++ + +As the number of self-hosted services I use regularly has grown, managing them by hand began to feel wrong, and occasionally annoying and tedious. It ends now! + +{{< image + src="images/portainer-container-dashboard.png" + caption="Container dashboard - many apps" >}} + +In the beginning, my workflow was pretty sloppy since much of the deployment/management depended on me remembering to restart a container or update a version, manually copy a secret here and there. Things like this. Totally maintainable, sustainable with some effort, but it is not *the way*. And to some degree these manual interventions will always be necessary, but it will serve me well to reduce them and automate them away as much as reasonable to help these apps live far into the future with little fuss. Many of these apps have proven their usefulness to me over many years, so I have begun to take their management more seriously, slowly taking steps toward git-ops patterns and best practices. Before: scattered `.env` files. Flat directory of compose files. No documentation. Manually copying secrets by hand. Manually rebuilding host dependencies by hand. Now: a single command issued by a single click of a button (maybe I am overstating the simplicity here...nevertheless) will handle mostly everything I care about. Host setup, Compose projects, rendered secrets, state directories, validation, and deployment are all described in one place. Easy! + +## Git Repo as Source of Truth + +I am trying to move my app hosting toward git-ops patterns, where the git repo is the "source of truth" which describes the desired state of my home apps/services etc. The repo checkout itself should stay disposable. This will be useful to me because changes become reviewable and repeatable. If I move a service to a different host, for example, or a port changes, or a secret is added, or a stack is disabled, that change will be obvious in git logs, and it will be easily deployable since everything is in one place. + +Prior to this overhaul, I had one repo for each host. I decided to combine all host config to a single repo for simplicity, since managing the deployment logic across two and possibly many more repos would sort of defeat the original purpose of making things simpler and more robust. Configuration would drift apart, and all of my annoyances under the previous 'workflow' would return. + +## Ansible + +The main tool making the magic happen is Ansible. This is sort of the 'infrastructure-as-code' layer, configuring the OS host environment in all the ways necessary to run the apps. It installs packages, configures Docker, creates state directories, manages the deploy user, mounts storage, renders secrets, and starts containers, stopping short of actually provisioning the VM/host itself (Maybe coming soon). + +{{< image + src="images/actions-deploy.png" + caption="Gitea Actions deploy workflow" >}} + + +### Inventory + +I used Ansible inventory to map and name the hosts, set the connection details, and group machines so the same playbooks behave differently for each host. In the new `home-ops` repo, inventory lives in `ansible/inventories/production/`. Shared defaults live in `group_vars/all.yml`, host-specific settings live in, for example `group_vars/edge.yml`,`group_vars/server.yml`. This is how Ansible can decide which Compose projects run on the `edge` host and which run on the `server` host. + +### Roles + +I am using roles to keep the playbooks organized. There is some reuse between each playbook since the host env is basically the same. The playbook will define the order of operations, while the roles will be the steps of host setup: + +- related work stays together +- shared setup can be reused across host groups +- `edge`-only and `server`-only behavior stay separated +- common tasks do not need to be copied between playbooks + +In this repo: +- `common` prepares the basic host environment +- `docker` installs Docker and Compose +- `state` creates runtime state directories +- `storage-mounts` manages NAS mounts +- `edge-host` handles edge-specific host setup +- `step-ca` syncs private CA trust +- `preflight` checks required rendered files +- `compose-projects` runs the enabled Compose apps + +### Compose Projects + +Each app "stack" is a Compose project under `stacks/apps/` and `stacks/edge/`. A stack can include more than one container, since many apps depend on supporting services/containers, e.g. web container, a database, a cache, a worker. + +### State Directories + +Ansible creates `state_root` and app-specific state directories. Compose uses `${STATE_ROOT:-/opt/home-ops-state}`. + +State lives outside the repo because as I mention above, the repo checkout should stay disposable. A deploy can replace or update `/opt/home-ops` without deleting databases, uploads, generated config, caches, or other runtime data. The repo describes the desired configuration; `/opt/home-ops-state` holds the mutable state created by running services. + +### Deployment Paths + +Production deployments can run from Gitea Actions or manually from a control machine. A control machine is a machine running Ansible which connects to the target hosts over SSH and applies the playbooks there. Manual Ansible runs are useful for development, testing, and recovery when the Gitea workflow is unavailable. For example, from a prepared control machine: + +```sh +ansible-playbook playbooks/server.yml +``` + +But the simplest/easiest way to deploy is to click a button in a web UI. My `Deploy` workflow supports `all`, `edge`, and `server` targets. + + +{{< image + src="images/deploy-workflow.png" + caption="Deploy workflow in Gitea" >}} + +### Data + +Large data which ought to live on my NAS are bind-mounted under `/mnt/data` and `/mnt/backup`. Ansible can create mount point directories, but NAS ownership and ACLs are managed on the NAS side. The point is to keep large media and backup data out of the repo host filesystems. They take up a lot of space and do not belong on the fast, limited storage on container hosts. + +Some services use Docker volumes for persistent state, for convenience, since I am happy to let Docker manage the data and do not much care about these details. For long-lived or state that I care about, I want to use `/opt/home-ops-state` which is easier to reason about because it has an explicit path and can neatly be included in a backup/restore operation. + +### Secrets + +Secrets are stored in SOPS and rendered before Compose starts. Plaintext `.env` files, private keys, and decrypted secrets are not committed to the repo. Keeping encrypted secrets in the repo is useful because the secret manifest can live next to the configuration that needs it. I can see that a service requires a `.env` file or mounted secret file without committing the plaintext values. Secret changes also get version history like any other infra change and the actual secret values remain encrypted. This is much preferred to keeping secrets as random files on one host and manually copying them onto others. The repo can describe which secret files must exist, where they should be rendered, and which services consume them. SOPS handles the encryption and Ansible handles rendering the files during deploy. + +#### App Env Secrets + +Most app secrets are rendered from the same SOPS `secret_files` list. Ansible decrypts the SOPS file on the control machine, then writes each secret file to its destination before Compose starts. The app secrets mentioned are mostly in the form of a per-service `.env` files that live next to the app's `compose.yaml`. Compose reads that file and uses the values for environment variables. A couple of other apps use the same idea, but instead require app-specific secret files rather than `.env` files. These are still SOPS-rendered secrets, but the destination is a mounted runtime file rather than a `.env` file. + +#### Deploy Secrets + +Deploy secrets are not consumed by the apps themselves but instead are used by the deployment system so Gitea Actions or a control machine can connect to the hosts and render production secrets. + +For example, the Gitea deploy workflow uses secrets: +- `DEPLOY_SSH_KEY`: allows the action runner to SSH to the inventory hosts as the `deploy` user +- `SOPS_AGE_KEY`: decrypts `secrets/sops/production.sops.yml` + + +{{< image + src="images/actions-secrets.png" + caption="Gitea Actions secrets" >}} + + +This separation is useful because deploy credentials have a different lifecycle from app credentials. Rotating a deploy key should not require changing app `.env` files, and changing an app password should not affect the deployment path. + +### Deploy User + +The `deploy` user is used for production Ansible runs and gets passwordless sudo through a dedicated sudoers file. This gives the ansible automations its own dedicated identity instead of using my personal login. There is a playbook for creating the user, installing the committed public key, and granting passwordless sudo. + + +## Renovate + +Renovate watches my Docker images and opens PRs when updates are available, but only if I select the proposed update in the Dependency Dashboard. This dashboard is a Gitea Issue the `renovate-bot` user creates and manages, and it is used by Renovate to decide which images should be updated by way of a pull request. Since I do not want every Docker image involved yet I limit Renovate to only the Compose stacks in my whitelist. + +{{< image + src="images/renovate.png" + caption="Renovate dependency dashboard" >}} + +The Renovate workflow can run on a schedule or manually from Gitea Actions. Renovate proposes updates which I can review and deploy manually (or automatically if I want, I suppose). It is highly configurable, but I have kept my own configuration simple and mostly manual for now. + +This workflow requires some secrets: +- `RENOVATE_TOKEN` Gitea access token so Renovate can create issues and PRs. +- `RENOVATE_GITHUB_COM_TOKEN` used by Renovate to access GitHub-hosted release notes and metadata. + +## Future Changes + +Future changes will mostly be about making hidden state less hidden, which means moving important Docker volumes into explicit state paths. I may also revisit secrets management approach later if I want to experiment with using a secret manager e.g. HashiCorp Vault. It is still possible to go a layer deeper than the apps, into the true infrastructure layer with Terraform or OpenTofu. Lots of possibilities, but for now, it is good enough. + +✅ Done. \ No newline at end of file