<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>homelab</title><link>https://docs.magelssen.io/decisions/</link><description>Recent content on homelab</description><generator>Hugo</generator><language>en</language><atom:link href="https://docs.magelssen.io/decisions/index.xml" rel="self" type="application/rss+xml"/><item><title/><link>https://docs.magelssen.io/decisions/0001-bare-metal-kubernetes-nodes/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0001-bare-metal-kubernetes-nodes/</guid><description>&lt;h1 id="kubernetes-on-bare-metal-nodes-not-vms"&gt;Kubernetes on bare-metal nodes, not VMs&lt;a class="anchor" href="#kubernetes-on-bare-metal-nodes-not-vms" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;The cluster needed nodes. Proxmox was already there to run VMs, so the nodes could be VMs on&#10;it, or separate physical machines. The lab is for learning Kubernetes the way it runs on real&#10;hardware, and for the Kubernetes certifications.&lt;/p&gt;&#10;&lt;h2 id="options"&gt;Options&lt;a class="anchor" href="#options" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;VMs on Proxmox&lt;/strong&gt;: cheap to make and throw away, with snapshots before risky upgrades. But&#10;the NAS, the hypervisor and the whole cluster would share one machine, so one hardware&#10;failure or Proxmox upgrade takes everything down. The iGPU (for video transcoding) would&#10;need passthrough to one VM.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Kubernetes on the TrueNAS or Proxmox host itself&lt;/strong&gt;: mixes the cluster with the storage.&#10;Rejected for the same reason.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Separate physical machines&lt;/strong&gt; (chosen): two Lenovo ThinkCentre M720q and a Raspberry Pi 5.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h2 id="decision"&gt;Decision&lt;a class="anchor" href="#decision" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Chosen: separate physical machines, because the NAS, the hypervisor and the cluster can then&#10;be rebuilt or restarted without touching each other, and the cluster deals with real nodes:&#10;real disks, a real iGPU and two CPU architectures.&lt;/p&gt;</description></item><item><title/><link>https://docs.magelssen.io/decisions/0002-truenas-as-a-vm-on-proxmox/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0002-truenas-as-a-vm-on-proxmox/</guid><description>&lt;h1 id="truenas-as-a-vm-on-proxmox"&gt;TrueNAS as a VM on Proxmox&lt;a class="anchor" href="#truenas-as-a-vm-on-proxmox" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;One machine had to hold the disks (NAS) and also run VMs and containers for the&#10;services that are not in Kubernetes (Docker images, Pi-hole, Plex, Tailscale,&#10;the Omada controller). Either the NAS runs the VMs, or a hypervisor runs the&#10;NAS.&lt;/p&gt;&#10;&lt;h2 id="options"&gt;Options&lt;a class="anchor" href="#options" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;TrueNAS on the bare machine, running VMs and apps itself&lt;/strong&gt;: TrueNAS sees the disks&#10;directly. But VMs and apps are not what TrueNAS is built for, and they have&#10;changed a lot between releases. Tried out TrueNAS on the bare machine, but had&#10;problems with a lot of the apps (vms).&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Proxmox only, with ZFS and NFS/SMB set up by hand&lt;/strong&gt;: one layer less, but no NAS interface&#10;for shares, users, snapshot tasks, replication and disk health. All of it by hand on the&#10;hypervisor.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Proxmox on the bare machine, TrueNAS in a VM&lt;/strong&gt; (chosen).&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h2 id="decision"&gt;Decision&lt;a class="anchor" href="#decision" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Chosen: Proxmox with TrueNAS in a VM, because each does what it is built for. Proxmox is a&#10;full hypervisor with LXCs, backups, an API and a Terraform provider, and TrueNAS is a full NAS.&#10;The disks are passed through to the TrueNAS VM, so the pool (&lt;code&gt;Firelink&lt;/code&gt;) belongs to TrueNAS&#10;and could move to another machine.&lt;/p&gt;</description></item><item><title/><link>https://docs.magelssen.io/decisions/0003-flux-for-gitops/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0003-flux-for-gitops/</guid><description>&lt;h1 id="flux-for-gitops-not-argo-cd"&gt;Flux for GitOps, not Argo CD&lt;a class="anchor" href="#flux-for-gitops-not-argo-cd" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Everything in the cluster should come from Git, applied by a controller in the cluster, so&#10;nothing is applied by hand and every change has a commit. The two big GitOps tools are Flux&#10;and Argo CD. Both are CNCF graduated projects and both pull from Git.&lt;/p&gt;&#10;&lt;h2 id="options"&gt;Options&lt;a class="anchor" href="#options" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;Argo CD&lt;/strong&gt;: a web UI that shows every app, its resources and the diff to Git, with&#10;single sign-on and buttons to sync and roll back. More to run (API server, repo server,&#10;Redis, Dex), and the UI makes it easy to change things outside Git. Helm charts are&#10;rendered with &lt;code&gt;helm template&lt;/code&gt;, so they are not real Helm releases.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Flux&lt;/strong&gt; (chosen): a set of small controllers configured only with Kubernetes resources&#10;(&lt;code&gt;GitRepository&lt;/code&gt;, &lt;code&gt;Kustomization&lt;/code&gt;, &lt;code&gt;HelmRelease&lt;/code&gt;). No UI. Helm charts are installed as real&#10;Helm releases by helm-controller.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h2 id="decision"&gt;Decision&lt;a class="anchor" href="#decision" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Chosen: Flux, because the lack of a UI means working through &lt;code&gt;kubectl&lt;/code&gt;, &lt;code&gt;flux&lt;/code&gt; and the&#10;manifests, which is the point of the lab, and because Flux is what I use at work.&lt;/p&gt;</description></item><item><title/><link>https://docs.magelssen.io/decisions/0004-ingress-nginx/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0004-ingress-nginx/</guid><description>&lt;h1 id="ingress-nginx-for-traffic-into-the-cluster"&gt;ingress-nginx for traffic into the cluster&lt;a class="anchor" href="#ingress-nginx-for-traffic-into-the-cluster" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;The apps in the cluster needed to be reached over HTTPS by name, through one IP from MetalLB.&#10;The standard way was an &lt;code&gt;Ingress&lt;/code&gt; resource and an ingress controller. In November 2025 the&#10;Kubernetes project had announced that ingress-nginx would be retired in March 2026, and&#10;pointed to the Gateway API as the successor.&lt;/p&gt;&#10;&lt;h2 id="options"&gt;Options&lt;a class="anchor" href="#options" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;Gateway API from the start&lt;/strong&gt; (Envoy Gateway or another controller): the future, but&#10;skips the API that most clusters, charts and guides still use.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Traefik with Ingress&lt;/strong&gt;: also common, but adds its own resources on top.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;ingress-nginx&lt;/strong&gt; (chosen): the most used ingress controller, with &lt;code&gt;Ingress&lt;/code&gt; resources.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h2 id="decision"&gt;Decision&lt;a class="anchor" href="#decision" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Chosen: ingress-nginx, on purpose, although it was going to be retired: to get experience&#10;with the &lt;code&gt;Ingress&lt;/code&gt; API that most clusters still run, and then with moving a running cluster&#10;from Ingress to the Gateway API, which many teams have to do now.&lt;/p&gt;</description></item><item><title/><link>https://docs.magelssen.io/decisions/0005-external-secrets-with-aws-ssm/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0005-external-secrets-with-aws-ssm/</guid><description>&lt;h1 id="secrets-in-aws-ssm-through-external-secrets"&gt;Secrets in AWS SSM through External Secrets&lt;a class="anchor" href="#secrets-in-aws-ssm-through-external-secrets" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Apps need passwords and keys, and everything else is in Git. The repository is public, so&#10;whatever is in Git can be read by anyone, forever, also after it is deleted. Until then, the&#10;few Secrets in the cluster were made by hand.&lt;/p&gt;&#10;&lt;h2 id="options"&gt;Options&lt;a class="anchor" href="#options" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;SOPS + age&lt;/strong&gt;: secrets encrypted in Git, and Flux decrypts them. Simple and built into&#10;Flux. But the encrypted files are public for good: if the age key ever leaks, every secret&#10;in the history leaks with it.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Sealed Secrets&lt;/strong&gt;: also encrypted in Git, with the same problem. The key is in the cluster,&#10;so rebuilding the cluster without a backup of it means sealing every secret again.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Vault or OpenBao in the cluster&lt;/strong&gt;: a real secrets server, but heavy for a homelab, and it&#10;needs storage, unsealing and its own backups. The cluster would hold the secrets it needs&#10;to start.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;External Secrets + AWS SSM Parameter Store&lt;/strong&gt; (chosen): the secrets live in AWS, and Git&#10;only has the names.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h2 id="decision"&gt;Decision&lt;a class="anchor" href="#decision" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Chosen: External Secrets with AWS SSM, because no secret is in Git at all, not even&#10;encrypted, and SSM is free at this size. AWS is already used for Route53.&lt;/p&gt;</description></item><item><title/><link>https://docs.magelssen.io/decisions/0006-envoy-gateway/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0006-envoy-gateway/</guid><description>&lt;h1 id="envoy-gateway-gateway-api-instead-of-ingress-nginx"&gt;Envoy Gateway (Gateway API) instead of ingress-nginx&lt;a class="anchor" href="#envoy-gateway-gateway-api-instead-of-ingress-nginx" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;All traffic came in through ingress-nginx (&lt;a href="https://docs.magelssen.io/decisions/0004-ingress-nginx/"&gt;0004&lt;/a&gt;), chosen knowing it&#10;would be retired, so that the move to the Gateway API could be done on a running cluster. The&#10;Kubernetes project stopped fixing ingress-nginx in March 2026 and points to the Gateway API as&#10;the successor to Ingress. Some routes go to services outside the cluster with&#10;self-signed certificates (Proxmox, TrueNAS, Omada), and the Gateway API&amp;rsquo;s own&#10;&lt;code&gt;BackendTLSPolicy&lt;/code&gt; cannot skip certificate checks.&lt;/p&gt;</description></item><item><title/><link>https://docs.magelssen.io/decisions/0007-restic-for-laptop-backups/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0007-restic-for-laptop-backups/</guid><description>&lt;h1 id="restic-for-laptop-backups-not-kopia"&gt;restic for laptop backups, not Kopia&lt;a class="anchor" href="#restic-for-laptop-backups-not-kopia" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Laptops need backups to the NAS, also when they are not at home. The backup server runs in&#10;the cluster behind Envoy Gateway, with its data on NFS (&lt;code&gt;Catacombs/backups&lt;/code&gt;). A laptop that&#10;is stolen or hit by ransomware must not be able to delete its own backups.&lt;/p&gt;&#10;&lt;h2 id="options"&gt;Options&lt;a class="anchor" href="#options" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;Kopia&lt;/strong&gt;: fast, with a web UI and a repository server with users. But the&#10;server speaks gRPC, which needs HTTP/2 and TLS all the way to the pod, so it needs its own&#10;certificate and extra Envoy settings. The server cannot stop a client from deleting&#10;snapshots.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;BorgBackup&lt;/strong&gt;: mature, but runs over SSH, which does not fit behind the HTTP gateway.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;restic with rest-server&lt;/strong&gt; (chosen): restic encrypts on the laptop, and rest-server is a&#10;small HTTP server for the repositories.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h2 id="decision"&gt;Decision&lt;a class="anchor" href="#decision" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Chosen: restic with rest-server, because rest-server has an append-only mode (a laptop can add&#10;backups, but not delete or change them), it works as plain HTTP behind Envoy Gateway, and the&#10;same tool can be used later for the offsite copies.&lt;/p&gt;</description></item><item><title/><link>https://docs.magelssen.io/decisions/0008-renovate-for-dependency-updates/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0008-renovate-for-dependency-updates/</guid><description>&lt;h1 id="renovate-for-dependency-updates"&gt;Renovate for dependency updates&lt;a class="anchor" href="#renovate-for-dependency-updates" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Versions are pinned everywhere: Helm charts, images, Flux itself, GitHub Actions (by SHA) and&#10;pre-commit hooks. They were checked by hand once a month against &lt;code&gt;docs/operations/versions.md&lt;/code&gt;, which&#10;was slow and easy to forget. A tool should notice new versions and open pull requests.&lt;/p&gt;&#10;&lt;h2 id="options"&gt;Options&lt;a class="anchor" href="#options" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;Dependabot&lt;/strong&gt;: built into GitHub, but has no Flux support (HelmReleases and the Flux&#10;components are not updated), and Kubernetes images need one entry per directory.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Flux image automation&lt;/strong&gt;: runs in the cluster and commits new image tags. Only images, and&#10;it needs write access to the repository.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Updatecli&lt;/strong&gt;: flexible, but every dependency needs a hand-written rule.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Renovate&lt;/strong&gt; (chosen): understands Flux, Kubernetes manifests, SHA-pinned actions and&#10;pre-commit, with grouping, schedules and a minimum release age built in.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h2 id="decision"&gt;Decision&lt;a class="anchor" href="#decision" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Chosen: Renovate, as the hosted Mend Renovate app, because it covers almost everything pinned&#10;in the repository and needs nothing running in the homelab.&lt;/p&gt;</description></item><item><title/><link>https://docs.magelssen.io/decisions/0009-github-app-for-flux/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0009-github-app-for-flux/</guid><description>&lt;h1 id="a-github-app-for-fluxs-commit-statuses"&gt;A GitHub App for Flux&amp;rsquo;s commit statuses&lt;a class="anchor" href="#a-github-app-for-fluxs-commit-statuses" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Flux posts a commit status to GitHub for each commit it applies, so GitHub shows what actually&#10;runs in the cluster. For that, the cluster needs a login to GitHub that can write commit&#10;statuses. The repository is public, and the login is stored in the cluster.&lt;/p&gt;&#10;&lt;h2 id="options"&gt;Options&lt;a class="anchor" href="#options" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;Classic personal access token&lt;/strong&gt;: the &lt;code&gt;repo&lt;/code&gt; scope gives full access to all my&#10;repositories. Far too much.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Fine-grained personal access token&lt;/strong&gt;: can be limited to commit statuses on one&#10;repository. But it belongs to my account, the statuses show up as me, and it has to be&#10;renewed when it expires.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Deploy key&lt;/strong&gt;: SSH access to Git only. It cannot post statuses.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;GitHub App&lt;/strong&gt; (chosen): its own identity, installed on this repository only, with the one&#10;permission it needs.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h2 id="decision"&gt;Decision&lt;a class="anchor" href="#decision" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Chosen: a GitHub App (&lt;code&gt;larma-homelab-flux&lt;/code&gt;), because it has the least access, is not tied to&#10;my account, and shows up as its own bot. Flux supports App logins directly.&lt;/p&gt;</description></item><item><title/><link>https://docs.magelssen.io/decisions/0010-no-bot-account-for-claude/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0010-no-bot-account-for-claude/</guid><description>&lt;h1 id="no-bot-account-or-auto-approval-for-claudes-pull-requests"&gt;No bot account or auto-approval for Claude&amp;rsquo;s pull requests&lt;a class="anchor" href="#no-bot-account-or-auto-approval-for-claudes-pull-requests" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Claude Code writes most changes in this repository, on a branch. I read them, commit them&#10;under my own name (with a &lt;code&gt;Co-Authored-By&lt;/code&gt; line for Claude), open the pull request and merge&#10;it. The ruleset on &lt;code&gt;main&lt;/code&gt; requires a pull request and the checks, but 0 approvals, because&#10;GitHub does not let anyone approve their own pull request. So OpenSSF Scorecard gives a low&#10;Code-Review score, and the rule &amp;ldquo;a change needs an approval&amp;rdquo; cannot be turned on.&lt;/p&gt;</description></item><item><title/><link>https://docs.magelssen.io/decisions/0011-alerting-with-alertmanager-and-telegram/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0011-alerting-with-alertmanager-and-telegram/</guid><description>&lt;h1 id="alerts-from-alertmanager-to-telegram-and-uptime-kuma-moved-to-proxmox"&gt;Alerts from Alertmanager to Telegram, and Uptime Kuma moved to Proxmox&lt;a class="anchor" href="#alerts-from-alertmanager-to-telegram-and-uptime-kuma-moved-to-proxmox" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Prometheus and Alertmanager run in the cluster (kube-prometheus-stack), with the chart&amp;rsquo;s default&#10;alert rules, but Alertmanager has no receiver: alerts fire and nobody is told. On 2026-11-17 all&#10;sites were down for 29 minutes while every pod was healthy and Flux was green. It was found by&#10;chance.&lt;/p&gt;&#10;&lt;p&gt;Uptime Kuma also runs in the cluster, and it showed no downtime. From a pod, kube-proxy sends&#10;traffic for a LoadBalancer IP straight to the backend pod (the Service has &lt;code&gt;ipMode: VIP&lt;/code&gt;). The&#10;check never asks the network who has 192.168.30.200, so it skips MetalLB, which is the part that&#10;broke. Anything in the cluster also goes down with the cluster, so it cannot report that.&lt;/p&gt;</description></item><item><title/><link>https://docs.magelssen.io/decisions/0012-pihole-stays-in-an-lxc/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0012-pihole-stays-in-an-lxc/</guid><description>&lt;h1 id="pi-hole-stays-in-an-lxc-on-proxmox-not-in-kubernetes"&gt;Pi-hole stays in an LXC on Proxmox, not in Kubernetes&lt;a class="anchor" href="#pi-hole-stays-in-an-lxc-on-proxmox-not-in-kubernetes" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;The goal is to move the services on Proxmox into Kubernetes. Pi-hole (LXC 104,&#10;&lt;code&gt;192.168.20.20&lt;/code&gt;) is the DNS server for the whole home network: it answers internet&#10;lookups for every client, and it holds the local records that point the Kubernetes&#10;hostnames (&lt;code&gt;*.magelssen.io&lt;/code&gt;) to the Envoy Gateway at &lt;code&gt;192.168.30.200&lt;/code&gt;. Unlike the&#10;other services, the cluster itself depends on it.&lt;/p&gt;</description></item><item><title/><link>https://docs.magelssen.io/decisions/0013-uptime-kuma-without-docker/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0013-uptime-kuma-without-docker/</guid><description>&lt;h1 id="uptime-kuma-installed-with-nodejs-in-its-lxc-not-in-docker"&gt;Uptime Kuma installed with Node.js in its LXC, not in Docker&lt;a class="anchor" href="#uptime-kuma-installed-with-nodejs-in-its-lxc-not-in-docker" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;&lt;a href="https://docs.magelssen.io/decisions/0011-alerting-with-alertmanager-and-telegram/"&gt;ADR 0011&lt;/a&gt; moved Uptime Kuma out of the&#10;cluster into an LXC on Proxmox. That left how to run it inside the LXC. Uptime Kuma ships a&#10;Docker image (which the cluster used) and supports a plain Node.js install from source.&#10;The old docker LXC (102), which ran Docker with Portainer, was just retired.&lt;/p&gt;&#10;&lt;h2 id="options"&gt;Options&lt;a class="anchor" href="#options" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;Docker in the LXC&lt;/strong&gt;: the same image as in the cluster. But Proxmox recommends running&#10;application containers in a VM, not in an LXC; Docker in an unprivileged LXC needs&#10;&lt;code&gt;nesting&lt;/code&gt; and &lt;code&gt;keyctl&lt;/code&gt; and breaks now and then on Proxmox or kernel updates. It adds a&#10;daemon and a layer for one app.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;A small VM with Docker&lt;/strong&gt;: what Proxmox recommends for Docker, but a full VM (kernel,&#10;more RAM and disk) for one small Node.js app.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Node.js in the LXC&lt;/strong&gt; (chosen): Uptime Kuma&amp;rsquo;s documented install without Docker, git tag&#10;pinned, run by systemd.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h2 id="decision"&gt;Decision&lt;a class="anchor" href="#decision" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;Chosen: Node.js in the LXC, because it is the simplest setup that Proxmox supports well,&#10;and it fits being managed by Ansible later. Debian 13 ships a new enough Node.js (20.19,&#10;Uptime Kuma 2.x needs 20.4 or newer), so no extra apt repository is needed.&lt;/p&gt;</description></item><item><title/><link>https://docs.magelssen.io/decisions/0014-infrastructure-services-stay-outside-kubernetes/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0014-infrastructure-services-stay-outside-kubernetes/</guid><description>&lt;h1 id="services-needed-to-run-or-fix-the-cluster-stay-outside-kubernetes"&gt;Services needed to run or fix the cluster stay outside Kubernetes&lt;a class="anchor" href="#services-needed-to-run-or-fix-the-cluster-stay-outside-kubernetes" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;The goal has been to move the services on Proxmox into Kubernetes. Two exceptions were made&#10;one at a time: Pi-hole (&lt;a href="https://docs.magelssen.io/decisions/0012-pihole-stays-in-an-lxc/"&gt;ADR 0012&lt;/a&gt;) and Uptime Kuma&#10;(&lt;a href="https://docs.magelssen.io/decisions/0011-alerting-with-alertmanager-and-telegram/"&gt;ADR 0011&lt;/a&gt;). The next candidate was the&#10;Omada controller (issue #102), which manages the router, the switch and the access point.&#10;It raised the same question a third time, so this record turns the exceptions into a rule.&lt;/p&gt;</description></item><item><title/><link>https://docs.magelssen.io/decisions/0015-hugo-for-the-docs-website/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://docs.magelssen.io/decisions/0015-hugo-for-the-docs-website/</guid><description>&lt;h1 id="a-hugo-website-built-from-docs-published-to-github-pages"&gt;A Hugo website built from docs/, published to GitHub Pages&lt;a class="anchor" href="#a-hugo-website-built-from-docs-published-to-github-pages" aria-label="Link"&gt;&lt;/a&gt;&lt;/h1&gt;&#10;&lt;h2 id="context"&gt;Context&lt;a class="anchor" href="#context" aria-label="Link"&gt;&lt;/a&gt;&lt;/h2&gt;&#10;&lt;p&gt;The documentation is Markdown in &lt;code&gt;docs/&lt;/code&gt;. It is read on GitHub, and by Claude&#10;from the files. A website with navigation and search makes it easier to read, but a separate&#10;wiki or a second copy of the text would drift away from the repository, and Claude could not&#10;read it. So the website must be built from the same files, without changing how they read&#10;on GitHub: no generator-only syntax, relative links to &lt;code&gt;.md&lt;/code&gt; files, &lt;code&gt;README.md&lt;/code&gt; as the index of&#10;a directory, and the title as the first &lt;code&gt;# heading&lt;/code&gt;.&lt;/p&gt;</description></item></channel></rss>