Private infrastructure

GitLab CI runs tests, dependency audits and a Trivy scan for secrets, dependencies and the image before anything reaches the registry. Renovate opens version bump merge requests and automerges routine ones; roughly a dozen high blast radius packages and every major version wait for a human click.

All 32 Argo CD applications auto-sync, so there is exactly one gate per change. Images are pinned by digest, each written in exactly one place. A gate that always passes is not a gate: where a second manual step was removed, it was replaced by a CI check that fails the merge, not by trust.

Three things sit deliberately outside GitOps: node upgrades (they cordon nodes and swap binaries, not containers), the vendored Wazuh manifests, and secret values, which no reconciler can supply. Argo CD replaced a hand written redeploy poller that classified commits by changed path and misidentified more than half of the chart bumps.

Prevention is a chain, and every link has a deliberate failure direction. The edge router forwards web traffic and nothing else, and intrusion prevention runs there. Ingress then arrives on a MetalLB address that floats with the Traefik pod and preserves the real client address; it replaced the K3s default load balancer, which masqueraded every request to one internal address, so every client looked the same and there was nothing to ban. A firewall bouncer on every node drops banned sources on all ports ahead of the proxy, and its chains fail open, so a crash costs protection, never access. Inside the proxy the two kinds of route run parallel rather than stacked: a LAN route carries an address allowlist, a public route carries the ban middleware and an application layer WAF that fails closed, and neither carries the other's. The login portal sits in front of some routes of each kind.

Name resolution is not the exposure boundary; the middleware is. A route added without the allowlist would be reachable from outside the moment it exists, whatever DNS says, so a CI check fails any merge that adds one without it. Inside the cluster, the namespaces holding identity, secrets and the SIEM are default deny for ingress, the application namespace has an egress baseline, Pod Security Admission applies per namespace, and workloads that tolerate it run in user namespaces.

Behind all of that, Wazuh (HIDS and SIEM) correlates. CrowdSec reads the ingress logs, Falco watches syscalls through eBPF, ClamAV scans on a schedule, Lynis audits host posture weekly, and one agent per node does curated realtime file integrity monitoring. The two hosts that cannot run an agent, the NAS and the router, report over syslog; the router can send to exactly one destination, so a relay takes the stream and fans it out to both Loki and the SIEM, holding routine traffic lines below the alerting threshold so that full logging cannot flood it. Trivy is deliberately not wired in, being a build time gate rather than runtime detection. High severity alerts page via Telegram.

One hostile request, end to end: the WAF matches the payload and refuses it, the agent reading the access log crosses a scenario threshold and records a ban, and within seconds every node has pulled that decision and drops the source on all ports, so the next request never reaches the proxy at all. Repeat offences escalate the duration, and the ban itself travels back into the SIEM as an event like any other.

Argo CD is the most privileged workload and is fenced accordingly: a default deny NetworkPolicy pair limits it to DNS, the API server and outbound HTTPS. The chart's own policies had to be disabled to get there, because its shipped server policy allowed ingress from every pod in the cluster, and NetworkPolicies are additive. Found by reading the live object, not the chart.

All three signals persist to MinIO on NVMe. A nightly job mirrors the day's objects to a compressed dataset on the HDD pool, so the spinning disks stay asleep instead of absorbing continuous writes.

Gatus does not probe ports. Its interesting checks query Mimir: unavailable replicas anywhere in the app namespace, exporters that have stopped producing metrics, and Argo CD applications that can no longer be rendered. That last case is invisible from every other angle, because the pods keep running while the cluster silently stops tracking git.

Readiness probe coverage went from 7 to 19 of 31 containers, which is what gave the replica metric meaning: a container without a probe is Ready the moment it starts. Exporter sidecars deliberately have none, because a pod is only Ready when every container is, and a probe on a sidecar would pull the application out of its Service. Monitoring failure would cause an outage.

Three senders reach one chat and nothing else does: the SIEM for high severity correlations, Grafana for a handful of metric rules (containers that restart repeatedly, containers the OOM killer takes, a security tool that stops reporting), and Gatus for everything better expressed as a question than as a threshold. A check must not depend on the thing it watches, so the data sources have liveness checks of their own, and the one that matters most reads the SIEM's own delivery log through Loki, from a different system, because the only thing that would notice a dead pager is the pager. A second, independent check counts high severity alerts in the SIEM's index, so a silent delivery path shows up twice. Jobs whose only output is an exit status push a heartbeat instead of being polled, which makes a job that stopped running as visible as one that failed.

What deliberately pages nobody is as considered as what does. Argo CD's own notifications are off, because a pager is not a chat channel; the image scanner stays a build time gate and never pages; routine firewall and traffic lines are held below the alerting threshold. Each of those is a decision that can be revisited, not an omission.

Authelia provides the login portal and acts as OIDC provider, binding read-only to LDAP. Browser-only apps sit behind forwardAuth. Apps with native clients that cannot follow a login redirect bind to LDAP directly. Argo CD uses both: forwardAuth keeps anonymous traffic off it entirely, and OIDC tells its own RBAC who you are.

LDAP rather than an OIDC-first design because it is the common denominator across the services that matter here. Apps behind forwardAuth are ClusterIP only: a NodePort would SNAT traffic into the pod CIDR that the identity header trusts, making it forgeable from any host on the LAN.

Values live in OpenBao, and the cluster reads them for itself through the Vault Secrets Operator. Each namespace has its own path prefix, its own read policy and its own authentication role, so a policy can only ever expose that one namespace's secrets. The operator holds no standing credential: it asks the API server for a short-lived token belonging to the namespace's own service account, the store validates that token back against the API server before trusting it, and what comes back carries nothing but that namespace's policy, for an hour.

From there the operator writes an ordinary Kubernetes Secret, owned by the object that asked for it, and re-reads on a timer. Consumers are restarted only when the value actually changed, compared by hash rather than by timestamp, so a rotation reaches every workload that needs it while a refresh that changed nothing disturbs no one.

An NVMe pool with small record sizes holds service configs and databases, bind-mounted as hostPath into the LXC node. A second NVMe pool backs democratic-csi for dynamically provisioned volumes (observability, Wazuh, Endpaper). The HDD pool serves bulk data over static NFS volumes and holds the cold archive and snapshot replication.

SQLite and MariaDB locking over NFS is unsafe, so anything database-bearing uses hostPath. iSCSI is impossible here: the LXC worker's network namespace blocks the kernel initiator, which is also why Longhorn was abandoned. Backups are ZFS snapshots replicated to a second pool, on-box only.