ci: install docker + kustomize as static binaries, not distro packages

CI has been failing silently since 2026-08-03. The run for 9f1cbca did
execute (Gitea action_run #6, status=failure, 13m) but died before pushing an
image, so no package version exists for that commit and ArgoCD has been
serving the 2026-07-20 build ever since.

Cause: the "Ensure docker + kustomize are available" step ran
`apt-get install -y docker.io`. The runner's job image was node:18-bullseye
and Debian 11 went EOL, so debian-security stopped serving those package
versions. `apt-get update` still succeeds, so the failure surfaces only at
install time as 404s on docker.io, containerd, systemd and libcap2.

Fetch both tools as pinned static binaries instead. These have no distro
archive dependency, so the same rot cannot recur when bookworm ages out.
Verified in the real job environment: the static client installs and reaches
the host daemon through the mounted socket.

The runner job image has separately been moved to node:22-bookworm, which
also resolves the immediate breakage, but the static-binary approach is kept
as the durable fix.

Also refresh the prerequisites header: items 3 and 4 still described the
pre-ADR-0008 registry design (k3s now pulls over public HTTPS, and the
private package's pull auth comes from External Secrets).

.gitea/workflows/ci.yml

@@ -8,10 +8,10 @@
 #      jobs can't see it, enable socket passthrough in act_runner config or switch this to kaniko.
 #   2. The RUNNER HOST's dockerd must treat 10.66.15.22:3000 as an insecure (HTTP) registry
 #      ("insecure-registries": ["10.66.15.22:3000"] in /etc/docker/daemon.json).
-#   3. K3S must trust the same HTTP registry — add a registries.yaml entry (k3s_registry_mirrors)
-#      for 10.66.15.22:3000, then restart k3s, or the pods can't pull the image.
-#   4. The Gitea container package should be PUBLIC (simplest). If PRIVATE, add an imagePullSecret
-#      (see k8s/deployment.yaml) and grant the run token package:write.
+#   3. K3S does NOT need registries.yaml — it pulls the same package over public HTTPS
+#      (git.7tl-homelab.com, valid cert). See ADR-0008. No node config, no cluster restart.
+#   4. The Gitea package is PRIVATE; k3s pull auth comes from External Secrets ->
+#      k8s/externalsecret-registry.yaml -> `gitea-registry` imagePullSecret. See ADR-0008.
 #   5. secrets.GITEA_TOKEN (auto-injected per run) needs write access to this repo (tag-bump commit)
 #      and the package registry.
 
@@ -61,12 +61,32 @@ jobs:
         id: tag
         run: echo "sha=$(echo ${{ github.sha }} | cut -c1-12)" >> "$GITHUB_OUTPUT"
 
+      # Both tools are fetched as STATIC BINARIES, never from distro packages.
+      # History: this step used `apt-get install -y docker.io`, which broke CI on 2026-08-03 when
+      # Debian 11 (bullseye — the runner's job image is node:18-bullseye) went EOL and the
+      # debian-security pool stopped serving those package versions. `apt-get update` still
+      # succeeds, so the failure surfaces only at install time as a wall of 404s. Static binaries
+      # have no distro-archive dependency and pin an exact version, so this cannot rot the same way.
       - name: Ensure docker + kustomize are available
+        env:
+          DOCKER_CLI_VERSION: '28.3.2'
+          KUSTOMIZE_VERSION: 'v5.4.3'
         run: |
-          command -v docker >/dev/null 2>&1 || { apt-get update && apt-get install -y docker.io; }
-          command -v kustomize >/dev/null 2>&1 || \
-            curl -sSL "https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.4.3/kustomize_v5.4.3_linux_amd64.tar.gz" \
+          set -euo pipefail
+          if ! command -v docker >/dev/null 2>&1; then
+            curl -sSL --retry 3 --retry-delay 5 \
+              "https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKER_CLI_VERSION}.tgz" -o /tmp/docker.tgz
+            # We only need the client; the daemon is the runner host's, via the mounted socket.
+            tar xzf /tmp/docker.tgz -C /tmp --strip-components=1 docker/docker
+            install -m 0755 /tmp/docker /usr/local/bin/docker
+          fi
+          if ! command -v kustomize >/dev/null 2>&1; then
+            curl -sSL --retry 3 --retry-delay 5 \
+              "https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${KUSTOMIZE_VERSION}/kustomize_${KUSTOMIZE_VERSION}_linux_amd64.tar.gz" \
               | tar xz -C /usr/local/bin
+          fi
+          docker version --format 'docker client={{.Client.Version}} server={{.Server.Version}}'
+          kustomize version
 
       # CI_TOKEN: a Gitea Personal Access Token (scopes: write:package, write:repository) owned by
       # the repo owner, stored as a repo Actions secret. The auto GITEA_TOKEN lacks reliable package

docs/launch-checklist.md

@@ -0,0 +1,243 @@
+# Launch Checklist — DNS Cutover to the Astro Rebuild
+
+**Single source of truth** for everything standing between today and pointing
+`bluecapstrategies.com` at the homelab k3s deployment.
+
+Verified against the working tree, the cluster, and ArgoCD on **2026-09-18**. Supersedes the
+task lists that were scattered across `planning/parity-checklist.md` (git-ignored, content-only)
+and `docs/seo-aeo-optimization.md` §2/§7 (SEO-only, stale) — those now point here.
+
+Status legend: `[ ]` open · `[x]` done · `[?]` needs verification · 🛑 owner decision
+
+---
+
+## 0. As-built state (2026-09-18)
+
+| Thing | State |
+|-------|-------|
+| ArgoCD app `bcs-website` | Synced / Healthy |
+| Pods | 2/2 Running, ns `bluecap-strategies` |
+| Deployed image | `git.7tl-homelab.com/bmr_bluecap/bcs-website:fc64909f3df1` (built **2026-07-20**) |
+| `k8s/kustomization.yaml` newTag | `fc64909f3df1` — same, so ArgoCD is correct; CI never bumped it |
+| HEAD | `9f1cbca` (2026-08-03), pushed to `origin/main`, **never deployed** |
+| Ingress | `bluecap-website` (traefik), hosts apex + `www`, **port 80 only, no TLS** |
+| `npm run build` | green, 21 pages, 0 errors / 0 warnings |
+| `npm run typecheck` | green, 28 files, 0/0/0 |
+| `npm run seo:lint` | green, 21 pages clean |
+| `npm run test:e2e` | **42 passed / 4 failed** (stale assertions, see §2.1) |
+| CI root cause | **found + fixed in-repo 2026-09-22** — see §1.2; needs a green run to confirm |
+| Old WordPress site | untouched and serving — must stay that way until cutover |
+| Gitea host | KVM VM `gitea-1` (10.66.15.22), **Debian 12**, docker compose — *not* in k3s |
+| Gitea / act_runner | 1.27.3 / 3.4.2 — both healthy |
+| CI job image | **`node:22-bookworm`** (Debian 12) as of 2026-09-22 — was EOL bullseye; see §1.2 |
+
+**The headline:** the public-facing deployment is ~7 weeks behind `main`. The Aug 3 exact-copy
+work is not live.
+
+---
+
+## 1. Cutover blockers
+
+Nothing below is optional. Cutover is unsafe until all of §1 is closed.
+
+### 1.1 TLS is not configured 🛑 hard blocker
+- [ ] Uncomment the `tls:` block and `cert-manager.io/cluster-issuer` annotation in
+      `k8s/ingress.yaml`; set `secretName: bluecap-website-tls`.
+- [ ] Use the existing `letsencrypt-prod` ClusterIssuer (present in-cluster, 472d old, Ready).
+- [ ] **Resolve the ordering problem:** HTTP-01 validation requires DNS to already point at the
+      cluster, but we don't want to move DNS before a cert exists. Pick one:
+      - DNS-01 issuer (preferred — issues the cert *before* cutover, zero insecure window), or
+      - accept a short window: cut DNS, then let HTTP-01 solve, serving plain HTTP for ~1–2 min.
+- [ ] Verify `https://` serves a valid chain and HTTP→HTTPS redirects before announcing.
+
+### 1.2 CI/CD is stalled — no deploy since 2026-07-20 🛑 hard blocker
+**ROOT CAUSE FOUND AND FIXED IN-REPO 2026-09-22.** Not a runner problem — the runner was online
+and the run *did* execute.
+
+Evidence (Gitea 1.27.3 on `gitea-1`, sqlite `action_run` for repo_id 3):
+- Run #6, commit `9f1cbca1b148`, created 2026-08-03 13:46:53, **status 2 = failure**, ran 13m
+  (13:47:02 → 14:00:11) on runner 59.
+- No `9f1cbca1b148` entry in `package_version` → it died **at or before** the image push.
+- Job logs are past retention (`/data/gitea/actions_log` no longer holds run 205), so the cause
+  was reproduced directly instead.
+
+**The cause:** the `Ensure docker + kustomize are available` step ran
+`apt-get install -y docker.io`. The job container image is **`node:18-bullseye`** (Debian 11),
+and bullseye went EOL — `debian-security` no longer serves those package versions. `apt-get
+update` still succeeds, so this fails only at install time, with 404s on `docker.io`,
+`containerd`, `systemd`, `libcap2`. Reproduced verbatim in a `node:18-bullseye` container on the
+runner host. That timing matches exactly: last green run 2026-07-20, first failure 2026-08-03.
+
+**The fix (applied to `.gitea/workflows/ci.yml`):** fetch the docker CLI and kustomize as pinned
+**static binaries** (`download.docker.com` / GitHub releases) instead of distro packages. Verified
+in the real job environment: the static client installs, reports `server=28.1.1` through the
+mounted socket, and `docker ps` succeeds. kustomize tarball and `git` are fine.
+
+Everything else on the runner path checked out and needs no change:
+- [x] act_runner container up, `v3.4.2`, registered and polling (the daily 07:30 "connection
+      refused" entries are the Gitea restart window, harmless).
+- [x] Runner scope **already fixed** — runner id 59 is `owner_id=0, repo_id=0` (instance-scoped),
+      so the old user-scoped problem is resolved. 58 stale duplicate runner rows remain; cosmetic,
+      prune with the role's `gitea_runner_prune` tag when convenient.
+- [x] Runner host dockerd trusts `10.66.15.22:3000` as insecure — confirmed in `docker info`.
+- [x] `config.yaml` pins jobs to `gitea_network` and passes the docker socket — correct.
+- [x] `CI_TOKEN` repo secret present (repo_id 3).
+- [x] `paths-ignore` did not swallow the push — the commit touched `src/**`.
+- [x] `docker build` of the current tree succeeds locally, so the Dockerfile and lockfile are fine.
+
+Remaining:
+- [ ] **Re-trigger CI and confirm a green run** — the fix is verified at the step level but the
+      full pipeline has not run end-to-end since July. Requires a push touching a non-ignored path.
+- [ ] Confirm the tag-bump commit lands and ArgoCD syncs the new image.
+- [ ] Do not cut DNS onto a stale image — confirm the deployed tag matches HEAD.
+- [x] **CI job image upgraded 2026-09-22.** `gitea_runner_labels` in
+      `~/Homelab/ansible/roles/gitea/defaults/main.yml` changed `node:18-bullseye` →
+      **`node:22-bookworm`**; playbook run succeeded (ok=40 changed=8 failed=0). Runner **60**
+      registered and online, `owner_id=0` (instance-scoped), label confirmed in `/data/.runner`.
+      The job image now matches the host VM and k3s nodes (Debian 12) and the Node 22 that
+      workflows install anyway.
+      - Gotcha for next time: `--tags gitea_runner_prune` restricts the run to *only* that tag, so
+        it cannot be combined with `-e gitea_runner_force_reregister=true`. Two separate runs.
+      - Gotcha 2: `gitea_admin` had `must_change_password=1` (set during the 1.27.3 container
+        recreate on 2026-09-14), which made Gitea reject **all** admin API calls with 403 — not
+        401, so credentials were fine. Cleared 2026-09-22 with
+        `docker exec -u git gitea gitea admin user must-change-password --unset gitea_admin`
+        (password unchanged; vault still valid). The admin API works again.
+- [ ] Optional cleanup: prune the stale duplicate runner rows (59 of them now). Unblocked by the
+      admin-API fix above. `ansible-playbook -i inventory/hosts.ini gitea-deploy.yml --tags
+      gitea_runner_prune`. Purely cosmetic — deletes only runners the API reports offline.
+
+**Note on the two fixes:** CI now has *two* independent reasons to work — the workflow no longer
+uses apt at all, and the job image is no longer EOL (bookworm still serves `docker.io` fine). The
+static-binary approach remains the more durable of the two; keep it regardless.
+
+### 1.3 Developer-facing copy is visible on the live contact page 🛑
+`src/pages/contact.astro:37-41` renders, to real visitors:
+> "The form is static-first and posts to the configured endpoint when
+> `PUBLIC_CONTACT_FORM_ENDPOINT` is set. Without an endpoint, it opens an email…"
+- [ ] Delete that `.form-note`. Replace with visitor-appropriate copy or nothing.
+
+### 1.4 Contact form has no endpoint 🛑
+`PUBLIC_CONTACT_FORM_ENDPOINT` is `""` in `k8s/configmap.yaml`, so the form falls back to a
+mailto handoff. Inbound leads will be lost or awkward on day one.
+- [ ] Choose an endpoint (Formspree / webhook / ntfy / self-hosted) and set it in the ConfigMap.
+- [ ] Test a real submission end-to-end and confirm `/thank-you/` is reached.
+- [ ] Confirm the delivery address is monitored.
+
+### 1.5 Placeholder blog post would go live and be indexed 🛑
+`src/content/insights/example-draft.md` has `draft: true`, but the flag is **declared in the Zod
+schema and never enforced** — no `filter` on `getCollection("insights")` in
+`src/pages/insights/[slug].astro` or the index. Result: `/insights/example-draft/` is built *and*
+listed in `dist/sitemap-0.xml`, so "Example draft insight" ships to Google.
+- [ ] Either filter drafts out of `getCollection` (both the index and `getStaticPaths`) — the
+      durable fix — or delete the file.
+- [ ] Re-run `npm run build` and confirm the URL is gone from `sitemap-0.xml`.
+- [ ] Decide whether `/insights/` should ship at all with zero real articles (see §3).
+
+### 1.6 Canonical host is undecided 🛑 owner
+The ingress answers on **both** apex and `www`; `astro.config.mjs`, `robots.txt`, and the
+ConfigMap all hardcode `https://www.bluecapstrategies.com`. Serving identical content on two
+hosts with no redirect splits SEO signal from day one.
+- [ ] Owner picks apex or `www`.
+- [ ] Configure a 301 from the loser to the winner (traefik middleware or edge).
+- [ ] Reconcile `PUBLIC_SITE_URL`, canonical tags, `robots.txt` Sitemap line, and the ingress.
+
+---
+
+## 2. Should-fix before cutover
+
+### 2.1 e2e suite is red — stale assertions, not a regression
+4 of 46 tests fail. Both failures assert **invented copy that the exact-copy work correctly
+removed**:
+- `tests/site.spec.ts:67` waits for a link named `Explore services`; the homepage CTA is now
+  `All services` (matches live WP).
+- `tests/site.spec.ts:101` waits for a heading matching `/Focused expertise/`, which no longer
+  exists on `/focus-areas/`.
+
+Navigation itself works. The tests are wrong, not the site.
+- [ ] Update the two tests to assert current copy.
+- [ ] Get back to 46/46 so the suite is trustworthy at cutover.
+
+### 2.2 Repo hygiene
+Commit `9f1cbca` committed ~50 `.playwright-mcp/` screenshots and YAML snapshots plus a 2.8 MB
+`home-full.png` at the repo root.
+- [ ] Add `.playwright-mcp/` to `.gitignore`; remove the tracked artifacts and `home-full.png`.
+
+### 2.3 Uptime monitoring is not deployed
+Per ADR-0004 the plan is to reuse the homelab `monylog` stack (not Uptime Kuma). Pod metrics and
+nginx logs already flow via Alloy; the external-availability piece was never added.
+- [?] Verify current state on monylog-1 — the items below are from notes, not re-verified today.
+- [ ] Add `blackbox_exporter` + `web-bluecap.yml` alert rules; re-run the monylog ansible role to
+      reload Prometheus.
+- [ ] Confirm an external HTTPS check fires to Pushover on a simulated outage. **Do this before
+      cutover** — the first hours on new infrastructure are when you most need paging.
+
+### 2.4 Analytics is a no-op
+`src/components/Analytics.astro` is wired into `BaseLayout` but gated on `PUBLIC_` env vars that
+are absent from `k8s/configmap.yaml`, so nothing is collected.
+- [ ] Decide whether to launch with analytics or add it after (ADR-0005: PostHog + Umami trial).
+- [ ] **PostHog requires a privacy/consent notice before production use** — that gates it, not
+      the code.
+- [ ] Set the env vars and confirm pageviews plus a consultation/CTA event.
+
+---
+
+## 3. Owner decisions (Brad)
+
+These are yours, not mine. Several block content that is otherwise ready.
+
+- [ ] 🛑 **Canonical host** — apex vs `www` (also §1.6).
+- [ ] 🛑 **Contact details** — confirm `804-332-6618` / `info@bluecapstrategies.com` are the
+      production values. (The live WP contact page carries placeholders `123-456-8912` /
+      `info@gmail.com`, deliberately **not** reproduced; see `docs/content-deviations.md`.)
+- [ ] 🛑 **AI training-bot policy** — `GPTBot` and `ClaudeBot` are currently **Allow: /** in
+      `public/robots.txt`, with a ready-to-uncomment `Disallow: /`. Search and answer crawlers
+      stay allowed either way. Decide before the site is publicly crawlable.
+- [ ] 🛑 **Brad Rodgers bio + credentials + portrait** — gates `Person` schema, the About
+      buildout, and principal-led E-E-A-T/GEO positioning. Core asset, still missing.
+- [ ] 🛑 **Real insights articles** — no fabricated or unreviewed AI content. Until one exists,
+      `/insights/` is an empty section in the nav (see §1.5).
+- [ ] 🛑 **Social profiles** — the live site only has per-page share links and empty `#` hrefs.
+      Provide real profiles or omit the footer block.
+- [ ] 🛑 **`/test/` page** — the live site has a staging duplicate of the homepage. Default: omit.
+- [ ] 🛑 **Proof modules / representative engagements** — no fabricated client work or dollar
+      figures. Provide real ones or ship without.
+
+---
+
+## 4. Post-cutover (not blocking)
+
+- [ ] Google Search Console + Bing Webmaster verified; `sitemap-index.xml` submitted.
+- [ ] Per-page 1200×630 OG images. Every page currently shares one fallback; `BaseLayout.astro:49`
+      emits a single `og:image`.
+- [ ] `robots` meta wired to a `noindex` frontmatter flag. `/thank-you/` is excluded from the
+      sitemap but emits **no** `noindex` meta tag.
+- [ ] `article:published_time` / `article:modified_time` OG tags on insights posts (absent today).
+- [ ] `lastmod` in the sitemap from content dates.
+- [ ] Lighthouse ≥ 95 on SEO / Perf / Best-Practices / Accessibility; watch CLS from unsized images.
+- [ ] Trailing-slash audit — `trailingSlash: "always"`; confirm canonicals and internal links agree.
+- [ ] Decommission plan for the WordPress/Bluehost instance (keep it live and reachable through
+      cutover + a rollback window; see the "keep old site live" constraint).
+
+---
+
+## 5. Already done — do not redo
+
+Recorded because the older checklists still list several of these as open.
+
+- [x] Full copy parity: home, about, services, focus areas, all 8 service detail pages, all 4
+      focus-area detail pages — verbatim live copy in Markdown content collections (ADR-0002).
+- [x] Real imagery imported to `public/assets/live-site/` (+ `/icons`); placeholders deleted.
+- [x] Zod `seoSchema` in `src/content.config.ts` — build **fails** on title/description/OG
+      violations.
+- [x] `scripts/seo-lint.mjs` + `npm run seo:lint` — single-h1, heading order, canonical, alt text,
+      JSON-LD parse. Green on 21 pages.
+- [x] `BreadcrumbList` schema (`src/utils/schema.ts`) applied across detail pages — the SEO doc
+      still lists this as to-do; it is done.
+- [x] Per-agent `public/robots.txt`; `@astrojs/sitemap` excluding `/thank-you/`.
+- [x] Deviations logged in `docs/content-deviations.md`.
+- [x] GitOps: Gitea Actions CI, Gitea Packages registry, ArgoCD Application, External-Secrets
+      registry pull auth over public HTTPS (ADR-0007, ADR-0008). Proven end-to-end in July —
+      currently stalled, see §1.2.
+- [x] Monitoring approach decided: reuse `monylog`, not Uptime Kuma (ADR-0004). The SEO doc's
+      "Uptime Kuma" line is superseded.

docs/seo-aeo-optimization.md

@@ -80,14 +80,20 @@ Already implemented in `src/layouts/BaseLayout.astro` (keep + enforce):
 
 To add/verify:
 
-- [ ] **Per-page OG images** — currently every page falls back to `/assets/water-strategy.jpg`.
-      Generate topic-specific OG images (services, focus areas, insights) at 1200×630.
-- [ ] **`robots` meta** wired to the `noindex` frontmatter flag (for `/thank-you/`, staging).
-- [ ] **`article:published_time` / `modified_time`** OG tags on insights posts.
-- [ ] **Breadcrumbs** (visible + `BreadcrumbList` schema) on services/focus/insights detail pages.
-- [ ] **Sitemap** already via `@astrojs/sitemap` (excludes `/thank-you/`); add `lastmod` from
+> **Tracked in [`launch-checklist.md`](./launch-checklist.md) §4.** The list that used to live
+> here went stale (breadcrumbs are done). This section keeps the *reference* detail below; the
+> live task state is in the launch checklist only.
+
+- **Per-page OG images** — every page currently falls back to one shared image
+  (`BaseLayout.astro` emits a single `og:image`). Generate topic-specific 1200×630 images for
+  services, focus areas, and insights.
+- **`robots` meta** wired to a `noindex` frontmatter flag (for `/thank-you/`, staging). Not wired
+  today — `/thank-you/` is sitemap-excluded but emits no `noindex`.
+- **`article:published_time` / `modified_time`** OG tags on insights posts.
+- **Breadcrumbs** — DONE. `BreadcrumbList` in `src/utils/schema.ts`, applied on detail pages.
+- **Sitemap** — via `@astrojs/sitemap`, excludes `/thank-you/`. Still to add: `lastmod` from
   content dates for insights.
-- [ ] **Trailing-slash consistency** — config uses `trailingSlash: "always"`; ensure canonical +
+- **Trailing-slash consistency** — config uses `trailingSlash: "always"`; ensure canonical +
   internal links all match to avoid duplicate-URL dilution.
 
 ## 3. Structured data (JSON-LD) matrix
@@ -214,16 +220,14 @@ Decided: **run PostHog and Umami side-by-side initially** and keep whichever fit
 
 ## 7. Pre-launch SEO checklist
 
-- [ ] Zod SEO schema in place; build fails on title/description/OG/canonical violations.
-- [ ] `scripts/seo-lint.mjs` (single H1, heading order, canonical, alt text, JSON-LD parses) green in CI.
-- [ ] Every page: unique title, description in window, canonical, OG image, valid JSON-LD.
-- [ ] Organization/WebSite/Service/Person/Article/Breadcrumb schema per §3, all visible-content-backed.
-- [ ] `robots.txt` per §5 (owner-decided); `sitemap-index.xml` correct; `/thank-you/` + staging `noindex`.
-- [ ] Canonical host decided (apex vs `www`) + redirect configured at the edge.
-- [ ] Search Console + Bing verified, sitemap submitted.
-- [ ] Analytics deployed (Umami) + consultation/CTA events firing.
-- [ ] Uptime Kuma + external check live and alerting.
-- [ ] Lighthouse ≥ 95 across the board; no CLS from unsized images.
+> **Moved.** This list is no longer maintained here — it duplicated, and disagreed with, reality
+> (it listed the Zod schema, `seo-lint`, and breadcrumbs as open when all three are done, and
+> called for Uptime Kuma, which ADR-0004 replaced with `monylog`).
+>
+> **The maintained checklist is [`launch-checklist.md`](./launch-checklist.md).** SEO items are
+> in §3 (owner decisions: canonical host, robots policy) and §4 (post-cutover: OG images,
+> `noindex` meta, article times, `lastmod`, Lighthouse, trailing slashes). What is already
+> complete is recorded in §5.
 
 ## Sources (verified 2026-07-15)