From 450bba8bea0ca8ae5d1aa4ca5a8d22b0e3a3bcb5 Mon Sep 17 00:00:00 2001 From: Brad Rodgers Date: Fri, 23 Oct 2026 14:19:52 -0400 Subject: [PATCH] docs: rewrite launch checklist as post-cutover status Records the live architecture, launch-day findings (Pangolin cert retry, health-check Host header, nginx 404 loop, CI rebase, homelab DNS interception, runner socket) and the remaining open items. --- docs/launch-checklist.md | 516 +++++++++++---------------------------- 1 file changed, 148 insertions(+), 368 deletions(-) diff --git a/docs/launch-checklist.md b/docs/launch-checklist.md index 738c2ee..a0e94a6 100644 --- a/docs/launch-checklist.md +++ b/docs/launch-checklist.md @@ -1,416 +1,196 @@ -# Launch Checklist — DNS Cutover to the Astro Rebuild +# Launch Checklist — Post-Cutover Status -**Single source of truth** for everything standing between today and pointing -`bluecapstrategies.com` at the homelab k3s deployment. - -Verified against the working tree, the cluster, and ArgoCD on **2026-09-18**. Supersedes the -task lists that were scattered across `planning/parity-checklist.md` (git-ignored, content-only) -and `docs/seo-aeo-optimization.md` §2/§7 (SEO-only, stale) — those now point here. +`www.bluecapstrategies.com` went **live on the Astro rebuild on 2026-10-23**. This file is now the +single source of truth for what is running, how it is wired, and what is left. The pre-launch +runbook it replaced is in git history (`git log -- docs/launch-checklist.md`, before 2026-10-23). Status legend: `[ ]` open · `[x]` done · `[?]` needs verification · 🛑 owner decision --- -## 0. As-built state (2026-09-18) +## 0. As-built state (2026-10-23) | Thing | State | |-------|-------| -| ArgoCD app `bcs-website` | Synced / Healthy | -| Pods | 2/2 Running, ns `bluecap-strategies` | -| Deployed image | `git.7tl-homelab.com/bmr_bluecap/bcs-website:fc64909f3df1` (built **2026-07-20**) | -| `k8s/kustomization.yaml` newTag | `fc64909f3df1` — same, so ArgoCD is correct; CI never bumped it | -| HEAD | `9f1cbca` (2026-08-03), pushed to `origin/main`, **never deployed** | -| Ingress | `bluecap-website` (traefik), hosts apex + `www`, **port 80 only, no TLS** | -| `npm run build` | green, 21 pages, 0 errors / 0 warnings | -| `npm run typecheck` | green, 28 files, 0/0/0 | -| `npm run seo:lint` | green, 21 pages clean | -| `npm run test:e2e` | **44 passed / 4 failed** (the 4 are stale assertions, see §2.1) | -| CI root cause | **found + fixed in-repo 2026-09-22** — see §1.2; needs a green run to confirm | -| Staging URL | **https://bcs.int.7tl-homelab.com** — LAN-only, valid cert (§2.6) | -| Public traffic path | Cloudflare → Pangolin VPS 23.94.111.133 → newt tunnel → k3s :443 (§1.1) | -| Pangolin | org `rhh-homelab`, site `7tl_lab` (newt, online), 23 resources, Pangolin 1.22.2 | -| Contact form | **working** — Formspree, verified end to end 2026-10-09 (§1.4) | -| Old WordPress site | untouched and serving — must stay that way until cutover | -| Gitea host | KVM VM `gitea-1` (10.66.15.22), **Debian 12**, docker compose — *not* in k3s | -| Gitea / act_runner | 1.27.3 / 3.4.2 — both healthy | -| CI job image | **`node:22-bookworm`** (Debian 12) as of 2026-09-22 — was EOL bullseye; see §1.2 | +| Public site | **https://www.bluecapstrategies.com** — canonical. Apex 301s to www, path preserved | +| Old WordPress / Bluehost | **Gone.** Access lost and it was returning 404 before cutover; nothing to roll back to | +| DNS | Cloudflare, **DNS-only (grey cloud)** for apex, `www`, `stats` → `23.94.111.133` | +| Edge | Pangolin VPS 23.94.111.133 (Traefik + CrowdSec), org `rhh-homelab`, site `7tl_lab` | +| Pangolin resources | `Bluecap-www`, `Bluecap-base`, `stats` — 3 https targets `10.66.15.30/.31/.32:443` each | +| Edge certs | Let's Encrypt HTTP-01 via Pangolin's `letsencrypt` resolver, issued 2026-10-23 | +| Cluster certs | cert-manager `letsencrypt-prod` DNS-01: `bluecap-website-tls` (www + apex), `bcs-int-tls` | +| ArgoCD | `bcs-website` and `umami` — Synced / Healthy | +| Deployed image | tracks `main` via CI tag-bump (`ci: deploy ` commits) | +| Staging | https://bcs.int.7tl-homelab.com — LAN only | +| Contact form | Formspree, verified end to end 2026-10-09 | +| Analytics | **Umami** (self-hosted, `stats.bluecapstrategies.com`) + **GA4** `G-Z273QL1F0N`, both verified recording 2026-10-23 | +| Monitoring | monylog blackbox probes on www + apex → Alertmanager → **Pushover** | +| Search | Google Search Console domain property verified; sitemap submitted and fetched by Googlebot | +| Mail | MXroute. SPF pass, DKIM pass (selector `x`), DMARC `p=none` — mail-tester 9.5/10 | +| `npm run build` / `typecheck` / `seo:lint` | green, 21 pages | +| `npm run test:e2e` | 44 passed / 4 failed — stale assertions, see §2.1 | -**The headline:** the public-facing deployment is ~7 weeks behind `main`. The Aug 3 exact-copy -work is not live. +### Traffic path +``` +visitor → Cloudflare DNS (DNS-only) → Pangolin VPS 23.94.111.133 (TLS #1, CrowdSec) + → gerbil/newt tunnel → k3s traefik 10.66.15.30/.31/.32 :443 (TLS #2) + → bluecap-website (nginx, 2 replicas) ns bluecap-strategies + → umami (+ umami-postgres) ns umami +``` --- -## 1. Cutover blockers +## 1. Open — do soon -Nothing below is optional. Cutover is unsafe until all of §1 is closed. +### 1.1 External uptime check 🛑 +monylog runs inside the homelab, so a homelab power or internet outage takes the site **and** its +monitor down together, and nothing pages. Needs a vantage point outside the homelab. +- [ ] Decide: hosted (UptimeRobot / Healthchecks.io) or Uptime Kuma on the Pangolin VPS. +- [ ] Probe `https://www.bluecapstrategies.com/` and route alerts to Pushover. -### 1.1 Cutover runbook — traffic path and TLS 🛑 hard blocker -Verified against the live Cloudflare zone, the Pangolin VPS and the cluster on **2026-10-09**. +### 1.2 Homelab intercepts port-53 DNS +Found 2026-10-23. Queries from the homelab — including k3s nodes — to *any* DNS server, even +Cloudflare's authoritative NS, are answered by a local recursive resolver (`dig @arely.ns.cloudflare.com` +returns `rd ra` with no `aa`). A name looked up before its record exists gets **NXDOMAIN cached for +30 minutes** (the zone's SOA negative TTL). -#### The actual traffic path -Public traffic cannot reach k3s directly; it has to transit the Pangolin VPS. +This, not Cloudflare, explains the 4-hour cert-manager DNS-01 stall of 2026-10-09 and the +`stats` NXDOMAIN at launch. Consequences: +- Pointing cert-manager at `1.1.1.1/8.8.8.8` (the fix previously suggested here) **will not help** + while port 53 is intercepted. +- To see true public DNS, query from the VPS or via DoH: + `curl -H 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=&type=A'` +- Create DNS records *before* anything in the homelab (cert-manager included) looks the name up. -``` -visitor → Cloudflare DNS (bluecapstrategies.com) - → Pangolin VPS 23.94.111.133 (Traefik v3.7.13 + CrowdSec, terminates TLS) - → gerbil/newt tunnel → site `7tl_lab` (org `rhh-homelab`) - → k3s traefik 10.66.15.30/.31/.32 :443 (second TLS hop) - → bluecap-website service -``` +- [ ] 🛑 Owner decision: leave as is, or exempt the k3s nodes from the redirect. The redirect rule + is not in `~/Homelab/mikrotik/*.rsc`; find it on the router (or Pi-hole/AdGuard, if any). +- [?] `umami-stats-tls` (internal hop for `stats`) was still pending on the stale NXDOMAIN at + 18:30 UTC on launch day; it should issue by itself once the cache expires. Visitor-facing + TLS for `stats` is Pangolin's and is fine. Check with `kubectl -n umami get certificate`. -**Two certificates are needed, from two different systems:** +### 1.3 Gitea runner loses Docker after a dockerd restart +`dockerd` on `gitea-1` runs with `live-restore=true`. When it restarts, containers keep running +but `act_runner`'s bind mount of `/var/run/docker.sock` points at the dead socket, so every CI +job fails in its first second with `Cannot connect to the Docker daemon`. Happened 2026-10-21 → +discovered 2026-10-23. +- **Workaround:** `sudo docker restart act_runner` on gitea-1. +- [ ] Durable fix in the `gitea` ansible role: disable live-restore on that VM, or restart the + runner after dockerd restarts (systemd `PartOf=`/`ExecStartPost` hook). -| Hop | Issued by | Challenge | Can pre-issue? | -|-----|-----------|-----------|----------------| -| edge (Pangolin Traefik) | its own `letsencrypt` resolver | **HTTP-01** on entrypoint `web` | **No** | -| internal (k3s ingress) | cert-manager `letsencrypt-prod` | **DNS-01** via Cloudflare | **Yes** | +### 1.4 `/insights/` ships empty, with developer-facing copy 🛑 +`src/pages/insights/index.astro` shows two cards written for a developer ("Publishing path", +"RSS support") and a hero calling the section a "Future publishing home". The section is in the +primary nav with zero articles. +- [ ] 🛑 Ship it or not. +- [ ] If yes: delete both note-cards and rewrite the hero as reader-facing copy. +- [ ] If no: drop it from the primary nav and exclude the route from the sitemap. -#### Corrections to earlier assumptions in this doc -- There is **one** Cloudflare token, not two. Pangolin's Traefik has **no** Cloudflare - credentials — it uses HTTP-01. Nothing to configure on the VPS for DNS. -- The "ordering problem SOLVED" claim was **half right**. cert-manager does use DNS-01 and can - pre-issue (proven on the staging host, §2.6). But **Pangolin cannot** — its - `certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint: web` means the edge cert is - only obtainable once DNS already points at the VPS. A short window at cutover is unavoidable - unless the edge resolver is converted to DNS-01, which would change renewal behaviour for all - 23 existing resources and is not worth it to save seconds. -- Earlier advice said "use DNS-only, matching your existing pattern." The BlueCap zone is in fact - **already fully proxied** (orange cloud) while the homelab zones are DNS-only. It is a real - decision — see below. - -#### Prerequisites (all invisible to the public; the old site keeps serving) -- [x] Cloudflare token now covers the zone — `7tl-homelab.com`, **`bluecapstrategies.com`**, - `rodgersweb.com` (done 2026-10-09). Token is DNS-edit-only, which is correct least privilege. -- [x] **k3s ingress — DONE 2026-10-09 (commit `6f80282`).** Both production hostnames added to - the `tls:` block as `bluecap-website-tls`. Certificate **Ready**, issued by Let's Encrypt - via DNS-01: `CN=www.bluecapstrategies.com`, SANs `bluecapstrategies.com` + - `www.bluecapstrategies.com`, valid to **2027-01-07**, auto-renewing. Verified the cluster - serves it for both names (`--resolve` straight to 10.66.15.30 → 200, correct cert, rebuild - content). The public site is untouched and still on Cloudflare/Bluehost throughout. - Both names are on one cert, so the canonical-host decision stays open. - - Issuance stalled ~4h: the TXT records were present in Cloudflare's API immediately but - Cloudflare's own authoritative nameservers returned NXDOMAIN for them for a long while. - It cleared on its own. If this recurs, cert-manager's DNS-01 self-check resolver is the - thing to look at — it is currently CoreDNS (`nameservers=["10.43.0.10:53"]`), and the - usual hardening is to point it at public resolvers instead: - `--dns01-recursive-nameservers-only=true --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53`. - Worth doing before cutover day so renewals do not inherit this fragility (§2.7). -- [ ] **Pangolin:** add `bluecapstrategies.com` as a domain under org **`rhh-homelab`** - (`orgDomains`), `certResolver=letsencrypt`. -- [ ] **Pangolin:** create the resource(s) with **three targets**, matching the established - pattern for every k3s-backed resource (Authentik, paperless, immich, dav all do this): - `10.66.15.30`, `10.66.15.31`, `10.66.15.32` — method **https**, port **443**. - Set the host header so k3s traefik routes on the real hostname. -- [ ] 🛑 **Check Cloudflare's SSL/TLS mode in the dashboard** (the API token cannot read zone - settings). If it is **Flexible**, Cloudflare talks HTTP to Pangolin, Pangolin redirects to - HTTPS, and you get a redirect loop. Must be **Full (strict)** if the record stays proxied. -- [ ] **Test before any DNS change:** `curl --resolve bluecapstrategies.com:443:23.94.111.133` - and confirm the full path serves the site. - -#### Pangolin resource targets (for step 2) -Three targets, matching every other k3s-backed resource: - -| Target | Method | Port | -|--------|--------|------| -| `10.66.15.30` | https | 443 | -| `10.66.15.31` | https | 443 | -| `10.66.15.32` | https | 443 | - -`k3-control-1/2/3`. Three gives node-level failover. Set the host header to the real hostname so -k3s traefik routes on it. The cluster now holds a valid cert for both production names, so the -https hop verifies properly. - -#### Proxied vs DNS-only — decide before cutover 🛑 -The zone is currently **proxied**: apex `A → 50.87.136.52 PROXIED`, `www CNAME → apex PROXIED`. - -- **DNS-only (recommended for cutover):** the exact path all 23 working resources use. CrowdSec - sees real client IPs as currently configured; HTTP-01 goes direct. Cost: no Cloudflare CDN/DDoS, - and the VPS IP becomes visible. -- **Stay proxied:** keeps CDN/DDoS and hides the origin, but CrowdSec will see Cloudflare IPs - unless it is configured to trust `CF-Connecting-IP` — bans would land on the wrong addresses or - on Cloudflare itself. Also requires Full (strict) per above. - -Revisit proxying after launch as a deliberate change with the CrowdSec real-IP config done -properly — not as part of cutover. - -#### Cutover -- [ ] Point apex + `www` at **23.94.111.133** in Cloudflare (replacing `50.87.136.52`). -- [ ] **Immediately** `curl https://bluecapstrategies.com` yourself to trigger Pangolin's HTTP-01 - issuance, before a real visitor hits an unissued cert. -- [ ] Verify a valid chain on both hostnames and that HTTP→HTTPS redirects. -- [ ] Apply the canonical-host decision (§1.6): redirect the non-canonical name. - -**Rollback:** set the A record back to `50.87.136.52`. All records are `ttl=1` (Cloudflare "Auto", -~300s for DNS-only), so propagation and rollback are both ~5 minutes — not hours. - -#### Mail is not affected -MX (`eagle.mxlogin.com` / `eagle-relay.mxlogin.com`) and `mail.bluecapstrategies.com` are **not -proxied** and are untouched by the cutover. MXroute keeps serving mail throughout. See -`mxroute-mail-setup` notes. - -- [ ] Minor cleanup (post-launch): `autoconfig` and `autodiscover` still point at Bluehost - (`50.87.136.52`) while mail is on MXroute — stale records that can hand mail clients the - wrong settings. - -### 1.2 CI/CD is stalled — no deploy since 2026-07-20 🛑 hard blocker -**ROOT CAUSE FOUND AND FIXED IN-REPO 2026-09-22.** Not a runner problem — the runner was online -and the run *did* execute. - -Evidence (Gitea 1.27.3 on `gitea-1`, sqlite `action_run` for repo_id 3): -- Run #6, commit `9f1cbca1b148`, created 2026-08-03 13:46:53, **status 2 = failure**, ran 13m - (13:47:02 → 14:00:11) on runner 59. -- No `9f1cbca1b148` entry in `package_version` → it died **at or before** the image push. -- Job logs are past retention (`/data/gitea/actions_log` no longer holds run 205), so the cause - was reproduced directly instead. - -**The cause:** the `Ensure docker + kustomize are available` step ran -`apt-get install -y docker.io`. The job container image is **`node:18-bullseye`** (Debian 11), -and bullseye went EOL — `debian-security` no longer serves those package versions. `apt-get -update` still succeeds, so this fails only at install time, with 404s on `docker.io`, -`containerd`, `systemd`, `libcap2`. Reproduced verbatim in a `node:18-bullseye` container on the -runner host. That timing matches exactly: last green run 2026-07-20, first failure 2026-08-03. - -**The fix (applied to `.gitea/workflows/ci.yml`):** fetch the docker CLI and kustomize as pinned -**static binaries** (`download.docker.com` / GitHub releases) instead of distro packages. Verified -in the real job environment: the static client installs, reports `server=28.1.1` through the -mounted socket, and `docker ps` succeeds. kustomize tarball and `git` are fine. - -Everything else on the runner path checked out and needs no change: -- [x] act_runner container up, `v3.4.2`, registered and polling (the daily 07:30 "connection - refused" entries are the Gitea restart window, harmless). -- [x] Runner scope **already fixed** — runner id 59 is `owner_id=0, repo_id=0` (instance-scoped), - so the old user-scoped problem is resolved. 58 stale duplicate runner rows remain; cosmetic, - prune with the role's `gitea_runner_prune` tag when convenient. -- [x] Runner host dockerd trusts `10.66.15.22:3000` as insecure — confirmed in `docker info`. -- [x] `config.yaml` pins jobs to `gitea_network` and passes the docker socket — correct. -- [x] `CI_TOKEN` repo secret present (repo_id 3). -- [x] `paths-ignore` did not swallow the push — the commit touched `src/**`. -- [x] `docker build` of the current tree succeeds locally, so the Dockerfile and lockfile are fine. - -Remaining: -- [ ] **Re-trigger CI and confirm a green run** — the fix is verified at the step level but the - full pipeline has not run end-to-end since July. Requires a push touching a non-ignored path. -- [ ] Confirm the tag-bump commit lands and ArgoCD syncs the new image. -- [ ] Do not cut DNS onto a stale image — confirm the deployed tag matches HEAD. -- [x] **CI job image upgraded 2026-09-22.** `gitea_runner_labels` in - `~/Homelab/ansible/roles/gitea/defaults/main.yml` changed `node:18-bullseye` → - **`node:22-bookworm`**; playbook run succeeded (ok=40 changed=8 failed=0). Runner **60** - registered and online, `owner_id=0` (instance-scoped), label confirmed in `/data/.runner`. - The job image now matches the host VM and k3s nodes (Debian 12) and the Node 22 that - workflows install anyway. - - Gotcha for next time: `--tags gitea_runner_prune` restricts the run to *only* that tag, so - it cannot be combined with `-e gitea_runner_force_reregister=true`. Two separate runs. - - Gotcha 2: `gitea_admin` had `must_change_password=1` (set during the 1.27.3 container - recreate on 2026-09-14), which made Gitea reject **all** admin API calls with 403 — not - 401, so credentials were fine. Cleared 2026-09-22 with - `docker exec -u git gitea gitea admin user must-change-password --unset gitea_admin` - (password unchanged; vault still valid). The admin API works again. -- [ ] Optional cleanup: prune the stale duplicate runner rows (59 of them now). Unblocked by the - admin-API fix above. `ansible-playbook -i inventory/hosts.ini gitea-deploy.yml --tags - gitea_runner_prune`. Purely cosmetic — deletes only runners the API reports offline. - -**Note on the two fixes:** CI now has *two* independent reasons to work — the workflow no longer -uses apt at all, and the job image is no longer EOL (bookworm still serves `docker.io` fine). The -static-binary approach remains the more durable of the two; keep it regardless. - -### 1.3 Developer-facing copy on the contact page ✅ FIXED 2026-09-23 -Two notes written for developers, not visitors, were being rendered: -- the form note explaining `PUBLIC_CONTACT_FORM_ENDPOINT`, and -- a "Discovery note" about the WordPress page holding placeholder contact values. - -Replaced the first with reader-facing copy and dropped the second (the deviation it described is -already recorded in `docs/content-deviations.md`). Commit `4e354d2`. - -### 1.4 Contact form endpoint — plumbing DONE 2026-09-23, needs your Formspree ID 🛑 -**Correction to the original entry:** it said to set `PUBLIC_CONTACT_FORM_ENDPOINT` in -`k8s/configmap.yaml`. That would never have worked. -- The Deployment has **no `env`/`envFrom`** referencing that ConfigMap — it is dead config. - (`PUBLIC_SITE_URL` is in the same state; it only appears to work because the default baked into - `astro.config.mjs` is already correct.) -- Even wired up it could not work: `output: "static"` means Astro **inlines** `PUBLIC_*` at image - build time, and the pod only runs nginx over pre-built HTML. There is no runtime knob. - -**Done (commit `af4cc4a`):** -- [x] `Dockerfile` takes `ARG PUBLIC_CONTACT_FORM_ENDPOINT` / `PUBLIC_SITE_URL` → `ENV` before build. -- [x] CI passes `--build-arg` from a `CONTACT_FORM_ENDPOINT` Actions secret. -- [x] ConfigMap rewritten as documentation, with a warning not to add `envFrom`. -- [x] Submit posts in the background via `fetch()` and redirects to our own `/thank-you/` instead - of navigating to the provider's page. -- [x] Failure path: stays on the page, re-enables the button, shows the phone/email fallback. -- [x] Honeypot `_gotcha` (Formspree treats it as spam), positioned off-screen rather than - `display:none` so bots that skip hidden fields still fill it. -- [x] e2e covers the real fetch path (mocked endpoint, asserts the payload) and the failure case. - -**Still needs you:** -- [ ] 🛑 Create the Formspree form for **info@bluecapstrategies.com** and confirm the address. -- [ ] 🛑 Add the endpoint as a Gitea repo Actions secret named **`CONTACT_FORM_ENDPOINT`** - (Settings → Actions → Secrets on `bmr_bluecap/bcs-website`), value - `https://formspree.io/f/`. -- [ ] Push any commit touching `src/**` so CI rebuilds with the secret baked in. -- [ ] Verify on the staging URL: submit a real inquiry, confirm it arrives, confirm the browser - lands on `/thank-you/`. -- [ ] Confirm the destination inbox is actually monitored. A working form delivering into an - unwatched inbox is its own kind of failure. - -**Degradation if the secret is missing:** the build still succeeds and the form falls back to -opening a `mailto:` draft. Survivable, but it fails silently for webmail users with no OS mail -client — so treat a missing secret as a launch blocker, not a warning. - -### 1.5 Draft insight was being published and indexed ✅ FIXED 2026-09-23 -`src/pages/insights/[slug].astro` called `getCollection("insights")` unfiltered, so the -`draft: true` placeholder built to `/insights/example-draft/` **and** appeared in -`sitemap-0.xml`. The schema declared the flag and `rss.xml.ts` already honoured it — only the page -route ignored it. Filtered in `getStaticPaths`, matching the RSS pattern. Build is 21 → 20 pages -and the URL is gone from the sitemap. Commit `4e354d2`. - -- [ ] 🛑 **Still open (owner):** whether `/insights/` should ship at all with zero real articles. - It is currently in the primary nav. See also §2.5. - -### 1.6 Canonical host is undecided 🛑 owner -The ingress answers on **both** apex and `www`; `astro.config.mjs`, `robots.txt`, and the -ConfigMap all hardcode `https://www.bluecapstrategies.com`. Serving identical content on two -hosts with no redirect splits SEO signal from day one. -- [ ] Owner picks apex or `www`. -- [ ] Configure a 301 from the loser to the winner (traefik middleware or edge). -- [ ] Reconcile `PUBLIC_SITE_URL`, canonical tags, `robots.txt` Sitemap line, and the ingress. +### 1.5 Privacy policy review 🛑 +`/privacy/` went live 2026-10-23, drafted from what the site actually does (contact form via +Formspree, Umami, GA4 with cookies, server/security logs). +- [ ] 🛑 Owner review of the text. It is a legal document. +- [ ] 🛑 Consent banner: GA4 sets cookies. Strictly, EU/UK visitors need opt-in consent; most + US-focused small-business sites run with a privacy policy only. Owner's call. Umami needs + neither. +- Anything added to `src/components/Analytics.astro` must be disclosed on `/privacy/`. --- -## 2. Should-fix before cutover - -### 2.5 `/insights/` index still renders developer-facing copy -Found while fixing §1.5. `src/pages/insights/index.astro` shows visitors two cards written for a -developer, not a reader: -- *"Publishing path"* — "Future posts can be added as Markdown or MDX in - `src/content/insights` with typed frontmatter for title, description, publish date…" -- *"RSS support"* — "The feed is ready for future article imports even though no public posts are - published yet." - -The hero also announces the section as a "Future publishing home", i.e. it tells visitors the -section is empty. Same defect class as §1.3, but left alone deliberately: what the page *should* -say depends on the owner's call in §1.5 on whether `/insights/` ships at all. - -- [ ] If `/insights/` ships: delete both note-cards and rewrite the hero as reader-facing copy. -- [ ] If it does not: drop it from the primary nav and exclude the route from the sitemap. - -### 2.7 cert-manager DNS-01 self-check uses CoreDNS -`cert-manager` logs `configured acme dns01 nameservers nameservers=["10.43.0.10:53"]`, i.e. it -validates DNS-01 propagation through the cluster's own CoreDNS rather than public resolvers. That -makes issuance sensitive to internal caching — including negative caching of the `_acme-challenge` -name it is waiting on. - -- [ ] Point it at public resolvers before cutover: - `--dns01-recursive-nameservers-only=true --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53`. - Not confirmed as the cause of the 4h stall on 2026-10-09 (Cloudflare's authoritative NS were - themselves returning NXDOMAIN for records its API reported as present), but it is the - standard hardening and removes one variable from certificate renewal. - -### 2.6 Staging URL ✅ LIVE 2026-09-23 -**https://bcs.int.7tl-homelab.com** — review the rebuild in a browser, valid cert, no warning. - -- LAN-only: `*.int.7tl-homelab.com` resolves to the internal traefik LB (10.66.15.30). Not - reachable from the internet; the real domain is untouched and WordPress still serves it. -- Cert issued by Let's Encrypt via cert-manager DNS-01 in ~75s, `CN=bcs.int.7tl-homelab.com`, - valid to 2026-12-21, auto-renewing. -- All 9 public routes verified 200. -- Commit `171c8c9`. Note it also corrected the issuer name in `k8s/ingress.yaml`: the commented - placeholder said `letsencrypt-production`, but the ClusterIssuer is **`letsencrypt-prod`** — - uncommenting it as written at cutover would have silently failed to issue. -- ArgoCD may take a few minutes to poll a push; force it with - `kubectl annotate application bcs-website -n argocd argocd.argoproj.io/refresh=hard --overwrite`. +## 2. Should fix ### 2.1 e2e suite is red — stale assertions, not a regression -4 of 46 tests fail. Both failures assert **invented copy that the exact-copy work correctly -removed**: -- `tests/site.spec.ts:67` waits for a link named `Explore services`; the homepage CTA is now - `All services` (matches live WP). -- `tests/site.spec.ts:101` waits for a heading matching `/Focused expertise/`, which no longer - exists on `/focus-areas/`. - -Navigation itself works. The tests are wrong, not the site. -- [ ] Update the two tests to assert current copy. -- [ ] Get back to 46/46 so the suite is trustworthy at cutover. +4 of 48 fail (2 tests × desktop/mobile). Both assert **copy the exact-copy work correctly removed**: +- `tests/site.spec.ts:63` waits for a link named `Explore services`; the homepage CTA is `All services`. +- `tests/site.spec.ts:93` waits for a heading matching `/Focused expertise/`, gone from `/focus-areas/`. +- [ ] Update both tests to current copy; get back to green so the suite is trustworthy. ### 2.2 Repo hygiene -Commit `9f1cbca` committed ~50 `.playwright-mcp/` screenshots and YAML snapshots plus a 2.8 MB +Commit `9f1cbca` committed ~50 `.playwright-mcp/` screenshots/snapshots plus a 2.8 MB `home-full.png` at the repo root. - [ ] Add `.playwright-mcp/` to `.gitignore`; remove the tracked artifacts and `home-full.png`. -### 2.3 Uptime monitoring is not deployed -Per ADR-0004 the plan is to reuse the homelab `monylog` stack (not Uptime Kuma). Pod metrics and -nginx logs already flow via Alloy; the external-availability piece was never added. -- [?] Verify current state on monylog-1 — the items below are from notes, not re-verified today. -- [ ] Add `blackbox_exporter` + `web-bluecap.yml` alert rules; re-run the monylog ansible role to - reload Prometheus. -- [ ] Confirm an external HTTPS check fires to Pushover on a simulated outage. **Do this before - cutover** — the first hours on new infrastructure are when you most need paging. - -### 2.4 Analytics is a no-op -`src/components/Analytics.astro` is wired into `BaseLayout` but gated on `PUBLIC_` env vars that -are absent from `k8s/configmap.yaml`, so nothing is collected. -- [ ] Decide whether to launch with analytics or add it after (ADR-0005: PostHog + Umami trial). -- [ ] **PostHog requires a privacy/consent notice before production use** — that gates it, not - the code. -- [ ] Set the env vars and confirm pageviews plus a consultation/CTA event. +### 2.3 DMARC tightening +- [ ] Tighten DMARC from `p=none` to `p=quarantine` after a few weeks of clean reports + to info@bluecapstrategies.com. --- ## 3. Owner decisions (Brad) -These are yours, not mine. Several block content that is otherwise ready. - -- [ ] 🛑 **Canonical host** — apex vs `www` (also §1.6). -- [ ] 🛑 **Contact details** — confirm `804-332-6618` / `info@bluecapstrategies.com` are the - production values. (The live WP contact page carries placeholders `123-456-8912` / - `info@gmail.com`, deliberately **not** reproduced; see `docs/content-deviations.md`.) -- [ ] 🛑 **AI training-bot policy** — `GPTBot` and `ClaudeBot` are currently **Allow: /** in - `public/robots.txt`, with a ready-to-uncomment `Disallow: /`. Search and answer crawlers - stay allowed either way. Decide before the site is publicly crawlable. +- [x] **Canonical host** — `www` (2026-10-23). Apex 301s via Traefik middleware in `k8s/ingress.yaml`. +- [x] **Analytics** — Umami + GA4 side by side (2026-10-23). +- [ ] 🛑 **Contact details** — confirm `804-332-6618` / `info@bluecapstrategies.com` are final. +- [ ] 🛑 **AI training-bot policy** — `GPTBot` and `ClaudeBot` are **Allow: /** in + `public/robots.txt`, with a ready-to-uncomment `Disallow: /`. The site is now crawlable. - [ ] 🛑 **Brad Rodgers bio + credentials + portrait** — gates `Person` schema, the About - buildout, and principal-led E-E-A-T/GEO positioning. Core asset, still missing. -- [ ] 🛑 **Real insights articles** — no fabricated or unreviewed AI content. Until one exists, - `/insights/` is an empty section in the nav (see §1.5). -- [ ] 🛑 **Social profiles** — the live site only has per-page share links and empty `#` hrefs. - Provide real profiles or omit the footer block. -- [ ] 🛑 **`/test/` page** — the live site has a staging duplicate of the homepage. Default: omit. -- [ ] 🛑 **Proof modules / representative engagements** — no fabricated client work or dollar - figures. Provide real ones or ship without. + buildout, and principal-led E-E-A-T/GEO positioning. +- [ ] 🛑 **Real insights articles** — no fabricated or unreviewed AI content (see §1.4). +- [ ] 🛑 **Social profiles** — provide real profiles or omit the footer block. +- [ ] 🛑 **Proof modules / representative engagements** — no fabricated client work. --- -## 4. Post-cutover (not blocking) +## 4. Post-launch improvements (not blocking) -- [ ] Google Search Console + Bing Webmaster verified; `sitemap-index.xml` submitted. -- [ ] Per-page 1200×630 OG images. Every page currently shares one fallback; `BaseLayout.astro:49` - emits a single `og:image`. +- [ ] Bing Webmaster Tools — import from Search Console (covers Bing, DuckDuckGo, ChatGPT search). +- [ ] Check Search Console in ~1 week: sitemap "Success", pages indexed, no 5xx in Coverage. +- [ ] Per-page 1200×630 OG images; `BaseLayout.astro` emits one shared fallback `og:image`. - [ ] `robots` meta wired to a `noindex` frontmatter flag. `/thank-you/` is excluded from the - sitemap but emits **no** `noindex` meta tag. -- [ ] `article:published_time` / `article:modified_time` OG tags on insights posts (absent today). + sitemap but emits no `noindex` meta tag. +- [ ] `article:published_time` / `article:modified_time` OG tags on insights posts. - [ ] `lastmod` in the sitemap from content dates. -- [ ] Lighthouse ≥ 95 on SEO / Perf / Best-Practices / Accessibility; watch CLS from unsized images. -- [ ] Trailing-slash audit — `trailingSlash: "always"`; confirm canonicals and internal links agree. -- [ ] Decommission plan for the WordPress/Bluehost instance (keep it live and reachable through - cutover + a rollback window; see the "keep old site live" constraint). +- [ ] Lighthouse ≥ 95 on SEO / Perf / Best-Practices / Accessibility. +- [ ] Umami/GA custom event for contact-form submits (today the `/thank-you/` pageview is the + conversion signal). +- [ ] Optional: move `bluecapstrategies.com` into Pangolin's `config.yml` `domains:` so it shows + "Config Managed" like the other three (needs a Pangolin restart; purely cosmetic). +- [ ] Deferred: VPS static failover for homelab outages (see memory note / earlier discussion). --- -## 5. Already done — do not redo +## 5. Launch-day record (2026-10-23) -Recorded because the older checklists still list several of these as open. +What happened, so it does not have to be rediscovered. -- [x] Full copy parity: home, about, services, focus areas, all 8 service detail pages, all 4 - focus-area detail pages — verbatim live copy in Markdown content collections (ADR-0002). -- [x] Real imagery imported to `public/assets/live-site/` (+ `/icons`); placeholders deleted. -- [x] Zod `seoSchema` in `src/content.config.ts` — build **fails** on title/description/OG - violations. -- [x] `scripts/seo-lint.mjs` + `npm run seo:lint` — single-h1, heading order, canonical, alt text, - JSON-LD parse. Green on 21 pages. -- [x] `BreadcrumbList` schema (`src/utils/schema.ts`) applied across detail pages — the SEO doc - still lists this as to-do; it is done. +- **Pangolin certs need a real config change to retry.** HTTP-01 failed while `www` was still + proxied (Let's Encrypt reached Cloudflare's IPv6, got 404) and hit the 5-failures/hour limit. + Clicking Save with no changes does nothing; toggling **Enable Resource** off/on (saving between) + re-adds the router and triggers issuance. +- **Pangolin health checks** probe the target IP with no hostname, which k3s traefik answers with + 404. Set the health check's Host header and TLS server name to the resource's hostname, path + `/` (or `/api/heartbeat` for Umami), expect 200. +- **nginx 404 loop** — `error_page` pointed at `/404/index.html` but Astro emits `404.html`, so + every unknown URL returned 500. Fixed in `d1c1726`; unknown URLs now return a real 404. +- **CI tag-bump push rejected** when `main` moved during a build (or on a re-run of an older + commit). The step now rebases onto `origin/main` and retries (`93415c6`). Do not "re-run" old + runs to deploy — push instead. +- **Cloudflare cleanup** — removed all Bluehost/cPanel leftovers (autoconfig/autodiscover, + cpanel/whm/webdisk/ftp, cal/carddav SRV, `localhost`), replaced Bluehost's `default._domainkey` + with MXroute's `x._domainkey`, added `_dmarc`. +- **Monitoring safety** — the monylog playbook only renders Pushover routing when `VAULT_TOKEN` + is set; without it Alertmanager silently falls back to ntfy. Always run it with the token. +- **Umami** ships with `admin` / `umami`. It was brought up LAN-only + (`umami.int.7tl-homelab.com`), the password changed, and only then exposed via Pangolin. + Manifests live in `homelab-gitops` (`apps/umami.yaml`, `analytics/umami/`); credentials in + OpenBao `cluster/umami`. + +--- + +## 6. Already done — do not redo + +- [x] Full copy parity with the WordPress site, verbatim in content collections (ADR-0002). +- [x] Real imagery in `public/assets/live-site/`; placeholders deleted. +- [x] Zod `seoSchema`; `scripts/seo-lint.mjs` (`npm run seo:lint`). +- [x] `BreadcrumbList` schema across detail pages. - [x] Per-agent `public/robots.txt`; `@astrojs/sitemap` excluding `/thank-you/`. -- [x] Deviations logged in `docs/content-deviations.md`. -- [x] GitOps: Gitea Actions CI, Gitea Packages registry, ArgoCD Application, External-Secrets - registry pull auth over public HTTPS (ADR-0007, ADR-0008). Proven end-to-end in July — - currently stalled, see §1.2. -- [x] Monitoring approach decided: reuse `monylog`, not Uptime Kuma (ADR-0004). The SEO doc's - "Uptime Kuma" line is superseded. +- [x] Draft insights excluded from routes and sitemap. +- [x] Contact form: build-time Formspree endpoint, background submit, honeypot, failure fallback. +- [x] GitOps: Gitea Actions CI → Gitea Packages → tag-bump commit → ArgoCD (ADR-0007, ADR-0008). +- [x] CI job image `node:22-bookworm`; docker CLI + kustomize as pinned static binaries. +- [x] Production TLS in-cluster (DNS-01) and at the edge (HTTP-01). +- [x] Canonical www + apex 301; real 404s. +- [x] Monitoring: blackbox probes + `WebsiteDown` / `WebsiteCertExpiringSoon` / `WebsiteSlow` / + `BlackboxExporterDown` rules in the monylog role (ADR-0004: monylog, not Uptime Kuma). +- [x] Analytics: Umami + GA4, production-host only; `/privacy/` page. +- [x] Search Console verified; sitemap submitted. +- [x] Mail authentication: SPF, DKIM, DMARC passing.