bcs-website/docs/launch-checklist.md

Launch Checklist — DNS Cutover to the Astro Rebuild

Single source of truth for everything standing between today and pointing bluecapstrategies.com at the homelab k3s deployment.

Verified against the working tree, the cluster, and ArgoCD on 2026-09-18. Supersedes the task lists that were scattered across planning/parity-checklist.md (git-ignored, content-only) and docs/seo-aeo-optimization.md §2/§7 (SEO-only, stale) — those now point here.

Status legend: [ ] open · [x] done · [?] needs verification · 🛑 owner decision

---

0. As-built state (2026-09-18)

Thing	State
ArgoCD app bcs-website	Synced / Healthy
Pods	2/2 Running, ns bluecap-strategies
Deployed image	git.7tl-homelab.com/bmr_bluecap/bcs-website:fc64909f3df1 (built 2026-07-20)
k8s/kustomization.yaml newTag	fc64909f3df1 — same, so ArgoCD is correct; CI never bumped it
HEAD	9f1cbca (2026-08-03), pushed to origin/main, never deployed
Ingress	bluecap-website (traefik), hosts apex + www, port 80 only, no TLS
npm run build	green, 21 pages, 0 errors / 0 warnings
npm run typecheck	green, 28 files, 0/0/0
npm run seo:lint	green, 21 pages clean
npm run test:e2e	44 passed / 4 failed (the 4 are stale assertions, see §2.1)
CI root cause	found + fixed in-repo 2026-09-22 — see §1.2; needs a green run to confirm
Staging URL	https://bcs.int.7tl-homelab.com — LAN-only, valid cert (§2.6)
Public traffic path	Cloudflare → Pangolin VPS 23.94.111.133 → newt tunnel → k3s :443 (§1.1)
Pangolin	org rhh-homelab, site 7tl_lab (newt, online), 23 resources, Pangolin 1.22.2
Contact form	working — Formspree, verified end to end 2026-10-09 (§1.4)
Old WordPress site	untouched and serving — must stay that way until cutover
Gitea host	KVM VM gitea-1 (10.66.15.22), Debian 12, docker compose — not in k3s
Gitea / act_runner	1.27.3 / 3.4.2 — both healthy
CI job image	node:22-bookworm (Debian 12) as of 2026-09-22 — was EOL bullseye; see §1.2

The headline: the public-facing deployment is ~7 weeks behind main. The Aug 3 exact-copy work is not live.

---

1. Cutover blockers

Nothing below is optional. Cutover is unsafe until all of §1 is closed.

1.1 Cutover runbook — traffic path and TLS 🛑 hard blocker

Verified against the live Cloudflare zone, the Pangolin VPS and the cluster on 2026-10-09.

The actual traffic path

Public traffic cannot reach k3s directly; it has to transit the Pangolin VPS.

visitor → Cloudflare DNS (bluecapstrategies.com)
        → Pangolin VPS 23.94.111.133  (Traefik v3.7.13 + CrowdSec, terminates TLS)
        → gerbil/newt tunnel  → site `7tl_lab` (org `rhh-homelab`)
        → k3s traefik 10.66.15.30/.31/.32 :443  (second TLS hop)
        → bluecap-website service

Two certificates are needed, from two different systems:

Hop	Issued by	Challenge	Can pre-issue?
edge (Pangolin Traefik)	its own letsencrypt resolver	HTTP-01 on entrypoint web	No
internal (k3s ingress)	cert-manager letsencrypt-prod	DNS-01 via Cloudflare	Yes

Corrections to earlier assumptions in this doc

• There is one Cloudflare token, not two. Pangolin's Traefik has no Cloudflare credentials — it uses HTTP-01. Nothing to configure on the VPS for DNS.
• The "ordering problem SOLVED" claim was half right. cert-manager does use DNS-01 and can pre-issue (proven on the staging host, §2.6). But Pangolin cannot — its certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint: web means the edge cert is only obtainable once DNS already points at the VPS. A short window at cutover is unavoidable unless the edge resolver is converted to DNS-01, which would change renewal behaviour for all 23 existing resources and is not worth it to save seconds.
• Earlier advice said "use DNS-only, matching your existing pattern." The BlueCap zone is in fact already fully proxied (orange cloud) while the homelab zones are DNS-only. It is a real decision — see below.

Prerequisites (all invisible to the public; the old site keeps serving)

• Cloudflare token now covers the zone — 7tl-homelab.com, bluecapstrategies.com, rodgersweb.com (done 2026-10-09). Token is DNS-edit-only, which is correct least privilege.
• k3s ingress — DONE 2026-10-09 (commit 6f80282). Both production hostnames added to the tls: block as bluecap-website-tls. Certificate Ready, issued by Let's Encrypt via DNS-01: CN=www.bluecapstrategies.com, SANs bluecapstrategies.com + www.bluecapstrategies.com, valid to 2027-01-07, auto-renewing. Verified the cluster serves it for both names (--resolve straight to 10.66.15.30 → 200, correct cert, rebuild content). The public site is untouched and still on Cloudflare/Bluehost throughout. Both names are on one cert, so the canonical-host decision stays open. - Issuance stalled ~4h: the TXT records were present in Cloudflare's API immediately but Cloudflare's own authoritative nameservers returned NXDOMAIN for them for a long while. It cleared on its own. If this recurs, cert-manager's DNS-01 self-check resolver is the thing to look at — it is currently CoreDNS (nameservers=["10.43.0.10:53"]), and the usual hardening is to point it at public resolvers instead: --dns01-recursive-nameservers-only=true --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53. Worth doing before cutover day so renewals do not inherit this fragility (§2.7).
• Pangolin: add bluecapstrategies.com as a domain under org rhh-homelab (orgDomains), certResolver=letsencrypt.
• Pangolin: create the resource(s) with three targets, matching the established pattern for every k3s-backed resource (Authentik, paperless, immich, dav all do this): 10.66.15.30, 10.66.15.31, 10.66.15.32 — method https, port 443. Set the host header so k3s traefik routes on the real hostname.
• 🛑 Check Cloudflare's SSL/TLS mode in the dashboard (the API token cannot read zone settings). If it is Flexible, Cloudflare talks HTTP to Pangolin, Pangolin redirects to HTTPS, and you get a redirect loop. Must be Full (strict) if the record stays proxied.
• Test before any DNS change: curl --resolve bluecapstrategies.com:443:23.94.111.133 and confirm the full path serves the site.

Pangolin resource targets (for step 2)

Three targets, matching every other k3s-backed resource:

Target	Method	Port
10.66.15.30	https	443
10.66.15.31	https	443
10.66.15.32	https	443

k3-control-1/2/3. Three gives node-level failover. Set the host header to the real hostname so k3s traefik routes on it. The cluster now holds a valid cert for both production names, so the https hop verifies properly.

Proxied vs DNS-only — decide before cutover 🛑

The zone is currently proxied: apex A → 50.87.136.52 PROXIED, www CNAME → apex PROXIED.

• DNS-only (recommended for cutover): the exact path all 23 working resources use. CrowdSec sees real client IPs as currently configured; HTTP-01 goes direct. Cost: no Cloudflare CDN/DDoS, and the VPS IP becomes visible.
• Stay proxied: keeps CDN/DDoS and hides the origin, but CrowdSec will see Cloudflare IPs unless it is configured to trust CF-Connecting-IP — bans would land on the wrong addresses or on Cloudflare itself. Also requires Full (strict) per above.

Revisit proxying after launch as a deliberate change with the CrowdSec real-IP config done properly — not as part of cutover.

Cutover

• Point apex + www at 23.94.111.133 in Cloudflare (replacing 50.87.136.52).
• Immediately curl https://bluecapstrategies.com yourself to trigger Pangolin's HTTP-01 issuance, before a real visitor hits an unissued cert.
• Verify a valid chain on both hostnames and that HTTP→HTTPS redirects.
• Apply the canonical-host decision (§1.6): redirect the non-canonical name.

Rollback: set the A record back to 50.87.136.52. All records are ttl=1 (Cloudflare "Auto", ~300s for DNS-only), so propagation and rollback are both ~5 minutes — not hours.

Mail is not affected

MX (eagle.mxlogin.com / eagle-relay.mxlogin.com) and mail.bluecapstrategies.com are not proxied and are untouched by the cutover. MXroute keeps serving mail throughout. See mxroute-mail-setup notes.

• Minor cleanup (post-launch): autoconfig and autodiscover still point at Bluehost (50.87.136.52) while mail is on MXroute — stale records that can hand mail clients the wrong settings.

1.2 CI/CD is stalled — no deploy since 2026-07-20 🛑 hard blocker

ROOT CAUSE FOUND AND FIXED IN-REPO 2026-09-22. Not a runner problem — the runner was online and the run did execute.

Evidence (Gitea 1.27.3 on gitea-1, sqlite action_run for repo_id 3):

• Run #6, commit 9f1cbca1b148, created 2026-08-03 13:46:53, status 2 = failure, ran 13m (13:47:02 → 14:00:11) on runner 59.
• No 9f1cbca1b148 entry in package_version → it died at or before the image push.
• Job logs are past retention (/data/gitea/actions_log no longer holds run 205), so the cause was reproduced directly instead.

The cause: the Ensure docker + kustomize are available step ran apt-get install -y docker.io. The job container image is node:18-bullseye (Debian 11), and bullseye went EOL — debian-security no longer serves those package versions. apt-get update still succeeds, so this fails only at install time, with 404s on docker.io, containerd, systemd, libcap2. Reproduced verbatim in a node:18-bullseye container on the runner host. That timing matches exactly: last green run 2026-07-20, first failure 2026-08-03.

The fix (applied to .gitea/workflows/ci.yml): fetch the docker CLI and kustomize as pinned static binaries (download.docker.com / GitHub releases) instead of distro packages. Verified in the real job environment: the static client installs, reports server=28.1.1 through the mounted socket, and docker ps succeeds. kustomize tarball and git are fine.

Everything else on the runner path checked out and needs no change:

• act_runner container up, v3.4.2, registered and polling (the daily 07:30 "connection refused" entries are the Gitea restart window, harmless).
• Runner scope already fixed — runner id 59 is owner_id=0, repo_id=0 (instance-scoped), so the old user-scoped problem is resolved. 58 stale duplicate runner rows remain; cosmetic, prune with the role's gitea_runner_prune tag when convenient.
• Runner host dockerd trusts 10.66.15.22:3000 as insecure — confirmed in docker info.
• config.yaml pins jobs to gitea_network and passes the docker socket — correct.
• CI_TOKEN repo secret present (repo_id 3).
• paths-ignore did not swallow the push — the commit touched src/**.
• docker build of the current tree succeeds locally, so the Dockerfile and lockfile are fine.

Remaining:

• Re-trigger CI and confirm a green run — the fix is verified at the step level but the full pipeline has not run end-to-end since July. Requires a push touching a non-ignored path.
• Confirm the tag-bump commit lands and ArgoCD syncs the new image.
• Do not cut DNS onto a stale image — confirm the deployed tag matches HEAD.
• CI job image upgraded 2026-09-22. gitea_runner_labels in ~/Homelab/ansible/roles/gitea/defaults/main.yml changed node:18-bullseye → node:22-bookworm; playbook run succeeded (ok=40 changed=8 failed=0). Runner 60 registered and online, owner_id=0 (instance-scoped), label confirmed in /data/.runner. The job image now matches the host VM and k3s nodes (Debian 12) and the Node 22 that workflows install anyway. - Gotcha for next time: --tags gitea_runner_prune restricts the run to only that tag, so it cannot be combined with -e gitea_runner_force_reregister=true. Two separate runs. - Gotcha 2: gitea_admin had must_change_password=1 (set during the 1.27.3 container recreate on 2026-09-14), which made Gitea reject all admin API calls with 403 — not 401, so credentials were fine. Cleared 2026-09-22 with docker exec -u git gitea gitea admin user must-change-password --unset gitea_admin (password unchanged; vault still valid). The admin API works again.
• Optional cleanup: prune the stale duplicate runner rows (59 of them now). Unblocked by the admin-API fix above. ansible-playbook -i inventory/hosts.ini gitea-deploy.yml --tags gitea_runner_prune. Purely cosmetic — deletes only runners the API reports offline.

Note on the two fixes: CI now has two independent reasons to work — the workflow no longer uses apt at all, and the job image is no longer EOL (bookworm still serves docker.io fine). The static-binary approach remains the more durable of the two; keep it regardless.

1.3 Developer-facing copy on the contact page ✅ FIXED 2026-09-23

Two notes written for developers, not visitors, were being rendered:

• the form note explaining PUBLIC_CONTACT_FORM_ENDPOINT, and
• a "Discovery note" about the WordPress page holding placeholder contact values.

Replaced the first with reader-facing copy and dropped the second (the deviation it described is already recorded in docs/content-deviations.md). Commit 4e354d2.

1.4 Contact form endpoint — plumbing DONE 2026-09-23, needs your Formspree ID 🛑

Correction to the original entry: it said to set PUBLIC_CONTACT_FORM_ENDPOINT in k8s/configmap.yaml. That would never have worked.

• The Deployment has no env/envFrom referencing that ConfigMap — it is dead config. (PUBLIC_SITE_URL is in the same state; it only appears to work because the default baked into astro.config.mjs is already correct.)
• Even wired up it could not work: output: "static" means Astro inlines PUBLIC_* at image build time, and the pod only runs nginx over pre-built HTML. There is no runtime knob.

Done (commit af4cc4a):

• Dockerfile takes ARG PUBLIC_CONTACT_FORM_ENDPOINT / PUBLIC_SITE_URL → ENV before build.
• CI passes --build-arg from a CONTACT_FORM_ENDPOINT Actions secret.
• ConfigMap rewritten as documentation, with a warning not to add envFrom.
• Submit posts in the background via fetch() and redirects to our own /thank-you/ instead of navigating to the provider's page.
• Failure path: stays on the page, re-enables the button, shows the phone/email fallback.
• Honeypot _gotcha (Formspree treats it as spam), positioned off-screen rather than display:none so bots that skip hidden fields still fill it.
• e2e covers the real fetch path (mocked endpoint, asserts the payload) and the failure case.

Still needs you:

• 🛑 Create the Formspree form for info@bluecapstrategies.com and confirm the address.
• 🛑 Add the endpoint as a Gitea repo Actions secret named CONTACT_FORM_ENDPOINT (Settings → Actions → Secrets on bmr_bluecap/bcs-website), value https://formspree.io/f/<your-id>.
• Push any commit touching src/** so CI rebuilds with the secret baked in.
• Verify on the staging URL: submit a real inquiry, confirm it arrives, confirm the browser lands on /thank-you/.
• Confirm the destination inbox is actually monitored. A working form delivering into an unwatched inbox is its own kind of failure.

Degradation if the secret is missing: the build still succeeds and the form falls back to opening a mailto: draft. Survivable, but it fails silently for webmail users with no OS mail client — so treat a missing secret as a launch blocker, not a warning.

1.5 Draft insight was being published and indexed ✅ FIXED 2026-09-23

src/pages/insights/[slug].astro called getCollection("insights") unfiltered, so the draft: true placeholder built to /insights/example-draft/ and appeared in sitemap-0.xml. The schema declared the flag and rss.xml.ts already honoured it — only the page route ignored it. Filtered in getStaticPaths, matching the RSS pattern. Build is 21 → 20 pages and the URL is gone from the sitemap. Commit 4e354d2.

• 🛑 Still open (owner): whether /insights/ should ship at all with zero real articles. It is currently in the primary nav. See also §2.5.

1.6 Canonical host is undecided 🛑 owner

The ingress answers on both apex and www; astro.config.mjs, robots.txt, and the ConfigMap all hardcode https://www.bluecapstrategies.com. Serving identical content on two hosts with no redirect splits SEO signal from day one.

• Owner picks apex or www.
• Configure a 301 from the loser to the winner (traefik middleware or edge).
• Reconcile PUBLIC_SITE_URL, canonical tags, robots.txt Sitemap line, and the ingress.

---

2. Should-fix before cutover

2.5 /insights/ index still renders developer-facing copy

Found while fixing §1.5. src/pages/insights/index.astro shows visitors two cards written for a developer, not a reader:

• "Publishing path" — "Future posts can be added as Markdown or MDX in src/content/insights with typed frontmatter for title, description, publish date…"
• "RSS support" — "The feed is ready for future article imports even though no public posts are published yet."

The hero also announces the section as a "Future publishing home", i.e. it tells visitors the section is empty. Same defect class as §1.3, but left alone deliberately: what the page should say depends on the owner's call in §1.5 on whether /insights/ ships at all.

• If /insights/ ships: delete both note-cards and rewrite the hero as reader-facing copy.
• If it does not: drop it from the primary nav and exclude the route from the sitemap.

2.7 cert-manager DNS-01 self-check uses CoreDNS

cert-manager logs configured acme dns01 nameservers nameservers=["10.43.0.10:53"], i.e. it validates DNS-01 propagation through the cluster's own CoreDNS rather than public resolvers. That makes issuance sensitive to internal caching — including negative caching of the _acme-challenge name it is waiting on.

• Point it at public resolvers before cutover: --dns01-recursive-nameservers-only=true --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53. Not confirmed as the cause of the 4h stall on 2026-10-09 (Cloudflare's authoritative NS were themselves returning NXDOMAIN for records its API reported as present), but it is the standard hardening and removes one variable from certificate renewal.

2.6 Staging URL ✅ LIVE 2026-09-23

https://bcs.int.7tl-homelab.com — review the rebuild in a browser, valid cert, no warning.

• LAN-only: *.int.7tl-homelab.com resolves to the internal traefik LB (10.66.15.30). Not reachable from the internet; the real domain is untouched and WordPress still serves it.
• Cert issued by Let's Encrypt via cert-manager DNS-01 in ~75s, CN=bcs.int.7tl-homelab.com, valid to 2026-12-21, auto-renewing.
• All 9 public routes verified 200.
• Commit 171c8c9. Note it also corrected the issuer name in k8s/ingress.yaml: the commented placeholder said letsencrypt-production, but the ClusterIssuer is letsencrypt-prod — uncommenting it as written at cutover would have silently failed to issue.
• ArgoCD may take a few minutes to poll a push; force it with kubectl annotate application bcs-website -n argocd argocd.argoproj.io/refresh=hard --overwrite.

2.1 e2e suite is red — stale assertions, not a regression

4 of 46 tests fail. Both failures assert invented copy that the exact-copy work correctly removed:

• tests/site.spec.ts:67 waits for a link named Explore services; the homepage CTA is now All services (matches live WP).
• tests/site.spec.ts:101 waits for a heading matching /Focused expertise/, which no longer exists on /focus-areas/.

Navigation itself works. The tests are wrong, not the site.

• Update the two tests to assert current copy.
• Get back to 46/46 so the suite is trustworthy at cutover.

2.2 Repo hygiene

Commit 9f1cbca committed ~50 .playwright-mcp/ screenshots and YAML snapshots plus a 2.8 MB home-full.png at the repo root.

• Add .playwright-mcp/ to .gitignore; remove the tracked artifacts and home-full.png.

2.3 Uptime monitoring is not deployed

Per ADR-0004 the plan is to reuse the homelab monylog stack (not Uptime Kuma). Pod metrics and nginx logs already flow via Alloy; the external-availability piece was never added.

• [?] Verify current state on monylog-1 — the items below are from notes, not re-verified today.
• Add blackbox_exporter + web-bluecap.yml alert rules; re-run the monylog ansible role to reload Prometheus.
• Confirm an external HTTPS check fires to Pushover on a simulated outage. Do this before cutover — the first hours on new infrastructure are when you most need paging.

2.4 Analytics is a no-op

src/components/Analytics.astro is wired into BaseLayout but gated on PUBLIC_ env vars that are absent from k8s/configmap.yaml, so nothing is collected.

• Decide whether to launch with analytics or add it after (ADR-0005: PostHog + Umami trial).
• PostHog requires a privacy/consent notice before production use — that gates it, not the code.
• Set the env vars and confirm pageviews plus a consultation/CTA event.

---

3. Owner decisions (Brad)

These are yours, not mine. Several block content that is otherwise ready.

• 🛑 Canonical host — apex vs www (also §1.6).
• 🛑 Contact details — confirm 804-332-6618 / info@bluecapstrategies.com are the production values. (The live WP contact page carries placeholders 123-456-8912 / info@gmail.com, deliberately not reproduced; see docs/content-deviations.md.)
• 🛑 AI training-bot policy — GPTBot and ClaudeBot are currently Allow: / in public/robots.txt, with a ready-to-uncomment Disallow: /. Search and answer crawlers stay allowed either way. Decide before the site is publicly crawlable.
• 🛑 Brad Rodgers bio + credentials + portrait — gates Person schema, the About buildout, and principal-led E-E-A-T/GEO positioning. Core asset, still missing.
• 🛑 Real insights articles — no fabricated or unreviewed AI content. Until one exists, /insights/ is an empty section in the nav (see §1.5).
• 🛑 Social profiles — the live site only has per-page share links and empty # hrefs. Provide real profiles or omit the footer block.
• 🛑 /test/ page — the live site has a staging duplicate of the homepage. Default: omit.
• 🛑 Proof modules / representative engagements — no fabricated client work or dollar figures. Provide real ones or ship without.

---

4. Post-cutover (not blocking)

• Google Search Console + Bing Webmaster verified; sitemap-index.xml submitted.
• Per-page 1200×630 OG images. Every page currently shares one fallback; BaseLayout.astro:49 emits a single og:image.
• robots meta wired to a noindex frontmatter flag. /thank-you/ is excluded from the sitemap but emits no noindex meta tag.
• article:published_time / article:modified_time OG tags on insights posts (absent today).
• lastmod in the sitemap from content dates.
• Lighthouse ≥ 95 on SEO / Perf / Best-Practices / Accessibility; watch CLS from unsized images.
• Trailing-slash audit — trailingSlash: "always"; confirm canonicals and internal links agree.
• Decommission plan for the WordPress/Bluehost instance (keep it live and reachable through cutover + a rollback window; see the "keep old site live" constraint).

---

5. Already done — do not redo

Recorded because the older checklists still list several of these as open.

• Full copy parity: home, about, services, focus areas, all 8 service detail pages, all 4 focus-area detail pages — verbatim live copy in Markdown content collections (ADR-0002).
• Real imagery imported to public/assets/live-site/ (+ /icons); placeholders deleted.
• Zod seoSchema in src/content.config.ts — build fails on title/description/OG violations.
• scripts/seo-lint.mjs + npm run seo:lint — single-h1, heading order, canonical, alt text, JSON-LD parse. Green on 21 pages.
• BreadcrumbList schema (src/utils/schema.ts) applied across detail pages — the SEO doc still lists this as to-do; it is done.
• Per-agent public/robots.txt; @astrojs/sitemap excluding /thank-you/.
• Deviations logged in docs/content-deviations.md.
• GitOps: Gitea Actions CI, Gitea Packages registry, ArgoCD Application, External-Secrets registry pull auth over public HTTPS (ADR-0007, ADR-0008). Proven end-to-end in July — currently stalled, see §1.2.
• Monitoring approach decided: reuse monylog, not Uptime Kuma (ADR-0004). The SEO doc's "Uptime Kuma" line is superseded.