bcs-website/docs/launch-checklist.md

# Launch Checklist — DNS Cutover to the Astro Rebuild

**Single source of truth** for everything standing between today and pointing
`bluecapstrategies.com` at the homelab k3s deployment.

Verified against the working tree, the cluster, and ArgoCD on **2026-09-18**. Supersedes the
task lists that were scattered across `planning/parity-checklist.md` (git-ignored, content-only)
and `docs/seo-aeo-optimization.md` §2/§7 (SEO-only, stale) — those now point here.

Status legend: `[ ]` open · `[x]` done · `[?]` needs verification · 🛑 owner decision

---

## 0. As-built state (2026-09-18)

| Thing | State |
|-------|-------|
| ArgoCD app `bcs-website` | Synced / Healthy |
| Pods | 2/2 Running, ns `bluecap-strategies` |
| Deployed image | `git.7tl-homelab.com/bmr_bluecap/bcs-website:fc64909f3df1` (built **2026-07-20**) |
| `k8s/kustomization.yaml` newTag | `fc64909f3df1` — same, so ArgoCD is correct; CI never bumped it |
| HEAD | `9f1cbca` (2026-08-03), pushed to `origin/main`, **never deployed** |
| Ingress | `bluecap-website` (traefik), hosts apex + `www`, **port 80 only, no TLS** |
| `npm run build` | green, 21 pages, 0 errors / 0 warnings |
| `npm run typecheck` | green, 28 files, 0/0/0 |
| `npm run seo:lint` | green, 21 pages clean |
| `npm run test:e2e` | **44 passed / 4 failed** (the 4 are stale assertions, see §2.1) |
| CI root cause | **found + fixed in-repo 2026-09-22** — see §1.2; needs a green run to confirm |
| Staging URL | **https://bcs.int.7tl-homelab.com** — LAN-only, valid cert (§2.6) |
| Public traffic path | Cloudflare → Pangolin VPS 23.94.111.133 → newt tunnel → k3s :443 (§1.1) |
| Pangolin | org `rhh-homelab`, site `7tl_lab` (newt, online), 23 resources, Pangolin 1.22.2 |
| Contact form | **working** — Formspree, verified end to end 2026-10-09 (§1.4) |
| Old WordPress site | untouched and serving — must stay that way until cutover |
| Gitea host | KVM VM `gitea-1` (10.66.15.22), **Debian 12**, docker compose — *not* in k3s |
| Gitea / act_runner | 1.27.3 / 3.4.2 — both healthy |
| CI job image | **`node:22-bookworm`** (Debian 12) as of 2026-09-22 — was EOL bullseye; see §1.2 |

**The headline:** the public-facing deployment is ~7 weeks behind `main`. The Aug 3 exact-copy
work is not live.

---

## 1. Cutover blockers

Nothing below is optional. Cutover is unsafe until all of §1 is closed.

### 1.1 Cutover runbook — traffic path and TLS 🛑 hard blocker
Verified against the live Cloudflare zone, the Pangolin VPS and the cluster on **2026-10-09**.

#### The actual traffic path
Public traffic cannot reach k3s directly; it has to transit the Pangolin VPS.

```
visitor → Cloudflare DNS (bluecapstrategies.com)
        → Pangolin VPS 23.94.111.133  (Traefik v3.7.13 + CrowdSec, terminates TLS)
        → gerbil/newt tunnel  → site `7tl_lab` (org `rhh-homelab`)
        → k3s traefik 10.66.15.30/.31/.32 :443  (second TLS hop)
        → bluecap-website service
```

**Two certificates are needed, from two different systems:**

| Hop | Issued by | Challenge | Can pre-issue? |
|-----|-----------|-----------|----------------|
| edge (Pangolin Traefik) | its own `letsencrypt` resolver | **HTTP-01** on entrypoint `web` | **No** |
| internal (k3s ingress) | cert-manager `letsencrypt-prod` | **DNS-01** via Cloudflare | **Yes** |

#### Corrections to earlier assumptions in this doc
- There is **one** Cloudflare token, not two. Pangolin's Traefik has **no** Cloudflare
  credentials — it uses HTTP-01. Nothing to configure on the VPS for DNS.
- The "ordering problem SOLVED" claim was **half right**. cert-manager does use DNS-01 and can
  pre-issue (proven on the staging host, §2.6). But **Pangolin cannot** — its
  `certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint: web` means the edge cert is
  only obtainable once DNS already points at the VPS. A short window at cutover is unavoidable
  unless the edge resolver is converted to DNS-01, which would change renewal behaviour for all
  23 existing resources and is not worth it to save seconds.
- Earlier advice said "use DNS-only, matching your existing pattern." The BlueCap zone is in fact
  **already fully proxied** (orange cloud) while the homelab zones are DNS-only. It is a real
  decision — see below.

#### Prerequisites (all invisible to the public; the old site keeps serving)
- [x] Cloudflare token now covers the zone — `7tl-homelab.com`, **`bluecapstrategies.com`**,
      `rodgersweb.com` (done 2026-10-09). Token is DNS-edit-only, which is correct least privilege.
- [ ] **k3s ingress:** add `www.bluecapstrategies.com` + `bluecapstrategies.com` to the `tls:`
      block in `k8s/ingress.yaml` (issuer annotation is already present and correct —
      `letsencrypt-prod`). cert-manager pre-issues via DNS-01, so do this **before** cutover and
      confirm the Certificate goes Ready. Needed because Pangolin proxies to **https:443**.
- [ ] **Pangolin:** add `bluecapstrategies.com` as a domain under org **`rhh-homelab`**
      (`orgDomains`), `certResolver=letsencrypt`.
- [ ] **Pangolin:** create the resource(s) with **three targets**, matching the established
      pattern for every k3s-backed resource (Authentik, paperless, immich, dav all do this):
      `10.66.15.30`, `10.66.15.31`, `10.66.15.32` — method **https**, port **443**.
      Set the host header so k3s traefik routes on the real hostname.
- [ ] 🛑 **Check Cloudflare's SSL/TLS mode in the dashboard** (the API token cannot read zone
      settings). If it is **Flexible**, Cloudflare talks HTTP to Pangolin, Pangolin redirects to
      HTTPS, and you get a redirect loop. Must be **Full (strict)** if the record stays proxied.
- [ ] **Test before any DNS change:** `curl --resolve bluecapstrategies.com:443:23.94.111.133`
      and confirm the full path serves the site.

#### Proxied vs DNS-only — decide before cutover 🛑
The zone is currently **proxied**: apex `A → 50.87.136.52 PROXIED`, `www CNAME → apex PROXIED`.

- **DNS-only (recommended for cutover):** the exact path all 23 working resources use. CrowdSec
  sees real client IPs as currently configured; HTTP-01 goes direct. Cost: no Cloudflare CDN/DDoS,
  and the VPS IP becomes visible.
- **Stay proxied:** keeps CDN/DDoS and hides the origin, but CrowdSec will see Cloudflare IPs
  unless it is configured to trust `CF-Connecting-IP` — bans would land on the wrong addresses or
  on Cloudflare itself. Also requires Full (strict) per above.

Revisit proxying after launch as a deliberate change with the CrowdSec real-IP config done
properly — not as part of cutover.

#### Cutover
- [ ] Point apex + `www` at **23.94.111.133** in Cloudflare (replacing `50.87.136.52`).
- [ ] **Immediately** `curl https://bluecapstrategies.com` yourself to trigger Pangolin's HTTP-01
      issuance, before a real visitor hits an unissued cert.
- [ ] Verify a valid chain on both hostnames and that HTTP→HTTPS redirects.
- [ ] Apply the canonical-host decision (§1.6): redirect the non-canonical name.

**Rollback:** set the A record back to `50.87.136.52`. All records are `ttl=1` (Cloudflare "Auto",
~300s for DNS-only), so propagation and rollback are both ~5 minutes — not hours.

#### Mail is not affected
MX (`eagle.mxlogin.com` / `eagle-relay.mxlogin.com`) and `mail.bluecapstrategies.com` are **not
proxied** and are untouched by the cutover. MXroute keeps serving mail throughout. See
`mxroute-mail-setup` notes.

- [ ] Minor cleanup (post-launch): `autoconfig` and `autodiscover` still point at Bluehost
      (`50.87.136.52`) while mail is on MXroute — stale records that can hand mail clients the
      wrong settings.

### 1.2 CI/CD is stalled — no deploy since 2026-07-20 🛑 hard blocker
**ROOT CAUSE FOUND AND FIXED IN-REPO 2026-09-22.** Not a runner problem — the runner was online
and the run *did* execute.

Evidence (Gitea 1.27.3 on `gitea-1`, sqlite `action_run` for repo_id 3):
- Run #6, commit `9f1cbca1b148`, created 2026-08-03 13:46:53, **status 2 = failure**, ran 13m
  (13:47:02 → 14:00:11) on runner 59.
- No `9f1cbca1b148` entry in `package_version` → it died **at or before** the image push.
- Job logs are past retention (`/data/gitea/actions_log` no longer holds run 205), so the cause
  was reproduced directly instead.

**The cause:** the `Ensure docker + kustomize are available` step ran
`apt-get install -y docker.io`. The job container image is **`node:18-bullseye`** (Debian 11),
and bullseye went EOL — `debian-security` no longer serves those package versions. `apt-get
update` still succeeds, so this fails only at install time, with 404s on `docker.io`,
`containerd`, `systemd`, `libcap2`. Reproduced verbatim in a `node:18-bullseye` container on the
runner host. That timing matches exactly: last green run 2026-07-20, first failure 2026-08-03.

**The fix (applied to `.gitea/workflows/ci.yml`):** fetch the docker CLI and kustomize as pinned
**static binaries** (`download.docker.com` / GitHub releases) instead of distro packages. Verified
in the real job environment: the static client installs, reports `server=28.1.1` through the
mounted socket, and `docker ps` succeeds. kustomize tarball and `git` are fine.

Everything else on the runner path checked out and needs no change:
- [x] act_runner container up, `v3.4.2`, registered and polling (the daily 07:30 "connection
      refused" entries are the Gitea restart window, harmless).
- [x] Runner scope **already fixed** — runner id 59 is `owner_id=0, repo_id=0` (instance-scoped),
      so the old user-scoped problem is resolved. 58 stale duplicate runner rows remain; cosmetic,
      prune with the role's `gitea_runner_prune` tag when convenient.
- [x] Runner host dockerd trusts `10.66.15.22:3000` as insecure — confirmed in `docker info`.
- [x] `config.yaml` pins jobs to `gitea_network` and passes the docker socket — correct.
- [x] `CI_TOKEN` repo secret present (repo_id 3).
- [x] `paths-ignore` did not swallow the push — the commit touched `src/**`.
- [x] `docker build` of the current tree succeeds locally, so the Dockerfile and lockfile are fine.

Remaining:
- [ ] **Re-trigger CI and confirm a green run** — the fix is verified at the step level but the
      full pipeline has not run end-to-end since July. Requires a push touching a non-ignored path.
- [ ] Confirm the tag-bump commit lands and ArgoCD syncs the new image.
- [ ] Do not cut DNS onto a stale image — confirm the deployed tag matches HEAD.
- [x] **CI job image upgraded 2026-09-22.** `gitea_runner_labels` in
      `~/Homelab/ansible/roles/gitea/defaults/main.yml` changed `node:18-bullseye` →
      **`node:22-bookworm`**; playbook run succeeded (ok=40 changed=8 failed=0). Runner **60**
      registered and online, `owner_id=0` (instance-scoped), label confirmed in `/data/.runner`.
      The job image now matches the host VM and k3s nodes (Debian 12) and the Node 22 that
      workflows install anyway.
      - Gotcha for next time: `--tags gitea_runner_prune` restricts the run to *only* that tag, so
        it cannot be combined with `-e gitea_runner_force_reregister=true`. Two separate runs.
      - Gotcha 2: `gitea_admin` had `must_change_password=1` (set during the 1.27.3 container
        recreate on 2026-09-14), which made Gitea reject **all** admin API calls with 403 — not
        401, so credentials were fine. Cleared 2026-09-22 with
        `docker exec -u git gitea gitea admin user must-change-password --unset gitea_admin`
        (password unchanged; vault still valid). The admin API works again.
- [ ] Optional cleanup: prune the stale duplicate runner rows (59 of them now). Unblocked by the
      admin-API fix above. `ansible-playbook -i inventory/hosts.ini gitea-deploy.yml --tags
      gitea_runner_prune`. Purely cosmetic — deletes only runners the API reports offline.

**Note on the two fixes:** CI now has *two* independent reasons to work — the workflow no longer
uses apt at all, and the job image is no longer EOL (bookworm still serves `docker.io` fine). The
static-binary approach remains the more durable of the two; keep it regardless.

### 1.3 Developer-facing copy on the contact page ✅ FIXED 2026-09-23
Two notes written for developers, not visitors, were being rendered:
- the form note explaining `PUBLIC_CONTACT_FORM_ENDPOINT`, and
- a "Discovery note" about the WordPress page holding placeholder contact values.

Replaced the first with reader-facing copy and dropped the second (the deviation it described is
already recorded in `docs/content-deviations.md`). Commit `4e354d2`.

### 1.4 Contact form endpoint — plumbing DONE 2026-09-23, needs your Formspree ID 🛑
**Correction to the original entry:** it said to set `PUBLIC_CONTACT_FORM_ENDPOINT` in
`k8s/configmap.yaml`. That would never have worked.
- The Deployment has **no `env`/`envFrom`** referencing that ConfigMap — it is dead config.
  (`PUBLIC_SITE_URL` is in the same state; it only appears to work because the default baked into
  `astro.config.mjs` is already correct.)
- Even wired up it could not work: `output: "static"` means Astro **inlines** `PUBLIC_*` at image
  build time, and the pod only runs nginx over pre-built HTML. There is no runtime knob.

**Done (commit `af4cc4a`):**
- [x] `Dockerfile` takes `ARG PUBLIC_CONTACT_FORM_ENDPOINT` / `PUBLIC_SITE_URL` → `ENV` before build.
- [x] CI passes `--build-arg` from a `CONTACT_FORM_ENDPOINT` Actions secret.
- [x] ConfigMap rewritten as documentation, with a warning not to add `envFrom`.
- [x] Submit posts in the background via `fetch()` and redirects to our own `/thank-you/` instead
      of navigating to the provider's page.
- [x] Failure path: stays on the page, re-enables the button, shows the phone/email fallback.
- [x] Honeypot `_gotcha` (Formspree treats it as spam), positioned off-screen rather than
      `display:none` so bots that skip hidden fields still fill it.
- [x] e2e covers the real fetch path (mocked endpoint, asserts the payload) and the failure case.

**Still needs you:**
- [ ] 🛑 Create the Formspree form for **info@bluecapstrategies.com** and confirm the address.
- [ ] 🛑 Add the endpoint as a Gitea repo Actions secret named **`CONTACT_FORM_ENDPOINT`**
      (Settings → Actions → Secrets on `bmr_bluecap/bcs-website`), value
      `https://formspree.io/f/<your-id>`.
- [ ] Push any commit touching `src/**` so CI rebuilds with the secret baked in.
- [ ] Verify on the staging URL: submit a real inquiry, confirm it arrives, confirm the browser
      lands on `/thank-you/`.
- [ ] Confirm the destination inbox is actually monitored. A working form delivering into an
      unwatched inbox is its own kind of failure.

**Degradation if the secret is missing:** the build still succeeds and the form falls back to
opening a `mailto:` draft. Survivable, but it fails silently for webmail users with no OS mail
client — so treat a missing secret as a launch blocker, not a warning.

### 1.5 Draft insight was being published and indexed ✅ FIXED 2026-09-23
`src/pages/insights/[slug].astro` called `getCollection("insights")` unfiltered, so the
`draft: true` placeholder built to `/insights/example-draft/` **and** appeared in
`sitemap-0.xml`. The schema declared the flag and `rss.xml.ts` already honoured it — only the page
route ignored it. Filtered in `getStaticPaths`, matching the RSS pattern. Build is 21 → 20 pages
and the URL is gone from the sitemap. Commit `4e354d2`.

- [ ] 🛑 **Still open (owner):** whether `/insights/` should ship at all with zero real articles.
      It is currently in the primary nav. See also §2.5.

### 1.6 Canonical host is undecided 🛑 owner
The ingress answers on **both** apex and `www`; `astro.config.mjs`, `robots.txt`, and the
ConfigMap all hardcode `https://www.bluecapstrategies.com`. Serving identical content on two
hosts with no redirect splits SEO signal from day one.
- [ ] Owner picks apex or `www`.
- [ ] Configure a 301 from the loser to the winner (traefik middleware or edge).
- [ ] Reconcile `PUBLIC_SITE_URL`, canonical tags, `robots.txt` Sitemap line, and the ingress.

---

## 2. Should-fix before cutover

### 2.5 `/insights/` index still renders developer-facing copy
Found while fixing §1.5. `src/pages/insights/index.astro` shows visitors two cards written for a
developer, not a reader:
- *"Publishing path"* — "Future posts can be added as Markdown or MDX in
  `src/content/insights` with typed frontmatter for title, description, publish date…"
- *"RSS support"* — "The feed is ready for future article imports even though no public posts are
  published yet."

The hero also announces the section as a "Future publishing home", i.e. it tells visitors the
section is empty. Same defect class as §1.3, but left alone deliberately: what the page *should*
say depends on the owner's call in §1.5 on whether `/insights/` ships at all.

- [ ] If `/insights/` ships: delete both note-cards and rewrite the hero as reader-facing copy.
- [ ] If it does not: drop it from the primary nav and exclude the route from the sitemap.

### 2.6 Staging URL ✅ LIVE 2026-09-23
**https://bcs.int.7tl-homelab.com** — review the rebuild in a browser, valid cert, no warning.

- LAN-only: `*.int.7tl-homelab.com` resolves to the internal traefik LB (10.66.15.30). Not
  reachable from the internet; the real domain is untouched and WordPress still serves it.
- Cert issued by Let's Encrypt via cert-manager DNS-01 in ~75s, `CN=bcs.int.7tl-homelab.com`,
  valid to 2026-12-21, auto-renewing.
- All 9 public routes verified 200.
- Commit `171c8c9`. Note it also corrected the issuer name in `k8s/ingress.yaml`: the commented
  placeholder said `letsencrypt-production`, but the ClusterIssuer is **`letsencrypt-prod`** —
  uncommenting it as written at cutover would have silently failed to issue.
- ArgoCD may take a few minutes to poll a push; force it with
  `kubectl annotate application bcs-website -n argocd argocd.argoproj.io/refresh=hard --overwrite`.

### 2.1 e2e suite is red — stale assertions, not a regression
4 of 46 tests fail. Both failures assert **invented copy that the exact-copy work correctly
removed**:
- `tests/site.spec.ts:67` waits for a link named `Explore services`; the homepage CTA is now
  `All services` (matches live WP).
- `tests/site.spec.ts:101` waits for a heading matching `/Focused expertise/`, which no longer
  exists on `/focus-areas/`.

Navigation itself works. The tests are wrong, not the site.
- [ ] Update the two tests to assert current copy.
- [ ] Get back to 46/46 so the suite is trustworthy at cutover.

### 2.2 Repo hygiene
Commit `9f1cbca` committed ~50 `.playwright-mcp/` screenshots and YAML snapshots plus a 2.8 MB
`home-full.png` at the repo root.
- [ ] Add `.playwright-mcp/` to `.gitignore`; remove the tracked artifacts and `home-full.png`.

### 2.3 Uptime monitoring is not deployed
Per ADR-0004 the plan is to reuse the homelab `monylog` stack (not Uptime Kuma). Pod metrics and
nginx logs already flow via Alloy; the external-availability piece was never added.
- [?] Verify current state on monylog-1 — the items below are from notes, not re-verified today.
- [ ] Add `blackbox_exporter` + `web-bluecap.yml` alert rules; re-run the monylog ansible role to
      reload Prometheus.
- [ ] Confirm an external HTTPS check fires to Pushover on a simulated outage. **Do this before
      cutover** — the first hours on new infrastructure are when you most need paging.

### 2.4 Analytics is a no-op
`src/components/Analytics.astro` is wired into `BaseLayout` but gated on `PUBLIC_` env vars that
are absent from `k8s/configmap.yaml`, so nothing is collected.
- [ ] Decide whether to launch with analytics or add it after (ADR-0005: PostHog + Umami trial).
- [ ] **PostHog requires a privacy/consent notice before production use** — that gates it, not
      the code.
- [ ] Set the env vars and confirm pageviews plus a consultation/CTA event.

---

## 3. Owner decisions (Brad)

These are yours, not mine. Several block content that is otherwise ready.

- [ ] 🛑 **Canonical host** — apex vs `www` (also §1.6).
- [ ] 🛑 **Contact details** — confirm `804-332-6618` / `info@bluecapstrategies.com` are the
      production values. (The live WP contact page carries placeholders `123-456-8912` /
      `info@gmail.com`, deliberately **not** reproduced; see `docs/content-deviations.md`.)
- [ ] 🛑 **AI training-bot policy** — `GPTBot` and `ClaudeBot` are currently **Allow: /** in
      `public/robots.txt`, with a ready-to-uncomment `Disallow: /`. Search and answer crawlers
      stay allowed either way. Decide before the site is publicly crawlable.
- [ ] 🛑 **Brad Rodgers bio + credentials + portrait** — gates `Person` schema, the About
      buildout, and principal-led E-E-A-T/GEO positioning. Core asset, still missing.
- [ ] 🛑 **Real insights articles** — no fabricated or unreviewed AI content. Until one exists,
      `/insights/` is an empty section in the nav (see §1.5).
- [ ] 🛑 **Social profiles** — the live site only has per-page share links and empty `#` hrefs.
      Provide real profiles or omit the footer block.
- [ ] 🛑 **`/test/` page** — the live site has a staging duplicate of the homepage. Default: omit.
- [ ] 🛑 **Proof modules / representative engagements** — no fabricated client work or dollar
      figures. Provide real ones or ship without.

---

## 4. Post-cutover (not blocking)

- [ ] Google Search Console + Bing Webmaster verified; `sitemap-index.xml` submitted.
- [ ] Per-page 1200×630 OG images. Every page currently shares one fallback; `BaseLayout.astro:49`
      emits a single `og:image`.
- [ ] `robots` meta wired to a `noindex` frontmatter flag. `/thank-you/` is excluded from the
      sitemap but emits **no** `noindex` meta tag.
- [ ] `article:published_time` / `article:modified_time` OG tags on insights posts (absent today).
- [ ] `lastmod` in the sitemap from content dates.
- [ ] Lighthouse ≥ 95 on SEO / Perf / Best-Practices / Accessibility; watch CLS from unsized images.
- [ ] Trailing-slash audit — `trailingSlash: "always"`; confirm canonicals and internal links agree.
- [ ] Decommission plan for the WordPress/Bluehost instance (keep it live and reachable through
      cutover + a rollback window; see the "keep old site live" constraint).

---

## 5. Already done — do not redo

Recorded because the older checklists still list several of these as open.

- [x] Full copy parity: home, about, services, focus areas, all 8 service detail pages, all 4
      focus-area detail pages — verbatim live copy in Markdown content collections (ADR-0002).
- [x] Real imagery imported to `public/assets/live-site/` (+ `/icons`); placeholders deleted.
- [x] Zod `seoSchema` in `src/content.config.ts` — build **fails** on title/description/OG
      violations.
- [x] `scripts/seo-lint.mjs` + `npm run seo:lint` — single-h1, heading order, canonical, alt text,
      JSON-LD parse. Green on 21 pages.
- [x] `BreadcrumbList` schema (`src/utils/schema.ts`) applied across detail pages — the SEO doc
      still lists this as to-do; it is done.
- [x] Per-agent `public/robots.txt`; `@astrojs/sitemap` excluding `/thank-you/`.
- [x] Deviations logged in `docs/content-deviations.md`.
- [x] GitOps: Gitea Actions CI, Gitea Packages registry, ArgoCD Application, External-Secrets
      registry pull auth over public HTTPS (ADR-0007, ADR-0008). Proven end-to-end in July —
      currently stalled, see §1.2.
- [x] Monitoring approach decided: reuse `monylog`, not Uptime Kuma (ADR-0004). The SEO doc's
      "Uptime Kuma" line is superseded.