bcs-website/docs/launch-checklist.md

# Launch Checklist — Post-Cutover Status

`www.bluecapstrategies.com` went **live on the Astro rebuild on 2026-10-23**. This file is now the
single source of truth for what is running, how it is wired, and what is left. The pre-launch
runbook it replaced is in git history (`git log -- docs/launch-checklist.md`, before 2026-10-23).

Status legend: `[ ]` open · `[x]` done · `[?]` needs verification · 🛑 owner decision

---

## 0. As-built state (2026-10-23)

| Thing | State |
|-------|-------|
| Public site | **https://www.bluecapstrategies.com** — canonical. Apex 301s to www, path preserved |
| Old WordPress / Bluehost | **Gone.** Access lost and it was returning 404 before cutover; nothing to roll back to |
| DNS | Cloudflare, **DNS-only (grey cloud)** for apex, `www`, `stats` → `23.94.111.133` |
| Edge | Pangolin VPS 23.94.111.133 (Traefik + CrowdSec), org `rhh-homelab`, site `7tl_lab` |
| Pangolin resources | `Bluecap-www`, `Bluecap-base`, `stats` — 3 https targets `10.66.15.30/.31/.32:443` each |
| Edge certs | Let's Encrypt HTTP-01 via Pangolin's `letsencrypt` resolver, issued 2026-10-23 |
| Cluster certs | cert-manager `letsencrypt-prod` DNS-01: `bluecap-website-tls` (www + apex), `bcs-int-tls` |
| ArgoCD | `bcs-website` and `umami` — Synced / Healthy |
| Deployed image | tracks `main` via CI tag-bump (`ci: deploy <sha>` commits) |
| Staging | https://bcs.int.7tl-homelab.com — LAN only |
| Contact form | Formspree, verified end to end 2026-10-09 |
| Analytics | **Umami** (self-hosted, `stats.bluecapstrategies.com`) + **GA4** `G-Z273QL1F0N`, both verified recording 2026-10-23 |
| Monitoring | monylog blackbox probes on www + apex → Alertmanager → **Pushover**; UptimeRobot (external) → **Pushover** |
| Search | Google Search Console domain property verified; sitemap submitted and fetched by Googlebot |
| Mail | MXroute. SPF pass, DKIM pass (selector `x`), DMARC `p=none` — mail-tester 9.5/10 |
| `npm run build` / `typecheck` / `seo:lint` | green, 21 pages |
| `npm run test:e2e` | **48/48 green** |

### Traffic path
```
visitor → Cloudflare DNS (DNS-only) → Pangolin VPS 23.94.111.133 (TLS #1, CrowdSec)
        → gerbil/newt tunnel → k3s traefik 10.66.15.30/.31/.32 :443 (TLS #2)
        → bluecap-website (nginx, 2 replicas)        ns bluecap-strategies
        → umami (+ umami-postgres)                   ns umami
```

---

## 1. Open — do soon

### 1.1 External uptime check ✅ 2026-10-23
monylog runs inside the homelab, so a homelab power or internet outage takes the site **and** its
monitor down together, and nothing pages. Needs a vantage point outside the homelab.
- [x] **UptimeRobot** (free tier, 5-min interval) probes `https://www.bluecapstrategies.com/`.
- [x] Alerts go to Pushover via a separate *email alert contact* (the `@pomail.net` gateway
      address), not the account email, so password resets etc. stay in the normal inbox.
- Reading alerts: UptimeRobot **and** monylog both firing = site/VPS problem; UptimeRobot
  alone (monylog silent) usually means the homelab itself is down.

### 1.2 Homelab intercepts port-53 DNS
Found 2026-10-23. Queries from the homelab — including k3s nodes — to *any* DNS server, even
Cloudflare's authoritative NS, are answered by a local recursive resolver (`dig @arely.ns.cloudflare.com`
returns `rd ra` with no `aa`). A name looked up before its record exists gets **NXDOMAIN cached for
30 minutes** (the zone's SOA negative TTL).

This, not Cloudflare, explains the 4-hour cert-manager DNS-01 stall of 2026-10-09 and the
`stats` NXDOMAIN at launch. Consequences:
- Pointing cert-manager at `1.1.1.1/8.8.8.8` (the fix previously suggested here) **will not help**
  while port 53 is intercepted.
- To see true public DNS, query from the VPS or via DoH:
  `curl -H 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=<name>&type=A'`
- Create DNS records *before* anything in the homelab (cert-manager included) looks the name up.

- [ ] 🛑 Owner decision: leave as is, or exempt the k3s nodes from the redirect. The redirect rule
      is not in `~/Homelab/mikrotik/*.rsc`; find it on the router (or Pi-hole/AdGuard, if any).
- [x] `umami-stats-tls` (internal hop for `stats`) stalled on the stale NXDOMAIN at launch and
      issued by itself once the cache expired (~18:35 UTC, 2026-10-23) — the expected pattern.

### 1.3 Gitea runner loses Docker after a dockerd restart
`dockerd` on `gitea-1` runs with `live-restore=true`. When it restarts, containers keep running
but `act_runner`'s bind mount of `/var/run/docker.sock` points at the dead socket, so every CI
job fails in its first second with `Cannot connect to the Docker daemon`. Happened 2026-10-21 →
discovered 2026-10-23.
- **Workaround:** `sudo docker restart act_runner` on gitea-1.
- [ ] Durable fix in the `gitea` ansible role: disable live-restore on that VM, or restart the
      runner after dockerd restarts (systemd `PartOf=`/`ExecStartPost` hook).

### 1.4 `/insights/` ✅ 2026-10-23
Now a reader-facing "Coming soon" page (hero + contact CTA); the developer-facing cards are gone.
It stays in the nav. When the first real article lands, restore a listing on this page.

### 1.5 Privacy policy ✅ 2026-10-23
`/privacy/` reviewed and accepted by the owner. No consent banner (owner's call).
Anything added to `src/components/Analytics.astro` must be disclosed on `/privacy/`.

---

## 2. Should fix

### 2.1 e2e suite ✅ green 2026-10-23 — 48/48
The stale assertions (invented copy removed by the exact-copy work) now check the real headings
and use the header Services link; the homepage body has no direct `/services/` link.

### 2.2 Repo hygiene
Commit `9f1cbca` committed ~50 `.playwright-mcp/` screenshots/snapshots plus a 2.8 MB
`home-full.png` at the repo root.
- [ ] Add `.playwright-mcp/` to `.gitignore`; remove the tracked artifacts and `home-full.png`.

### 2.3 DMARC tightening
- [ ] Tighten DMARC from `p=none` to `p=quarantine` after a few weeks of clean reports
      to info@bluecapstrategies.com.

---

## 3. Owner decisions (Brad)

- [x] **Canonical host** — `www` (2026-10-23). Apex 301s via Traefik middleware in `k8s/ingress.yaml`.
- [x] **Analytics** — Umami + GA4 side by side (2026-10-23).
- [ ] 🛑 **Contact details** — confirm `804-332-6618` / `info@bluecapstrategies.com` are final.
- [x] **AI training-bot policy** — training crawlers blocked (2026-10-23): GPTBot, ClaudeBot,
      Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Bytespider. Search and
      AI answer/search crawlers (Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, …) stay allowed.
- [ ] 🛑 **Brad Rodgers bio + credentials + portrait** — gates `Person` schema, the About
      buildout, and principal-led E-E-A-T/GEO positioning.
- [ ] 🛑 **Real insights articles** — no fabricated or unreviewed AI content. `/insights/` is
      "Coming soon" until the first one exists (§1.4).
- [ ] 🛑 **Social profiles** — provide real profiles or omit the footer block.
- [ ] 🛑 **Proof modules / representative engagements** — no fabricated client work.

---

## 4. Post-launch improvements (not blocking)

- [ ] Bing Webmaster Tools — import from Search Console (covers Bing, DuckDuckGo, ChatGPT search).
- [ ] Check Search Console in ~1 week: sitemap "Success", pages indexed, no 5xx in Coverage.
- [ ] Per-page 1200×630 OG images; `BaseLayout.astro` emits one shared fallback `og:image`.
- [ ] `robots` meta wired to a `noindex` frontmatter flag. `/thank-you/` is excluded from the
      sitemap but emits no `noindex` meta tag.
- [ ] `article:published_time` / `article:modified_time` OG tags on insights posts.
- [ ] `lastmod` in the sitemap from content dates.
- [ ] Lighthouse ≥ 95 on SEO / Perf / Best-Practices / Accessibility.
- [ ] Umami/GA custom event for contact-form submits (today the `/thank-you/` pageview is the
      conversion signal).
- [ ] Optional: move `bluecapstrategies.com` into Pangolin's `config.yml` `domains:` so it shows
      "Config Managed" like the other three (needs a Pangolin restart; purely cosmetic).
- [ ] Deferred: VPS static failover for homelab outages (see memory note / earlier discussion).

---

## 5. Launch-day record (2026-10-23)

What happened, so it does not have to be rediscovered.

- **Pangolin certs need a real config change to retry.** HTTP-01 failed while `www` was still
  proxied (Let's Encrypt reached Cloudflare's IPv6, got 404) and hit the 5-failures/hour limit.
  Clicking Save with no changes does nothing; toggling **Enable Resource** off/on (saving between)
  re-adds the router and triggers issuance.
- **Pangolin health checks** probe the target IP with no hostname, which k3s traefik answers with
  404. Set the health check's Host header and TLS server name to the resource's hostname, path
  `/` (or `/api/heartbeat` for Umami), expect 200.
- **nginx 404 loop** — `error_page` pointed at `/404/index.html` but Astro emits `404.html`, so
  every unknown URL returned 500. Fixed in `d1c1726`; unknown URLs now return a real 404.
- **CI tag-bump push rejected** when `main` moved during a build (or on a re-run of an older
  commit). The step now rebases onto `origin/main` and retries (`93415c6`). Do not "re-run" old
  runs to deploy — push instead.
- **Cloudflare cleanup** — removed all Bluehost/cPanel leftovers (autoconfig/autodiscover,
  cpanel/whm/webdisk/ftp, cal/carddav SRV, `localhost`), replaced Bluehost's `default._domainkey`
  with MXroute's `x._domainkey`, added `_dmarc`.
- **Monitoring safety** — the monylog playbook only renders Pushover routing when `VAULT_TOKEN`
  is set; without it Alertmanager silently falls back to ntfy. Always run it with the token.
- **Umami** ships with `admin` / `umami`. It was brought up LAN-only
  (`umami.int.7tl-homelab.com`), the password changed, and only then exposed via Pangolin.
  Manifests live in `homelab-gitops` (`apps/umami.yaml`, `analytics/umami/`); credentials in
  OpenBao `cluster/umami`.

---

## 6. Already done — do not redo

- [x] Full copy parity with the WordPress site, verbatim in content collections (ADR-0002).
- [x] Real imagery in `public/assets/live-site/`; placeholders deleted.
- [x] Zod `seoSchema`; `scripts/seo-lint.mjs` (`npm run seo:lint`).
- [x] `BreadcrumbList` schema across detail pages.
- [x] Per-agent `public/robots.txt`; `@astrojs/sitemap` excluding `/thank-you/`.
- [x] Draft insights excluded from routes and sitemap.
- [x] Contact form: build-time Formspree endpoint, background submit, honeypot, failure fallback.
- [x] GitOps: Gitea Actions CI → Gitea Packages → tag-bump commit → ArgoCD (ADR-0007, ADR-0008).
- [x] CI job image `node:22-bookworm`; docker CLI + kustomize as pinned static binaries.
- [x] Production TLS in-cluster (DNS-01) and at the edge (HTTP-01).
- [x] Canonical www + apex 301; real 404s.
- [x] Monitoring: blackbox probes + `WebsiteDown` / `WebsiteCertExpiringSoon` / `WebsiteSlow` /
      `BlackboxExporterDown` rules in the monylog role (ADR-0004: monylog, not Uptime Kuma).
- [x] Analytics: Umami + GA4, production-host only; `/privacy/` page.
- [x] Search Console verified; sitemap submitted.
- [x] Mail authentication: SPF, DKIM, DMARC passing.