diff --git a/docs/launch-checklist.md b/docs/launch-checklist.md index 8c89015..1658207 100644 --- a/docs/launch-checklist.md +++ b/docs/launch-checklist.md @@ -27,6 +27,9 @@ Status legend: `[ ]` open ยท `[x]` done ยท `[?]` needs verification ยท ๐Ÿ›‘ owne | `npm run test:e2e` | **44 passed / 4 failed** (the 4 are stale assertions, see ยง2.1) | | CI root cause | **found + fixed in-repo 2026-09-22** โ€” see ยง1.2; needs a green run to confirm | | Staging URL | **https://bcs.int.7tl-homelab.com** โ€” LAN-only, valid cert (ยง2.6) | +| Public traffic path | Cloudflare โ†’ Pangolin VPS 23.94.111.133 โ†’ newt tunnel โ†’ k3s :443 (ยง1.1) | +| Pangolin | org `rhh-homelab`, site `7tl_lab` (newt, online), 23 resources, Pangolin 1.22.2 | +| Contact form | **working** โ€” Formspree, verified end to end 2026-10-09 (ยง1.4) | | Old WordPress site | untouched and serving โ€” must stay that way until cutover | | Gitea host | KVM VM `gitea-1` (10.66.15.22), **Debian 12**, docker compose โ€” *not* in k3s | | Gitea / act_runner | 1.27.3 / 3.4.2 โ€” both healthy | @@ -41,19 +44,90 @@ work is not live. Nothing below is optional. Cutover is unsafe until all of ยง1 is closed. -### 1.1 TLS is not configured ๐Ÿ›‘ hard blocker -- [ ] Uncomment the `tls:` block and `cert-manager.io/cluster-issuer` annotation in - `k8s/ingress.yaml`; set `secretName: bluecap-website-tls`. -- [ ] Use the existing `letsencrypt-prod` ClusterIssuer (present in-cluster, 472d old, Ready). -- [x] **Ordering problem SOLVED.** `letsencrypt-prod` solves via **DNS-01 against Cloudflare**, - not HTTP-01, so a cert can be issued *before* any DNS points at the cluster โ€” no insecure - window. Proven end to end on 2026-09-23: the staging host issued a valid LE cert in ~75s - with no public reachability (ยง2.6). -- [ ] ๐Ÿ›‘ **Verify the one remaining dependency:** `bluecapstrategies.com` must be in the - Cloudflare account that cert-manager's API token can reach. The domain is at Bluehost - today, so this is probably **not** yet true. It is the single fact separating a clean - zero-downtime cutover from one with a gap โ€” check it early, not on launch day. -- [ ] Verify `https://` serves a valid chain and HTTPโ†’HTTPS redirects before announcing. +### 1.1 Cutover runbook โ€” traffic path and TLS ๐Ÿ›‘ hard blocker +Verified against the live Cloudflare zone, the Pangolin VPS and the cluster on **2026-10-09**. + +#### The actual traffic path +Public traffic cannot reach k3s directly; it has to transit the Pangolin VPS. + +``` +visitor โ†’ Cloudflare DNS (bluecapstrategies.com) + โ†’ Pangolin VPS 23.94.111.133 (Traefik v3.7.13 + CrowdSec, terminates TLS) + โ†’ gerbil/newt tunnel โ†’ site `7tl_lab` (org `rhh-homelab`) + โ†’ k3s traefik 10.66.15.30/.31/.32 :443 (second TLS hop) + โ†’ bluecap-website service +``` + +**Two certificates are needed, from two different systems:** + +| Hop | Issued by | Challenge | Can pre-issue? | +|-----|-----------|-----------|----------------| +| edge (Pangolin Traefik) | its own `letsencrypt` resolver | **HTTP-01** on entrypoint `web` | **No** | +| internal (k3s ingress) | cert-manager `letsencrypt-prod` | **DNS-01** via Cloudflare | **Yes** | + +#### Corrections to earlier assumptions in this doc +- There is **one** Cloudflare token, not two. Pangolin's Traefik has **no** Cloudflare + credentials โ€” it uses HTTP-01. Nothing to configure on the VPS for DNS. +- The "ordering problem SOLVED" claim was **half right**. cert-manager does use DNS-01 and can + pre-issue (proven on the staging host, ยง2.6). But **Pangolin cannot** โ€” its + `certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint: web` means the edge cert is + only obtainable once DNS already points at the VPS. A short window at cutover is unavoidable + unless the edge resolver is converted to DNS-01, which would change renewal behaviour for all + 23 existing resources and is not worth it to save seconds. +- Earlier advice said "use DNS-only, matching your existing pattern." The BlueCap zone is in fact + **already fully proxied** (orange cloud) while the homelab zones are DNS-only. It is a real + decision โ€” see below. + +#### Prerequisites (all invisible to the public; the old site keeps serving) +- [x] Cloudflare token now covers the zone โ€” `7tl-homelab.com`, **`bluecapstrategies.com`**, + `rodgersweb.com` (done 2026-10-09). Token is DNS-edit-only, which is correct least privilege. +- [ ] **k3s ingress:** add `www.bluecapstrategies.com` + `bluecapstrategies.com` to the `tls:` + block in `k8s/ingress.yaml` (issuer annotation is already present and correct โ€” + `letsencrypt-prod`). cert-manager pre-issues via DNS-01, so do this **before** cutover and + confirm the Certificate goes Ready. Needed because Pangolin proxies to **https:443**. +- [ ] **Pangolin:** add `bluecapstrategies.com` as a domain under org **`rhh-homelab`** + (`orgDomains`), `certResolver=letsencrypt`. +- [ ] **Pangolin:** create the resource(s) with **three targets**, matching the established + pattern for every k3s-backed resource (Authentik, paperless, immich, dav all do this): + `10.66.15.30`, `10.66.15.31`, `10.66.15.32` โ€” method **https**, port **443**. + Set the host header so k3s traefik routes on the real hostname. +- [ ] ๐Ÿ›‘ **Check Cloudflare's SSL/TLS mode in the dashboard** (the API token cannot read zone + settings). If it is **Flexible**, Cloudflare talks HTTP to Pangolin, Pangolin redirects to + HTTPS, and you get a redirect loop. Must be **Full (strict)** if the record stays proxied. +- [ ] **Test before any DNS change:** `curl --resolve bluecapstrategies.com:443:23.94.111.133` + and confirm the full path serves the site. + +#### Proxied vs DNS-only โ€” decide before cutover ๐Ÿ›‘ +The zone is currently **proxied**: apex `A โ†’ 50.87.136.52 PROXIED`, `www CNAME โ†’ apex PROXIED`. + +- **DNS-only (recommended for cutover):** the exact path all 23 working resources use. CrowdSec + sees real client IPs as currently configured; HTTP-01 goes direct. Cost: no Cloudflare CDN/DDoS, + and the VPS IP becomes visible. +- **Stay proxied:** keeps CDN/DDoS and hides the origin, but CrowdSec will see Cloudflare IPs + unless it is configured to trust `CF-Connecting-IP` โ€” bans would land on the wrong addresses or + on Cloudflare itself. Also requires Full (strict) per above. + +Revisit proxying after launch as a deliberate change with the CrowdSec real-IP config done +properly โ€” not as part of cutover. + +#### Cutover +- [ ] Point apex + `www` at **23.94.111.133** in Cloudflare (replacing `50.87.136.52`). +- [ ] **Immediately** `curl https://bluecapstrategies.com` yourself to trigger Pangolin's HTTP-01 + issuance, before a real visitor hits an unissued cert. +- [ ] Verify a valid chain on both hostnames and that HTTPโ†’HTTPS redirects. +- [ ] Apply the canonical-host decision (ยง1.6): redirect the non-canonical name. + +**Rollback:** set the A record back to `50.87.136.52`. All records are `ttl=1` (Cloudflare "Auto", +~300s for DNS-only), so propagation and rollback are both ~5 minutes โ€” not hours. + +#### Mail is not affected +MX (`eagle.mxlogin.com` / `eagle-relay.mxlogin.com`) and `mail.bluecapstrategies.com` are **not +proxied** and are untouched by the cutover. MXroute keeps serving mail throughout. See +`mxroute-mail-setup` notes. + +- [ ] Minor cleanup (post-launch): `autoconfig` and `autodiscover` still point at Bluehost + (`50.87.136.52`) while mail is on MXroute โ€” stale records that can hand mail clients the + wrong settings. ### 1.2 CI/CD is stalled โ€” no deploy since 2026-07-20 ๐Ÿ›‘ hard blocker **ROOT CAUSE FOUND AND FIXED IN-REPO 2026-09-22.** Not a runner problem โ€” the runner was online