bmr_bluecap/bcs-website
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: replace section 1.1 with a verified cutover runbook

Browse Source 
The old entry assumed the cluster could be pointed at directly and that
DNS-01 removed the certificate ordering problem. Both were wrong once the
real traffic path was checked.

Public traffic has to transit the Pangolin VPS, so two certificates are
needed from two different systems: the edge cert from Pangolin's own Traefik
resolver, and the internal-hop cert from cert-manager, because every
k3s-backed Pangolin resource proxies to https:443 (the pattern Authentik,
paperless, immich and dav all follow).

Pangolin's resolver uses HTTP-01 on entrypoint web, not DNS-01, so the edge
cert cannot be pre-issued and a short window at cutover is unavoidable.
cert-manager can still pre-issue the internal cert. There is also only one
Cloudflare token, not two: Pangolin holds no Cloudflare credentials at all.

Records the proxied-vs-DNS-only decision, which earlier advice got backwards
(this zone is already orange-clouded while the homelab zones are not), the
Flexible-SSL redirect-loop risk, the ~5 minute rollback from ttl=auto, and
that mail is unaffected because the MX records are not proxied.

Cloudflare token now covers bluecapstrategies.com.
This commit is contained in:
bmr_bluecap committed 2026-10-09 08:01:20 -04:00
1 parent b51a243728
commit a76a6da26c
1 file changed
+87 -13
Download Patch File Download Diff File Expand all files Collapse all files
docs/launch-checklist.md
+87 -13
View File
@@ -27,6 +27,9 @@ Status legend: `[ ]` open · `[x]` done · `[?]` needs verification · 🛑 owne
| `npm run test:e2e` | **44 passed / 4 failed** (the 4 are stale assertions, see §2.1) |
| CI root cause | **found + fixed in-repo 2026-09-22** — see §1.2; needs a green run to confirm |
| Staging URL | **https://bcs.int.7tl-homelab.com** — LAN-only, valid cert (§2.6) |
| Public traffic path | Cloudflare → Pangolin VPS 23.94.111.133 → newt tunnel → k3s :443 (§1.1) |
| Pangolin | org `rhh-homelab`, site `7tl_lab` (newt, online), 23 resources, Pangolin 1.22.2 |
| Contact form | **working** — Formspree, verified end to end 2026-10-09 (§1.4) |
| Old WordPress site | untouched and serving — must stay that way until cutover |
| Gitea host | KVM VM `gitea-1` (10.66.15.22), **Debian 12**, docker compose — *not* in k3s |
| Gitea / act_runner | 1.27.3 / 3.4.2 — both healthy |
@@ -41,19 +44,90 @@ work is not live.
Nothing below is optional. Cutover is unsafe until all of §1 is closed.
### 1.1 TLS is not configured 🛑 hard blocker
- [ ] Uncomment the `tls:` block and `cert-manager.io/cluster-issuer` annotation in
`k8s/ingress.yaml`; set `secretName: bluecap-website-tls`.
- [ ] Use the existing `letsencrypt-prod` ClusterIssuer (present in-cluster, 472d old, Ready).
- [x] **Ordering problem SOLVED.** `letsencrypt-prod` solves via **DNS-01 against Cloudflare**,
not HTTP-01, so a cert can be issued *before* any DNS points at the cluster — no insecure
window. Proven end to end on 2026-09-23: the staging host issued a valid LE cert in ~75s
with no public reachability (§2.6).
- [ ] 🛑 **Verify the one remaining dependency:** `bluecapstrategies.com` must be in the
Cloudflare account that cert-manager's API token can reach. The domain is at Bluehost
today, so this is probably **not** yet true. It is the single fact separating a clean
zero-downtime cutover from one with a gap — check it early, not on launch day.
- [ ] Verify `https://` serves a valid chain and HTTP→HTTPS redirects before announcing.
### 1.1 Cutover runbook — traffic path and TLS 🛑 hard blocker
Verified against the live Cloudflare zone, the Pangolin VPS and the cluster on **2026-10-09**.
#### The actual traffic path
Public traffic cannot reach k3s directly; it has to transit the Pangolin VPS.
```
visitor → Cloudflare DNS (bluecapstrategies.com)
→ Pangolin VPS 23.94.111.133 (Traefik v3.7.13 + CrowdSec, terminates TLS)
→ gerbil/newt tunnel → site `7tl_lab` (org `rhh-homelab`)
→ k3s traefik 10.66.15.30/.31/.32 :443 (second TLS hop)
→ bluecap-website service
```
**Two certificates are needed, from two different systems:**
| Hop | Issued by | Challenge | Can pre-issue? |
|-----|-----------|-----------|----------------|
| edge (Pangolin Traefik) | its own `letsencrypt` resolver | **HTTP-01** on entrypoint `web` | **No** |
| internal (k3s ingress) | cert-manager `letsencrypt-prod` | **DNS-01** via Cloudflare | **Yes** |
#### Corrections to earlier assumptions in this doc
- There is **one** Cloudflare token, not two. Pangolin's Traefik has **no** Cloudflare
credentials — it uses HTTP-01. Nothing to configure on the VPS for DNS.
- The "ordering problem SOLVED" claim was **half right**. cert-manager does use DNS-01 and can
pre-issue (proven on the staging host, §2.6). But **Pangolin cannot** — its
`certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint: web` means the edge cert is
only obtainable once DNS already points at the VPS. A short window at cutover is unavoidable
unless the edge resolver is converted to DNS-01, which would change renewal behaviour for all
23 existing resources and is not worth it to save seconds.
- Earlier advice said "use DNS-only, matching your existing pattern." The BlueCap zone is in fact
**already fully proxied** (orange cloud) while the homelab zones are DNS-only. It is a real
decision — see below.
#### Prerequisites (all invisible to the public; the old site keeps serving)
- [x] Cloudflare token now covers the zone — `7tl-homelab.com`, **`bluecapstrategies.com`**,
`rodgersweb.com` (done 2026-10-09). Token is DNS-edit-only, which is correct least privilege.
- [ ] **k3s ingress:** add `www.bluecapstrategies.com` + `bluecapstrategies.com` to the `tls:`
block in `k8s/ingress.yaml` (issuer annotation is already present and correct —
`letsencrypt-prod`). cert-manager pre-issues via DNS-01, so do this **before** cutover and
confirm the Certificate goes Ready. Needed because Pangolin proxies to **https:443**.
- [ ] **Pangolin:** add `bluecapstrategies.com` as a domain under org **`rhh-homelab`**
(`orgDomains`), `certResolver=letsencrypt`.
- [ ] **Pangolin:** create the resource(s) with **three targets**, matching the established
pattern for every k3s-backed resource (Authentik, paperless, immich, dav all do this):
`10.66.15.30`, `10.66.15.31`, `10.66.15.32` — method **https**, port **443**.
Set the host header so k3s traefik routes on the real hostname.
- [ ] 🛑 **Check Cloudflare's SSL/TLS mode in the dashboard** (the API token cannot read zone
settings). If it is **Flexible**, Cloudflare talks HTTP to Pangolin, Pangolin redirects to
HTTPS, and you get a redirect loop. Must be **Full (strict)** if the record stays proxied.
- [ ] **Test before any DNS change:** `curl --resolve bluecapstrategies.com:443:23.94.111.133`
and confirm the full path serves the site.
#### Proxied vs DNS-only — decide before cutover 🛑
The zone is currently **proxied**: apex `A → 50.87.136.52 PROXIED`, `www CNAME → apex PROXIED`.
- **DNS-only (recommended for cutover):** the exact path all 23 working resources use. CrowdSec
sees real client IPs as currently configured; HTTP-01 goes direct. Cost: no Cloudflare CDN/DDoS,
and the VPS IP becomes visible.
- **Stay proxied:** keeps CDN/DDoS and hides the origin, but CrowdSec will see Cloudflare IPs
unless it is configured to trust `CF-Connecting-IP` — bans would land on the wrong addresses or
on Cloudflare itself. Also requires Full (strict) per above.
Revisit proxying after launch as a deliberate change with the CrowdSec real-IP config done
properly — not as part of cutover.
#### Cutover
- [ ] Point apex + `www` at **23.94.111.133** in Cloudflare (replacing `50.87.136.52`).
- [ ] **Immediately** `curl https://bluecapstrategies.com` yourself to trigger Pangolin's HTTP-01
issuance, before a real visitor hits an unissued cert.
- [ ] Verify a valid chain on both hostnames and that HTTP→HTTPS redirects.
- [ ] Apply the canonical-host decision (§1.6): redirect the non-canonical name.
**Rollback:** set the A record back to `50.87.136.52`. All records are `ttl=1` (Cloudflare "Auto",
~300s for DNS-only), so propagation and rollback are both ~5 minutes — not hours.
#### Mail is not affected
MX (`eagle.mxlogin.com` / `eagle-relay.mxlogin.com`) and `mail.bluecapstrategies.com` are **not
proxied** and are untouched by the cutover. MXroute keeps serving mail throughout. See
`mxroute-mail-setup` notes.
- [ ] Minor cleanup (post-launch): `autoconfig` and `autodiscover` still point at Bluehost
(`50.87.136.52`) while mail is on MXroute — stale records that can hand mail clients the
wrong settings.
### 1.2 CI/CD is stalled — no deploy since 2026-07-20 🛑 hard blocker
**ROOT CAUSE FOUND AND FIXED IN-REPO 2026-09-22.** Not a runner problem — the runner was online