docs: replace section 1.1 with a verified cutover runbook
The old entry assumed the cluster could be pointed at directly and that DNS-01 removed the certificate ordering problem. Both were wrong once the real traffic path was checked. Public traffic has to transit the Pangolin VPS, so two certificates are needed from two different systems: the edge cert from Pangolin's own Traefik resolver, and the internal-hop cert from cert-manager, because every k3s-backed Pangolin resource proxies to https:443 (the pattern Authentik, paperless, immich and dav all follow). Pangolin's resolver uses HTTP-01 on entrypoint web, not DNS-01, so the edge cert cannot be pre-issued and a short window at cutover is unavoidable. cert-manager can still pre-issue the internal cert. There is also only one Cloudflare token, not two: Pangolin holds no Cloudflare credentials at all. Records the proxied-vs-DNS-only decision, which earlier advice got backwards (this zone is already orange-clouded while the homelab zones are not), the Flexible-SSL redirect-loop risk, the ~5 minute rollback from ttl=auto, and that mail is unaffected because the MX records are not proxied. Cloudflare token now covers bluecapstrategies.com.
This commit is contained in:
bmr_bluecap committed
1 parent
b51a243728
commit
a76a6da26c
1 file changed
+87
-13
+87
-13
@@ -27,6 +27,9 @@ Status legend: `[ ]` open · `[x]` done · `[?]` needs verification · 🛑 owne
|
||||
| `npm run test:e2e` | **44 passed / 4 failed** (the 4 are stale assertions, see §2.1) |
|
||||
| CI root cause | **found + fixed in-repo 2026-09-22** — see §1.2; needs a green run to confirm |
|
||||
| Staging URL | **https://bcs.int.7tl-homelab.com** — LAN-only, valid cert (§2.6) |
|
||||
| Public traffic path | Cloudflare → Pangolin VPS 23.94.111.133 → newt tunnel → k3s :443 (§1.1) |
|
||||
| Pangolin | org `rhh-homelab`, site `7tl_lab` (newt, online), 23 resources, Pangolin 1.22.2 |
|
||||
| Contact form | **working** — Formspree, verified end to end 2026-10-09 (§1.4) |
|
||||
| Old WordPress site | untouched and serving — must stay that way until cutover |
|
||||
| Gitea host | KVM VM `gitea-1` (10.66.15.22), **Debian 12**, docker compose — *not* in k3s |
|
||||
| Gitea / act_runner | 1.27.3 / 3.4.2 — both healthy |
|
||||
@@ -41,19 +44,90 @@ work is not live.
|
||||
|
||||
Nothing below is optional. Cutover is unsafe until all of §1 is closed.
|
||||
|
||||
### 1.1 TLS is not configured 🛑 hard blocker
|
||||
- [ ] Uncomment the `tls:` block and `cert-manager.io/cluster-issuer` annotation in
|
||||
`k8s/ingress.yaml`; set `secretName: bluecap-website-tls`.
|
||||
- [ ] Use the existing `letsencrypt-prod` ClusterIssuer (present in-cluster, 472d old, Ready).
|
||||
- [x] **Ordering problem SOLVED.** `letsencrypt-prod` solves via **DNS-01 against Cloudflare**,
|
||||
not HTTP-01, so a cert can be issued *before* any DNS points at the cluster — no insecure
|
||||
window. Proven end to end on 2026-09-23: the staging host issued a valid LE cert in ~75s
|
||||
with no public reachability (§2.6).
|
||||
- [ ] 🛑 **Verify the one remaining dependency:** `bluecapstrategies.com` must be in the
|
||||
Cloudflare account that cert-manager's API token can reach. The domain is at Bluehost
|
||||
today, so this is probably **not** yet true. It is the single fact separating a clean
|
||||
zero-downtime cutover from one with a gap — check it early, not on launch day.
|
||||
- [ ] Verify `https://` serves a valid chain and HTTP→HTTPS redirects before announcing.
|
||||
### 1.1 Cutover runbook — traffic path and TLS 🛑 hard blocker
|
||||
Verified against the live Cloudflare zone, the Pangolin VPS and the cluster on **2026-10-09**.
|
||||
|
||||
#### The actual traffic path
|
||||
Public traffic cannot reach k3s directly; it has to transit the Pangolin VPS.
|
||||
|
||||
```
|
||||
visitor → Cloudflare DNS (bluecapstrategies.com)
|
||||
→ Pangolin VPS 23.94.111.133 (Traefik v3.7.13 + CrowdSec, terminates TLS)
|
||||
→ gerbil/newt tunnel → site `7tl_lab` (org `rhh-homelab`)
|
||||
→ k3s traefik 10.66.15.30/.31/.32 :443 (second TLS hop)
|
||||
→ bluecap-website service
|
||||
```
|
||||
|
||||
**Two certificates are needed, from two different systems:**
|
||||
|
||||
| Hop | Issued by | Challenge | Can pre-issue? |
|
||||
|-----|-----------|-----------|----------------|
|
||||
| edge (Pangolin Traefik) | its own `letsencrypt` resolver | **HTTP-01** on entrypoint `web` | **No** |
|
||||
| internal (k3s ingress) | cert-manager `letsencrypt-prod` | **DNS-01** via Cloudflare | **Yes** |
|
||||
|
||||
#### Corrections to earlier assumptions in this doc
|
||||
- There is **one** Cloudflare token, not two. Pangolin's Traefik has **no** Cloudflare
|
||||
credentials — it uses HTTP-01. Nothing to configure on the VPS for DNS.
|
||||
- The "ordering problem SOLVED" claim was **half right**. cert-manager does use DNS-01 and can
|
||||
pre-issue (proven on the staging host, §2.6). But **Pangolin cannot** — its
|
||||
`certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint: web` means the edge cert is
|
||||
only obtainable once DNS already points at the VPS. A short window at cutover is unavoidable
|
||||
unless the edge resolver is converted to DNS-01, which would change renewal behaviour for all
|
||||
23 existing resources and is not worth it to save seconds.
|
||||
- Earlier advice said "use DNS-only, matching your existing pattern." The BlueCap zone is in fact
|
||||
**already fully proxied** (orange cloud) while the homelab zones are DNS-only. It is a real
|
||||
decision — see below.
|
||||
|
||||
#### Prerequisites (all invisible to the public; the old site keeps serving)
|
||||
- [x] Cloudflare token now covers the zone — `7tl-homelab.com`, **`bluecapstrategies.com`**,
|
||||
`rodgersweb.com` (done 2026-10-09). Token is DNS-edit-only, which is correct least privilege.
|
||||
- [ ] **k3s ingress:** add `www.bluecapstrategies.com` + `bluecapstrategies.com` to the `tls:`
|
||||
block in `k8s/ingress.yaml` (issuer annotation is already present and correct —
|
||||
`letsencrypt-prod`). cert-manager pre-issues via DNS-01, so do this **before** cutover and
|
||||
confirm the Certificate goes Ready. Needed because Pangolin proxies to **https:443**.
|
||||
- [ ] **Pangolin:** add `bluecapstrategies.com` as a domain under org **`rhh-homelab`**
|
||||
(`orgDomains`), `certResolver=letsencrypt`.
|
||||
- [ ] **Pangolin:** create the resource(s) with **three targets**, matching the established
|
||||
pattern for every k3s-backed resource (Authentik, paperless, immich, dav all do this):
|
||||
`10.66.15.30`, `10.66.15.31`, `10.66.15.32` — method **https**, port **443**.
|
||||
Set the host header so k3s traefik routes on the real hostname.
|
||||
- [ ] 🛑 **Check Cloudflare's SSL/TLS mode in the dashboard** (the API token cannot read zone
|
||||
settings). If it is **Flexible**, Cloudflare talks HTTP to Pangolin, Pangolin redirects to
|
||||
HTTPS, and you get a redirect loop. Must be **Full (strict)** if the record stays proxied.
|
||||
- [ ] **Test before any DNS change:** `curl --resolve bluecapstrategies.com:443:23.94.111.133`
|
||||
and confirm the full path serves the site.
|
||||
|
||||
#### Proxied vs DNS-only — decide before cutover 🛑
|
||||
The zone is currently **proxied**: apex `A → 50.87.136.52 PROXIED`, `www CNAME → apex PROXIED`.
|
||||
|
||||
- **DNS-only (recommended for cutover):** the exact path all 23 working resources use. CrowdSec
|
||||
sees real client IPs as currently configured; HTTP-01 goes direct. Cost: no Cloudflare CDN/DDoS,
|
||||
and the VPS IP becomes visible.
|
||||
- **Stay proxied:** keeps CDN/DDoS and hides the origin, but CrowdSec will see Cloudflare IPs
|
||||
unless it is configured to trust `CF-Connecting-IP` — bans would land on the wrong addresses or
|
||||
on Cloudflare itself. Also requires Full (strict) per above.
|
||||
|
||||
Revisit proxying after launch as a deliberate change with the CrowdSec real-IP config done
|
||||
properly — not as part of cutover.
|
||||
|
||||
#### Cutover
|
||||
- [ ] Point apex + `www` at **23.94.111.133** in Cloudflare (replacing `50.87.136.52`).
|
||||
- [ ] **Immediately** `curl https://bluecapstrategies.com` yourself to trigger Pangolin's HTTP-01
|
||||
issuance, before a real visitor hits an unissued cert.
|
||||
- [ ] Verify a valid chain on both hostnames and that HTTP→HTTPS redirects.
|
||||
- [ ] Apply the canonical-host decision (§1.6): redirect the non-canonical name.
|
||||
|
||||
**Rollback:** set the A record back to `50.87.136.52`. All records are `ttl=1` (Cloudflare "Auto",
|
||||
~300s for DNS-only), so propagation and rollback are both ~5 minutes — not hours.
|
||||
|
||||
#### Mail is not affected
|
||||
MX (`eagle.mxlogin.com` / `eagle-relay.mxlogin.com`) and `mail.bluecapstrategies.com` are **not
|
||||
proxied** and are untouched by the cutover. MXroute keeps serving mail throughout. See
|
||||
`mxroute-mail-setup` notes.
|
||||
|
||||
- [ ] Minor cleanup (post-launch): `autoconfig` and `autodiscover` still point at Bluehost
|
||||
(`50.87.136.52`) while mail is on MXroute — stale records that can hand mail clients the
|
||||
wrong settings.
|
||||
|
||||
### 1.2 CI/CD is stalled — no deploy since 2026-07-20 🛑 hard blocker
|
||||
**ROOT CAUSE FOUND AND FIXED IN-REPO 2026-09-22.** Not a runner problem — the runner was online
|
||||
|
||||
Reference in new issue
Block a user