Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6f802826df | ||
|
|
a76a6da26c |
No files matched your search
+87
-13
@@ -27,6 +27,9 @@ Status legend: `[ ]` open · `[x]` done · `[?]` needs verification · 🛑 owne
|
||||
| `npm run test:e2e` | **44 passed / 4 failed** (the 4 are stale assertions, see §2.1) |
|
||||
| CI root cause | **found + fixed in-repo 2026-09-22** — see §1.2; needs a green run to confirm |
|
||||
| Staging URL | **https://bcs.int.7tl-homelab.com** — LAN-only, valid cert (§2.6) |
|
||||
| Public traffic path | Cloudflare → Pangolin VPS 23.94.111.133 → newt tunnel → k3s :443 (§1.1) |
|
||||
| Pangolin | org `rhh-homelab`, site `7tl_lab` (newt, online), 23 resources, Pangolin 1.22.2 |
|
||||
| Contact form | **working** — Formspree, verified end to end 2026-10-09 (§1.4) |
|
||||
| Old WordPress site | untouched and serving — must stay that way until cutover |
|
||||
| Gitea host | KVM VM `gitea-1` (10.66.15.22), **Debian 12**, docker compose — *not* in k3s |
|
||||
| Gitea / act_runner | 1.27.3 / 3.4.2 — both healthy |
|
||||
@@ -41,19 +44,90 @@ work is not live.
|
||||
|
||||
Nothing below is optional. Cutover is unsafe until all of §1 is closed.
|
||||
|
||||
### 1.1 TLS is not configured 🛑 hard blocker
|
||||
- [ ] Uncomment the `tls:` block and `cert-manager.io/cluster-issuer` annotation in
|
||||
`k8s/ingress.yaml`; set `secretName: bluecap-website-tls`.
|
||||
- [ ] Use the existing `letsencrypt-prod` ClusterIssuer (present in-cluster, 472d old, Ready).
|
||||
- [x] **Ordering problem SOLVED.** `letsencrypt-prod` solves via **DNS-01 against Cloudflare**,
|
||||
not HTTP-01, so a cert can be issued *before* any DNS points at the cluster — no insecure
|
||||
window. Proven end to end on 2026-09-23: the staging host issued a valid LE cert in ~75s
|
||||
with no public reachability (§2.6).
|
||||
- [ ] 🛑 **Verify the one remaining dependency:** `bluecapstrategies.com` must be in the
|
||||
Cloudflare account that cert-manager's API token can reach. The domain is at Bluehost
|
||||
today, so this is probably **not** yet true. It is the single fact separating a clean
|
||||
zero-downtime cutover from one with a gap — check it early, not on launch day.
|
||||
- [ ] Verify `https://` serves a valid chain and HTTP→HTTPS redirects before announcing.
|
||||
### 1.1 Cutover runbook — traffic path and TLS 🛑 hard blocker
|
||||
Verified against the live Cloudflare zone, the Pangolin VPS and the cluster on **2026-10-09**.
|
||||
|
||||
#### The actual traffic path
|
||||
Public traffic cannot reach k3s directly; it has to transit the Pangolin VPS.
|
||||
|
||||
```
|
||||
visitor → Cloudflare DNS (bluecapstrategies.com)
|
||||
→ Pangolin VPS 23.94.111.133 (Traefik v3.7.13 + CrowdSec, terminates TLS)
|
||||
→ gerbil/newt tunnel → site `7tl_lab` (org `rhh-homelab`)
|
||||
→ k3s traefik 10.66.15.30/.31/.32 :443 (second TLS hop)
|
||||
→ bluecap-website service
|
||||
```
|
||||
|
||||
**Two certificates are needed, from two different systems:**
|
||||
|
||||
| Hop | Issued by | Challenge | Can pre-issue? |
|
||||
|-----|-----------|-----------|----------------|
|
||||
| edge (Pangolin Traefik) | its own `letsencrypt` resolver | **HTTP-01** on entrypoint `web` | **No** |
|
||||
| internal (k3s ingress) | cert-manager `letsencrypt-prod` | **DNS-01** via Cloudflare | **Yes** |
|
||||
|
||||
#### Corrections to earlier assumptions in this doc
|
||||
- There is **one** Cloudflare token, not two. Pangolin's Traefik has **no** Cloudflare
|
||||
credentials — it uses HTTP-01. Nothing to configure on the VPS for DNS.
|
||||
- The "ordering problem SOLVED" claim was **half right**. cert-manager does use DNS-01 and can
|
||||
pre-issue (proven on the staging host, §2.6). But **Pangolin cannot** — its
|
||||
`certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint: web` means the edge cert is
|
||||
only obtainable once DNS already points at the VPS. A short window at cutover is unavoidable
|
||||
unless the edge resolver is converted to DNS-01, which would change renewal behaviour for all
|
||||
23 existing resources and is not worth it to save seconds.
|
||||
- Earlier advice said "use DNS-only, matching your existing pattern." The BlueCap zone is in fact
|
||||
**already fully proxied** (orange cloud) while the homelab zones are DNS-only. It is a real
|
||||
decision — see below.
|
||||
|
||||
#### Prerequisites (all invisible to the public; the old site keeps serving)
|
||||
- [x] Cloudflare token now covers the zone — `7tl-homelab.com`, **`bluecapstrategies.com`**,
|
||||
`rodgersweb.com` (done 2026-10-09). Token is DNS-edit-only, which is correct least privilege.
|
||||
- [ ] **k3s ingress:** add `www.bluecapstrategies.com` + `bluecapstrategies.com` to the `tls:`
|
||||
block in `k8s/ingress.yaml` (issuer annotation is already present and correct —
|
||||
`letsencrypt-prod`). cert-manager pre-issues via DNS-01, so do this **before** cutover and
|
||||
confirm the Certificate goes Ready. Needed because Pangolin proxies to **https:443**.
|
||||
- [ ] **Pangolin:** add `bluecapstrategies.com` as a domain under org **`rhh-homelab`**
|
||||
(`orgDomains`), `certResolver=letsencrypt`.
|
||||
- [ ] **Pangolin:** create the resource(s) with **three targets**, matching the established
|
||||
pattern for every k3s-backed resource (Authentik, paperless, immich, dav all do this):
|
||||
`10.66.15.30`, `10.66.15.31`, `10.66.15.32` — method **https**, port **443**.
|
||||
Set the host header so k3s traefik routes on the real hostname.
|
||||
- [ ] 🛑 **Check Cloudflare's SSL/TLS mode in the dashboard** (the API token cannot read zone
|
||||
settings). If it is **Flexible**, Cloudflare talks HTTP to Pangolin, Pangolin redirects to
|
||||
HTTPS, and you get a redirect loop. Must be **Full (strict)** if the record stays proxied.
|
||||
- [ ] **Test before any DNS change:** `curl --resolve bluecapstrategies.com:443:23.94.111.133`
|
||||
and confirm the full path serves the site.
|
||||
|
||||
#### Proxied vs DNS-only — decide before cutover 🛑
|
||||
The zone is currently **proxied**: apex `A → 50.87.136.52 PROXIED`, `www CNAME → apex PROXIED`.
|
||||
|
||||
- **DNS-only (recommended for cutover):** the exact path all 23 working resources use. CrowdSec
|
||||
sees real client IPs as currently configured; HTTP-01 goes direct. Cost: no Cloudflare CDN/DDoS,
|
||||
and the VPS IP becomes visible.
|
||||
- **Stay proxied:** keeps CDN/DDoS and hides the origin, but CrowdSec will see Cloudflare IPs
|
||||
unless it is configured to trust `CF-Connecting-IP` — bans would land on the wrong addresses or
|
||||
on Cloudflare itself. Also requires Full (strict) per above.
|
||||
|
||||
Revisit proxying after launch as a deliberate change with the CrowdSec real-IP config done
|
||||
properly — not as part of cutover.
|
||||
|
||||
#### Cutover
|
||||
- [ ] Point apex + `www` at **23.94.111.133** in Cloudflare (replacing `50.87.136.52`).
|
||||
- [ ] **Immediately** `curl https://bluecapstrategies.com` yourself to trigger Pangolin's HTTP-01
|
||||
issuance, before a real visitor hits an unissued cert.
|
||||
- [ ] Verify a valid chain on both hostnames and that HTTP→HTTPS redirects.
|
||||
- [ ] Apply the canonical-host decision (§1.6): redirect the non-canonical name.
|
||||
|
||||
**Rollback:** set the A record back to `50.87.136.52`. All records are `ttl=1` (Cloudflare "Auto",
|
||||
~300s for DNS-only), so propagation and rollback are both ~5 minutes — not hours.
|
||||
|
||||
#### Mail is not affected
|
||||
MX (`eagle.mxlogin.com` / `eagle-relay.mxlogin.com`) and `mail.bluecapstrategies.com` are **not
|
||||
proxied** and are untouched by the cutover. MXroute keeps serving mail throughout. See
|
||||
`mxroute-mail-setup` notes.
|
||||
|
||||
- [ ] Minor cleanup (post-launch): `autoconfig` and `autodiscover` still point at Bluehost
|
||||
(`50.87.136.52`) while mail is on MXroute — stale records that can hand mail clients the
|
||||
wrong settings.
|
||||
|
||||
### 1.2 CI/CD is stalled — no deploy since 2026-07-20 🛑 hard blocker
|
||||
**ROOT CAUSE FOUND AND FIXED IN-REPO 2026-09-22.** Not a runner problem — the runner was online
|
||||
|
||||
+17
-1
@@ -25,10 +25,26 @@ metadata:
|
||||
cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
spec:
|
||||
tls:
|
||||
# Staging only, for now. Production hosts join this list at cutover (see header).
|
||||
- hosts:
|
||||
- bcs.int.7tl-homelab.com
|
||||
secretName: bcs-int-tls
|
||||
# Production hostnames. This is the INTERNAL hop only: public traffic terminates TLS at the
|
||||
# Pangolin VPS, which then proxies to these nodes over https:443 (the pattern every other
|
||||
# k3s-backed Pangolin resource uses), so the cluster needs its own valid cert for these names.
|
||||
#
|
||||
# Safe to issue before cutover and deliberately done so: letsencrypt-prod solves DNS-01 via
|
||||
# Cloudflare, which only needs a temporary TXT record in the zone — no A record has to point
|
||||
# here, and the live WordPress site is unaffected. Pangolin's own edge cert is HTTP-01 and
|
||||
# CANNOT be pre-issued; it is obtained on the first request after DNS moves. See
|
||||
# docs/launch-checklist.md section 1.1.
|
||||
#
|
||||
# Both names are listed because the canonical-host decision (section 1.6) is still open;
|
||||
# holding a cert for each keeps either choice available. The redirect is configured
|
||||
# separately and does not depend on this.
|
||||
- hosts:
|
||||
- www.bluecapstrategies.com
|
||||
- bluecapstrategies.com
|
||||
secretName: bluecap-website-tls
|
||||
rules:
|
||||
- host: bcs.int.7tl-homelab.com
|
||||
http:
|
||||
|
||||
Reference in new issue
Block a user