docs: rewrite launch checklist as post-cutover status
Records the live architecture, launch-day findings (Pangolin cert retry, health-check Host header, nginx 404 loop, CI rebase, homelab DNS interception, runner socket) and the remaining open items.
This commit is contained in:
bmr_bluecap committed
1 parent
c7033140f0
commit
450bba8bea
1 file changed
+148
-368
+148
-368
@@ -1,416 +1,196 @@
|
|||||||
# Launch Checklist — DNS Cutover to the Astro Rebuild
|
# Launch Checklist — Post-Cutover Status
|
||||||
|
|
||||||
**Single source of truth** for everything standing between today and pointing
|
`www.bluecapstrategies.com` went **live on the Astro rebuild on 2026-10-23**. This file is now the
|
||||||
`bluecapstrategies.com` at the homelab k3s deployment.
|
single source of truth for what is running, how it is wired, and what is left. The pre-launch
|
||||||
|
runbook it replaced is in git history (`git log -- docs/launch-checklist.md`, before 2026-10-23).
|
||||||
Verified against the working tree, the cluster, and ArgoCD on **2026-09-18**. Supersedes the
|
|
||||||
task lists that were scattered across `planning/parity-checklist.md` (git-ignored, content-only)
|
|
||||||
and `docs/seo-aeo-optimization.md` §2/§7 (SEO-only, stale) — those now point here.
|
|
||||||
|
|
||||||
Status legend: `[ ]` open · `[x]` done · `[?]` needs verification · 🛑 owner decision
|
Status legend: `[ ]` open · `[x]` done · `[?]` needs verification · 🛑 owner decision
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 0. As-built state (2026-09-18)
|
## 0. As-built state (2026-10-23)
|
||||||
|
|
||||||
| Thing | State |
|
| Thing | State |
|
||||||
|-------|-------|
|
|-------|-------|
|
||||||
| ArgoCD app `bcs-website` | Synced / Healthy |
|
| Public site | **https://www.bluecapstrategies.com** — canonical. Apex 301s to www, path preserved |
|
||||||
| Pods | 2/2 Running, ns `bluecap-strategies` |
|
| Old WordPress / Bluehost | **Gone.** Access lost and it was returning 404 before cutover; nothing to roll back to |
|
||||||
| Deployed image | `git.7tl-homelab.com/bmr_bluecap/bcs-website:fc64909f3df1` (built **2026-07-20**) |
|
| DNS | Cloudflare, **DNS-only (grey cloud)** for apex, `www`, `stats` → `23.94.111.133` |
|
||||||
| `k8s/kustomization.yaml` newTag | `fc64909f3df1` — same, so ArgoCD is correct; CI never bumped it |
|
| Edge | Pangolin VPS 23.94.111.133 (Traefik + CrowdSec), org `rhh-homelab`, site `7tl_lab` |
|
||||||
| HEAD | `9f1cbca` (2026-08-03), pushed to `origin/main`, **never deployed** |
|
| Pangolin resources | `Bluecap-www`, `Bluecap-base`, `stats` — 3 https targets `10.66.15.30/.31/.32:443` each |
|
||||||
| Ingress | `bluecap-website` (traefik), hosts apex + `www`, **port 80 only, no TLS** |
|
| Edge certs | Let's Encrypt HTTP-01 via Pangolin's `letsencrypt` resolver, issued 2026-10-23 |
|
||||||
| `npm run build` | green, 21 pages, 0 errors / 0 warnings |
|
| Cluster certs | cert-manager `letsencrypt-prod` DNS-01: `bluecap-website-tls` (www + apex), `bcs-int-tls` |
|
||||||
| `npm run typecheck` | green, 28 files, 0/0/0 |
|
| ArgoCD | `bcs-website` and `umami` — Synced / Healthy |
|
||||||
| `npm run seo:lint` | green, 21 pages clean |
|
| Deployed image | tracks `main` via CI tag-bump (`ci: deploy <sha>` commits) |
|
||||||
| `npm run test:e2e` | **44 passed / 4 failed** (the 4 are stale assertions, see §2.1) |
|
| Staging | https://bcs.int.7tl-homelab.com — LAN only |
|
||||||
| CI root cause | **found + fixed in-repo 2026-09-22** — see §1.2; needs a green run to confirm |
|
| Contact form | Formspree, verified end to end 2026-10-09 |
|
||||||
| Staging URL | **https://bcs.int.7tl-homelab.com** — LAN-only, valid cert (§2.6) |
|
| Analytics | **Umami** (self-hosted, `stats.bluecapstrategies.com`) + **GA4** `G-Z273QL1F0N`, both verified recording 2026-10-23 |
|
||||||
| Public traffic path | Cloudflare → Pangolin VPS 23.94.111.133 → newt tunnel → k3s :443 (§1.1) |
|
| Monitoring | monylog blackbox probes on www + apex → Alertmanager → **Pushover** |
|
||||||
| Pangolin | org `rhh-homelab`, site `7tl_lab` (newt, online), 23 resources, Pangolin 1.22.2 |
|
| Search | Google Search Console domain property verified; sitemap submitted and fetched by Googlebot |
|
||||||
| Contact form | **working** — Formspree, verified end to end 2026-10-09 (§1.4) |
|
| Mail | MXroute. SPF pass, DKIM pass (selector `x`), DMARC `p=none` — mail-tester 9.5/10 |
|
||||||
| Old WordPress site | untouched and serving — must stay that way until cutover |
|
| `npm run build` / `typecheck` / `seo:lint` | green, 21 pages |
|
||||||
| Gitea host | KVM VM `gitea-1` (10.66.15.22), **Debian 12**, docker compose — *not* in k3s |
|
| `npm run test:e2e` | 44 passed / 4 failed — stale assertions, see §2.1 |
|
||||||
| Gitea / act_runner | 1.27.3 / 3.4.2 — both healthy |
|
|
||||||
| CI job image | **`node:22-bookworm`** (Debian 12) as of 2026-09-22 — was EOL bullseye; see §1.2 |
|
|
||||||
|
|
||||||
**The headline:** the public-facing deployment is ~7 weeks behind `main`. The Aug 3 exact-copy
|
### Traffic path
|
||||||
work is not live.
|
```
|
||||||
|
visitor → Cloudflare DNS (DNS-only) → Pangolin VPS 23.94.111.133 (TLS #1, CrowdSec)
|
||||||
|
→ gerbil/newt tunnel → k3s traefik 10.66.15.30/.31/.32 :443 (TLS #2)
|
||||||
|
→ bluecap-website (nginx, 2 replicas) ns bluecap-strategies
|
||||||
|
→ umami (+ umami-postgres) ns umami
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Cutover blockers
|
## 1. Open — do soon
|
||||||
|
|
||||||
Nothing below is optional. Cutover is unsafe until all of §1 is closed.
|
### 1.1 External uptime check 🛑
|
||||||
|
monylog runs inside the homelab, so a homelab power or internet outage takes the site **and** its
|
||||||
|
monitor down together, and nothing pages. Needs a vantage point outside the homelab.
|
||||||
|
- [ ] Decide: hosted (UptimeRobot / Healthchecks.io) or Uptime Kuma on the Pangolin VPS.
|
||||||
|
- [ ] Probe `https://www.bluecapstrategies.com/` and route alerts to Pushover.
|
||||||
|
|
||||||
### 1.1 Cutover runbook — traffic path and TLS 🛑 hard blocker
|
### 1.2 Homelab intercepts port-53 DNS
|
||||||
Verified against the live Cloudflare zone, the Pangolin VPS and the cluster on **2026-10-09**.
|
Found 2026-10-23. Queries from the homelab — including k3s nodes — to *any* DNS server, even
|
||||||
|
Cloudflare's authoritative NS, are answered by a local recursive resolver (`dig @arely.ns.cloudflare.com`
|
||||||
|
returns `rd ra` with no `aa`). A name looked up before its record exists gets **NXDOMAIN cached for
|
||||||
|
30 minutes** (the zone's SOA negative TTL).
|
||||||
|
|
||||||
#### The actual traffic path
|
This, not Cloudflare, explains the 4-hour cert-manager DNS-01 stall of 2026-10-09 and the
|
||||||
Public traffic cannot reach k3s directly; it has to transit the Pangolin VPS.
|
`stats` NXDOMAIN at launch. Consequences:
|
||||||
|
- Pointing cert-manager at `1.1.1.1/8.8.8.8` (the fix previously suggested here) **will not help**
|
||||||
|
while port 53 is intercepted.
|
||||||
|
- To see true public DNS, query from the VPS or via DoH:
|
||||||
|
`curl -H 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=<name>&type=A'`
|
||||||
|
- Create DNS records *before* anything in the homelab (cert-manager included) looks the name up.
|
||||||
|
|
||||||
```
|
- [ ] 🛑 Owner decision: leave as is, or exempt the k3s nodes from the redirect. The redirect rule
|
||||||
visitor → Cloudflare DNS (bluecapstrategies.com)
|
is not in `~/Homelab/mikrotik/*.rsc`; find it on the router (or Pi-hole/AdGuard, if any).
|
||||||
→ Pangolin VPS 23.94.111.133 (Traefik v3.7.13 + CrowdSec, terminates TLS)
|
- [?] `umami-stats-tls` (internal hop for `stats`) was still pending on the stale NXDOMAIN at
|
||||||
→ gerbil/newt tunnel → site `7tl_lab` (org `rhh-homelab`)
|
18:30 UTC on launch day; it should issue by itself once the cache expires. Visitor-facing
|
||||||
→ k3s traefik 10.66.15.30/.31/.32 :443 (second TLS hop)
|
TLS for `stats` is Pangolin's and is fine. Check with `kubectl -n umami get certificate`.
|
||||||
→ bluecap-website service
|
|
||||||
```
|
|
||||||
|
|
||||||
**Two certificates are needed, from two different systems:**
|
### 1.3 Gitea runner loses Docker after a dockerd restart
|
||||||
|
`dockerd` on `gitea-1` runs with `live-restore=true`. When it restarts, containers keep running
|
||||||
|
but `act_runner`'s bind mount of `/var/run/docker.sock` points at the dead socket, so every CI
|
||||||
|
job fails in its first second with `Cannot connect to the Docker daemon`. Happened 2026-10-21 →
|
||||||
|
discovered 2026-10-23.
|
||||||
|
- **Workaround:** `sudo docker restart act_runner` on gitea-1.
|
||||||
|
- [ ] Durable fix in the `gitea` ansible role: disable live-restore on that VM, or restart the
|
||||||
|
runner after dockerd restarts (systemd `PartOf=`/`ExecStartPost` hook).
|
||||||
|
|
||||||
| Hop | Issued by | Challenge | Can pre-issue? |
|
### 1.4 `/insights/` ships empty, with developer-facing copy 🛑
|
||||||
|-----|-----------|-----------|----------------|
|
`src/pages/insights/index.astro` shows two cards written for a developer ("Publishing path",
|
||||||
| edge (Pangolin Traefik) | its own `letsencrypt` resolver | **HTTP-01** on entrypoint `web` | **No** |
|
"RSS support") and a hero calling the section a "Future publishing home". The section is in the
|
||||||
| internal (k3s ingress) | cert-manager `letsencrypt-prod` | **DNS-01** via Cloudflare | **Yes** |
|
primary nav with zero articles.
|
||||||
|
- [ ] 🛑 Ship it or not.
|
||||||
|
- [ ] If yes: delete both note-cards and rewrite the hero as reader-facing copy.
|
||||||
|
- [ ] If no: drop it from the primary nav and exclude the route from the sitemap.
|
||||||
|
|
||||||
#### Corrections to earlier assumptions in this doc
|
### 1.5 Privacy policy review 🛑
|
||||||
- There is **one** Cloudflare token, not two. Pangolin's Traefik has **no** Cloudflare
|
`/privacy/` went live 2026-10-23, drafted from what the site actually does (contact form via
|
||||||
credentials — it uses HTTP-01. Nothing to configure on the VPS for DNS.
|
Formspree, Umami, GA4 with cookies, server/security logs).
|
||||||
- The "ordering problem SOLVED" claim was **half right**. cert-manager does use DNS-01 and can
|
- [ ] 🛑 Owner review of the text. It is a legal document.
|
||||||
pre-issue (proven on the staging host, §2.6). But **Pangolin cannot** — its
|
- [ ] 🛑 Consent banner: GA4 sets cookies. Strictly, EU/UK visitors need opt-in consent; most
|
||||||
`certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint: web` means the edge cert is
|
US-focused small-business sites run with a privacy policy only. Owner's call. Umami needs
|
||||||
only obtainable once DNS already points at the VPS. A short window at cutover is unavoidable
|
neither.
|
||||||
unless the edge resolver is converted to DNS-01, which would change renewal behaviour for all
|
- Anything added to `src/components/Analytics.astro` must be disclosed on `/privacy/`.
|
||||||
23 existing resources and is not worth it to save seconds.
|
|
||||||
- Earlier advice said "use DNS-only, matching your existing pattern." The BlueCap zone is in fact
|
|
||||||
**already fully proxied** (orange cloud) while the homelab zones are DNS-only. It is a real
|
|
||||||
decision — see below.
|
|
||||||
|
|
||||||
#### Prerequisites (all invisible to the public; the old site keeps serving)
|
|
||||||
- [x] Cloudflare token now covers the zone — `7tl-homelab.com`, **`bluecapstrategies.com`**,
|
|
||||||
`rodgersweb.com` (done 2026-10-09). Token is DNS-edit-only, which is correct least privilege.
|
|
||||||
- [x] **k3s ingress — DONE 2026-10-09 (commit `6f80282`).** Both production hostnames added to
|
|
||||||
the `tls:` block as `bluecap-website-tls`. Certificate **Ready**, issued by Let's Encrypt
|
|
||||||
via DNS-01: `CN=www.bluecapstrategies.com`, SANs `bluecapstrategies.com` +
|
|
||||||
`www.bluecapstrategies.com`, valid to **2027-01-07**, auto-renewing. Verified the cluster
|
|
||||||
serves it for both names (`--resolve` straight to 10.66.15.30 → 200, correct cert, rebuild
|
|
||||||
content). The public site is untouched and still on Cloudflare/Bluehost throughout.
|
|
||||||
Both names are on one cert, so the canonical-host decision stays open.
|
|
||||||
- Issuance stalled ~4h: the TXT records were present in Cloudflare's API immediately but
|
|
||||||
Cloudflare's own authoritative nameservers returned NXDOMAIN for them for a long while.
|
|
||||||
It cleared on its own. If this recurs, cert-manager's DNS-01 self-check resolver is the
|
|
||||||
thing to look at — it is currently CoreDNS (`nameservers=["10.43.0.10:53"]`), and the
|
|
||||||
usual hardening is to point it at public resolvers instead:
|
|
||||||
`--dns01-recursive-nameservers-only=true --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53`.
|
|
||||||
Worth doing before cutover day so renewals do not inherit this fragility (§2.7).
|
|
||||||
- [ ] **Pangolin:** add `bluecapstrategies.com` as a domain under org **`rhh-homelab`**
|
|
||||||
(`orgDomains`), `certResolver=letsencrypt`.
|
|
||||||
- [ ] **Pangolin:** create the resource(s) with **three targets**, matching the established
|
|
||||||
pattern for every k3s-backed resource (Authentik, paperless, immich, dav all do this):
|
|
||||||
`10.66.15.30`, `10.66.15.31`, `10.66.15.32` — method **https**, port **443**.
|
|
||||||
Set the host header so k3s traefik routes on the real hostname.
|
|
||||||
- [ ] 🛑 **Check Cloudflare's SSL/TLS mode in the dashboard** (the API token cannot read zone
|
|
||||||
settings). If it is **Flexible**, Cloudflare talks HTTP to Pangolin, Pangolin redirects to
|
|
||||||
HTTPS, and you get a redirect loop. Must be **Full (strict)** if the record stays proxied.
|
|
||||||
- [ ] **Test before any DNS change:** `curl --resolve bluecapstrategies.com:443:23.94.111.133`
|
|
||||||
and confirm the full path serves the site.
|
|
||||||
|
|
||||||
#### Pangolin resource targets (for step 2)
|
|
||||||
Three targets, matching every other k3s-backed resource:
|
|
||||||
|
|
||||||
| Target | Method | Port |
|
|
||||||
|--------|--------|------|
|
|
||||||
| `10.66.15.30` | https | 443 |
|
|
||||||
| `10.66.15.31` | https | 443 |
|
|
||||||
| `10.66.15.32` | https | 443 |
|
|
||||||
|
|
||||||
`k3-control-1/2/3`. Three gives node-level failover. Set the host header to the real hostname so
|
|
||||||
k3s traefik routes on it. The cluster now holds a valid cert for both production names, so the
|
|
||||||
https hop verifies properly.
|
|
||||||
|
|
||||||
#### Proxied vs DNS-only — decide before cutover 🛑
|
|
||||||
The zone is currently **proxied**: apex `A → 50.87.136.52 PROXIED`, `www CNAME → apex PROXIED`.
|
|
||||||
|
|
||||||
- **DNS-only (recommended for cutover):** the exact path all 23 working resources use. CrowdSec
|
|
||||||
sees real client IPs as currently configured; HTTP-01 goes direct. Cost: no Cloudflare CDN/DDoS,
|
|
||||||
and the VPS IP becomes visible.
|
|
||||||
- **Stay proxied:** keeps CDN/DDoS and hides the origin, but CrowdSec will see Cloudflare IPs
|
|
||||||
unless it is configured to trust `CF-Connecting-IP` — bans would land on the wrong addresses or
|
|
||||||
on Cloudflare itself. Also requires Full (strict) per above.
|
|
||||||
|
|
||||||
Revisit proxying after launch as a deliberate change with the CrowdSec real-IP config done
|
|
||||||
properly — not as part of cutover.
|
|
||||||
|
|
||||||
#### Cutover
|
|
||||||
- [ ] Point apex + `www` at **23.94.111.133** in Cloudflare (replacing `50.87.136.52`).
|
|
||||||
- [ ] **Immediately** `curl https://bluecapstrategies.com` yourself to trigger Pangolin's HTTP-01
|
|
||||||
issuance, before a real visitor hits an unissued cert.
|
|
||||||
- [ ] Verify a valid chain on both hostnames and that HTTP→HTTPS redirects.
|
|
||||||
- [ ] Apply the canonical-host decision (§1.6): redirect the non-canonical name.
|
|
||||||
|
|
||||||
**Rollback:** set the A record back to `50.87.136.52`. All records are `ttl=1` (Cloudflare "Auto",
|
|
||||||
~300s for DNS-only), so propagation and rollback are both ~5 minutes — not hours.
|
|
||||||
|
|
||||||
#### Mail is not affected
|
|
||||||
MX (`eagle.mxlogin.com` / `eagle-relay.mxlogin.com`) and `mail.bluecapstrategies.com` are **not
|
|
||||||
proxied** and are untouched by the cutover. MXroute keeps serving mail throughout. See
|
|
||||||
`mxroute-mail-setup` notes.
|
|
||||||
|
|
||||||
- [ ] Minor cleanup (post-launch): `autoconfig` and `autodiscover` still point at Bluehost
|
|
||||||
(`50.87.136.52`) while mail is on MXroute — stale records that can hand mail clients the
|
|
||||||
wrong settings.
|
|
||||||
|
|
||||||
### 1.2 CI/CD is stalled — no deploy since 2026-07-20 🛑 hard blocker
|
|
||||||
**ROOT CAUSE FOUND AND FIXED IN-REPO 2026-09-22.** Not a runner problem — the runner was online
|
|
||||||
and the run *did* execute.
|
|
||||||
|
|
||||||
Evidence (Gitea 1.27.3 on `gitea-1`, sqlite `action_run` for repo_id 3):
|
|
||||||
- Run #6, commit `9f1cbca1b148`, created 2026-08-03 13:46:53, **status 2 = failure**, ran 13m
|
|
||||||
(13:47:02 → 14:00:11) on runner 59.
|
|
||||||
- No `9f1cbca1b148` entry in `package_version` → it died **at or before** the image push.
|
|
||||||
- Job logs are past retention (`/data/gitea/actions_log` no longer holds run 205), so the cause
|
|
||||||
was reproduced directly instead.
|
|
||||||
|
|
||||||
**The cause:** the `Ensure docker + kustomize are available` step ran
|
|
||||||
`apt-get install -y docker.io`. The job container image is **`node:18-bullseye`** (Debian 11),
|
|
||||||
and bullseye went EOL — `debian-security` no longer serves those package versions. `apt-get
|
|
||||||
update` still succeeds, so this fails only at install time, with 404s on `docker.io`,
|
|
||||||
`containerd`, `systemd`, `libcap2`. Reproduced verbatim in a `node:18-bullseye` container on the
|
|
||||||
runner host. That timing matches exactly: last green run 2026-07-20, first failure 2026-08-03.
|
|
||||||
|
|
||||||
**The fix (applied to `.gitea/workflows/ci.yml`):** fetch the docker CLI and kustomize as pinned
|
|
||||||
**static binaries** (`download.docker.com` / GitHub releases) instead of distro packages. Verified
|
|
||||||
in the real job environment: the static client installs, reports `server=28.1.1` through the
|
|
||||||
mounted socket, and `docker ps` succeeds. kustomize tarball and `git` are fine.
|
|
||||||
|
|
||||||
Everything else on the runner path checked out and needs no change:
|
|
||||||
- [x] act_runner container up, `v3.4.2`, registered and polling (the daily 07:30 "connection
|
|
||||||
refused" entries are the Gitea restart window, harmless).
|
|
||||||
- [x] Runner scope **already fixed** — runner id 59 is `owner_id=0, repo_id=0` (instance-scoped),
|
|
||||||
so the old user-scoped problem is resolved. 58 stale duplicate runner rows remain; cosmetic,
|
|
||||||
prune with the role's `gitea_runner_prune` tag when convenient.
|
|
||||||
- [x] Runner host dockerd trusts `10.66.15.22:3000` as insecure — confirmed in `docker info`.
|
|
||||||
- [x] `config.yaml` pins jobs to `gitea_network` and passes the docker socket — correct.
|
|
||||||
- [x] `CI_TOKEN` repo secret present (repo_id 3).
|
|
||||||
- [x] `paths-ignore` did not swallow the push — the commit touched `src/**`.
|
|
||||||
- [x] `docker build` of the current tree succeeds locally, so the Dockerfile and lockfile are fine.
|
|
||||||
|
|
||||||
Remaining:
|
|
||||||
- [ ] **Re-trigger CI and confirm a green run** — the fix is verified at the step level but the
|
|
||||||
full pipeline has not run end-to-end since July. Requires a push touching a non-ignored path.
|
|
||||||
- [ ] Confirm the tag-bump commit lands and ArgoCD syncs the new image.
|
|
||||||
- [ ] Do not cut DNS onto a stale image — confirm the deployed tag matches HEAD.
|
|
||||||
- [x] **CI job image upgraded 2026-09-22.** `gitea_runner_labels` in
|
|
||||||
`~/Homelab/ansible/roles/gitea/defaults/main.yml` changed `node:18-bullseye` →
|
|
||||||
**`node:22-bookworm`**; playbook run succeeded (ok=40 changed=8 failed=0). Runner **60**
|
|
||||||
registered and online, `owner_id=0` (instance-scoped), label confirmed in `/data/.runner`.
|
|
||||||
The job image now matches the host VM and k3s nodes (Debian 12) and the Node 22 that
|
|
||||||
workflows install anyway.
|
|
||||||
- Gotcha for next time: `--tags gitea_runner_prune` restricts the run to *only* that tag, so
|
|
||||||
it cannot be combined with `-e gitea_runner_force_reregister=true`. Two separate runs.
|
|
||||||
- Gotcha 2: `gitea_admin` had `must_change_password=1` (set during the 1.27.3 container
|
|
||||||
recreate on 2026-09-14), which made Gitea reject **all** admin API calls with 403 — not
|
|
||||||
401, so credentials were fine. Cleared 2026-09-22 with
|
|
||||||
`docker exec -u git gitea gitea admin user must-change-password --unset gitea_admin`
|
|
||||||
(password unchanged; vault still valid). The admin API works again.
|
|
||||||
- [ ] Optional cleanup: prune the stale duplicate runner rows (59 of them now). Unblocked by the
|
|
||||||
admin-API fix above. `ansible-playbook -i inventory/hosts.ini gitea-deploy.yml --tags
|
|
||||||
gitea_runner_prune`. Purely cosmetic — deletes only runners the API reports offline.
|
|
||||||
|
|
||||||
**Note on the two fixes:** CI now has *two* independent reasons to work — the workflow no longer
|
|
||||||
uses apt at all, and the job image is no longer EOL (bookworm still serves `docker.io` fine). The
|
|
||||||
static-binary approach remains the more durable of the two; keep it regardless.
|
|
||||||
|
|
||||||
### 1.3 Developer-facing copy on the contact page ✅ FIXED 2026-09-23
|
|
||||||
Two notes written for developers, not visitors, were being rendered:
|
|
||||||
- the form note explaining `PUBLIC_CONTACT_FORM_ENDPOINT`, and
|
|
||||||
- a "Discovery note" about the WordPress page holding placeholder contact values.
|
|
||||||
|
|
||||||
Replaced the first with reader-facing copy and dropped the second (the deviation it described is
|
|
||||||
already recorded in `docs/content-deviations.md`). Commit `4e354d2`.
|
|
||||||
|
|
||||||
### 1.4 Contact form endpoint — plumbing DONE 2026-09-23, needs your Formspree ID 🛑
|
|
||||||
**Correction to the original entry:** it said to set `PUBLIC_CONTACT_FORM_ENDPOINT` in
|
|
||||||
`k8s/configmap.yaml`. That would never have worked.
|
|
||||||
- The Deployment has **no `env`/`envFrom`** referencing that ConfigMap — it is dead config.
|
|
||||||
(`PUBLIC_SITE_URL` is in the same state; it only appears to work because the default baked into
|
|
||||||
`astro.config.mjs` is already correct.)
|
|
||||||
- Even wired up it could not work: `output: "static"` means Astro **inlines** `PUBLIC_*` at image
|
|
||||||
build time, and the pod only runs nginx over pre-built HTML. There is no runtime knob.
|
|
||||||
|
|
||||||
**Done (commit `af4cc4a`):**
|
|
||||||
- [x] `Dockerfile` takes `ARG PUBLIC_CONTACT_FORM_ENDPOINT` / `PUBLIC_SITE_URL` → `ENV` before build.
|
|
||||||
- [x] CI passes `--build-arg` from a `CONTACT_FORM_ENDPOINT` Actions secret.
|
|
||||||
- [x] ConfigMap rewritten as documentation, with a warning not to add `envFrom`.
|
|
||||||
- [x] Submit posts in the background via `fetch()` and redirects to our own `/thank-you/` instead
|
|
||||||
of navigating to the provider's page.
|
|
||||||
- [x] Failure path: stays on the page, re-enables the button, shows the phone/email fallback.
|
|
||||||
- [x] Honeypot `_gotcha` (Formspree treats it as spam), positioned off-screen rather than
|
|
||||||
`display:none` so bots that skip hidden fields still fill it.
|
|
||||||
- [x] e2e covers the real fetch path (mocked endpoint, asserts the payload) and the failure case.
|
|
||||||
|
|
||||||
**Still needs you:**
|
|
||||||
- [ ] 🛑 Create the Formspree form for **info@bluecapstrategies.com** and confirm the address.
|
|
||||||
- [ ] 🛑 Add the endpoint as a Gitea repo Actions secret named **`CONTACT_FORM_ENDPOINT`**
|
|
||||||
(Settings → Actions → Secrets on `bmr_bluecap/bcs-website`), value
|
|
||||||
`https://formspree.io/f/<your-id>`.
|
|
||||||
- [ ] Push any commit touching `src/**` so CI rebuilds with the secret baked in.
|
|
||||||
- [ ] Verify on the staging URL: submit a real inquiry, confirm it arrives, confirm the browser
|
|
||||||
lands on `/thank-you/`.
|
|
||||||
- [ ] Confirm the destination inbox is actually monitored. A working form delivering into an
|
|
||||||
unwatched inbox is its own kind of failure.
|
|
||||||
|
|
||||||
**Degradation if the secret is missing:** the build still succeeds and the form falls back to
|
|
||||||
opening a `mailto:` draft. Survivable, but it fails silently for webmail users with no OS mail
|
|
||||||
client — so treat a missing secret as a launch blocker, not a warning.
|
|
||||||
|
|
||||||
### 1.5 Draft insight was being published and indexed ✅ FIXED 2026-09-23
|
|
||||||
`src/pages/insights/[slug].astro` called `getCollection("insights")` unfiltered, so the
|
|
||||||
`draft: true` placeholder built to `/insights/example-draft/` **and** appeared in
|
|
||||||
`sitemap-0.xml`. The schema declared the flag and `rss.xml.ts` already honoured it — only the page
|
|
||||||
route ignored it. Filtered in `getStaticPaths`, matching the RSS pattern. Build is 21 → 20 pages
|
|
||||||
and the URL is gone from the sitemap. Commit `4e354d2`.
|
|
||||||
|
|
||||||
- [ ] 🛑 **Still open (owner):** whether `/insights/` should ship at all with zero real articles.
|
|
||||||
It is currently in the primary nav. See also §2.5.
|
|
||||||
|
|
||||||
### 1.6 Canonical host is undecided 🛑 owner
|
|
||||||
The ingress answers on **both** apex and `www`; `astro.config.mjs`, `robots.txt`, and the
|
|
||||||
ConfigMap all hardcode `https://www.bluecapstrategies.com`. Serving identical content on two
|
|
||||||
hosts with no redirect splits SEO signal from day one.
|
|
||||||
- [ ] Owner picks apex or `www`.
|
|
||||||
- [ ] Configure a 301 from the loser to the winner (traefik middleware or edge).
|
|
||||||
- [ ] Reconcile `PUBLIC_SITE_URL`, canonical tags, `robots.txt` Sitemap line, and the ingress.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. Should-fix before cutover
|
## 2. Should fix
|
||||||
|
|
||||||
### 2.5 `/insights/` index still renders developer-facing copy
|
|
||||||
Found while fixing §1.5. `src/pages/insights/index.astro` shows visitors two cards written for a
|
|
||||||
developer, not a reader:
|
|
||||||
- *"Publishing path"* — "Future posts can be added as Markdown or MDX in
|
|
||||||
`src/content/insights` with typed frontmatter for title, description, publish date…"
|
|
||||||
- *"RSS support"* — "The feed is ready for future article imports even though no public posts are
|
|
||||||
published yet."
|
|
||||||
|
|
||||||
The hero also announces the section as a "Future publishing home", i.e. it tells visitors the
|
|
||||||
section is empty. Same defect class as §1.3, but left alone deliberately: what the page *should*
|
|
||||||
say depends on the owner's call in §1.5 on whether `/insights/` ships at all.
|
|
||||||
|
|
||||||
- [ ] If `/insights/` ships: delete both note-cards and rewrite the hero as reader-facing copy.
|
|
||||||
- [ ] If it does not: drop it from the primary nav and exclude the route from the sitemap.
|
|
||||||
|
|
||||||
### 2.7 cert-manager DNS-01 self-check uses CoreDNS
|
|
||||||
`cert-manager` logs `configured acme dns01 nameservers nameservers=["10.43.0.10:53"]`, i.e. it
|
|
||||||
validates DNS-01 propagation through the cluster's own CoreDNS rather than public resolvers. That
|
|
||||||
makes issuance sensitive to internal caching — including negative caching of the `_acme-challenge`
|
|
||||||
name it is waiting on.
|
|
||||||
|
|
||||||
- [ ] Point it at public resolvers before cutover:
|
|
||||||
`--dns01-recursive-nameservers-only=true --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53`.
|
|
||||||
Not confirmed as the cause of the 4h stall on 2026-10-09 (Cloudflare's authoritative NS were
|
|
||||||
themselves returning NXDOMAIN for records its API reported as present), but it is the
|
|
||||||
standard hardening and removes one variable from certificate renewal.
|
|
||||||
|
|
||||||
### 2.6 Staging URL ✅ LIVE 2026-09-23
|
|
||||||
**https://bcs.int.7tl-homelab.com** — review the rebuild in a browser, valid cert, no warning.
|
|
||||||
|
|
||||||
- LAN-only: `*.int.7tl-homelab.com` resolves to the internal traefik LB (10.66.15.30). Not
|
|
||||||
reachable from the internet; the real domain is untouched and WordPress still serves it.
|
|
||||||
- Cert issued by Let's Encrypt via cert-manager DNS-01 in ~75s, `CN=bcs.int.7tl-homelab.com`,
|
|
||||||
valid to 2026-12-21, auto-renewing.
|
|
||||||
- All 9 public routes verified 200.
|
|
||||||
- Commit `171c8c9`. Note it also corrected the issuer name in `k8s/ingress.yaml`: the commented
|
|
||||||
placeholder said `letsencrypt-production`, but the ClusterIssuer is **`letsencrypt-prod`** —
|
|
||||||
uncommenting it as written at cutover would have silently failed to issue.
|
|
||||||
- ArgoCD may take a few minutes to poll a push; force it with
|
|
||||||
`kubectl annotate application bcs-website -n argocd argocd.argoproj.io/refresh=hard --overwrite`.
|
|
||||||
|
|
||||||
### 2.1 e2e suite is red — stale assertions, not a regression
|
### 2.1 e2e suite is red — stale assertions, not a regression
|
||||||
4 of 46 tests fail. Both failures assert **invented copy that the exact-copy work correctly
|
4 of 48 fail (2 tests × desktop/mobile). Both assert **copy the exact-copy work correctly removed**:
|
||||||
removed**:
|
- `tests/site.spec.ts:63` waits for a link named `Explore services`; the homepage CTA is `All services`.
|
||||||
- `tests/site.spec.ts:67` waits for a link named `Explore services`; the homepage CTA is now
|
- `tests/site.spec.ts:93` waits for a heading matching `/Focused expertise/`, gone from `/focus-areas/`.
|
||||||
`All services` (matches live WP).
|
- [ ] Update both tests to current copy; get back to green so the suite is trustworthy.
|
||||||
- `tests/site.spec.ts:101` waits for a heading matching `/Focused expertise/`, which no longer
|
|
||||||
exists on `/focus-areas/`.
|
|
||||||
|
|
||||||
Navigation itself works. The tests are wrong, not the site.
|
|
||||||
- [ ] Update the two tests to assert current copy.
|
|
||||||
- [ ] Get back to 46/46 so the suite is trustworthy at cutover.
|
|
||||||
|
|
||||||
### 2.2 Repo hygiene
|
### 2.2 Repo hygiene
|
||||||
Commit `9f1cbca` committed ~50 `.playwright-mcp/` screenshots and YAML snapshots plus a 2.8 MB
|
Commit `9f1cbca` committed ~50 `.playwright-mcp/` screenshots/snapshots plus a 2.8 MB
|
||||||
`home-full.png` at the repo root.
|
`home-full.png` at the repo root.
|
||||||
- [ ] Add `.playwright-mcp/` to `.gitignore`; remove the tracked artifacts and `home-full.png`.
|
- [ ] Add `.playwright-mcp/` to `.gitignore`; remove the tracked artifacts and `home-full.png`.
|
||||||
|
|
||||||
### 2.3 Uptime monitoring is not deployed
|
### 2.3 DMARC tightening
|
||||||
Per ADR-0004 the plan is to reuse the homelab `monylog` stack (not Uptime Kuma). Pod metrics and
|
- [ ] Tighten DMARC from `p=none` to `p=quarantine` after a few weeks of clean reports
|
||||||
nginx logs already flow via Alloy; the external-availability piece was never added.
|
to info@bluecapstrategies.com.
|
||||||
- [?] Verify current state on monylog-1 — the items below are from notes, not re-verified today.
|
|
||||||
- [ ] Add `blackbox_exporter` + `web-bluecap.yml` alert rules; re-run the monylog ansible role to
|
|
||||||
reload Prometheus.
|
|
||||||
- [ ] Confirm an external HTTPS check fires to Pushover on a simulated outage. **Do this before
|
|
||||||
cutover** — the first hours on new infrastructure are when you most need paging.
|
|
||||||
|
|
||||||
### 2.4 Analytics is a no-op
|
|
||||||
`src/components/Analytics.astro` is wired into `BaseLayout` but gated on `PUBLIC_` env vars that
|
|
||||||
are absent from `k8s/configmap.yaml`, so nothing is collected.
|
|
||||||
- [ ] Decide whether to launch with analytics or add it after (ADR-0005: PostHog + Umami trial).
|
|
||||||
- [ ] **PostHog requires a privacy/consent notice before production use** — that gates it, not
|
|
||||||
the code.
|
|
||||||
- [ ] Set the env vars and confirm pageviews plus a consultation/CTA event.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 3. Owner decisions (Brad)
|
## 3. Owner decisions (Brad)
|
||||||
|
|
||||||
These are yours, not mine. Several block content that is otherwise ready.
|
- [x] **Canonical host** — `www` (2026-10-23). Apex 301s via Traefik middleware in `k8s/ingress.yaml`.
|
||||||
|
- [x] **Analytics** — Umami + GA4 side by side (2026-10-23).
|
||||||
- [ ] 🛑 **Canonical host** — apex vs `www` (also §1.6).
|
- [ ] 🛑 **Contact details** — confirm `804-332-6618` / `info@bluecapstrategies.com` are final.
|
||||||
- [ ] 🛑 **Contact details** — confirm `804-332-6618` / `info@bluecapstrategies.com` are the
|
- [ ] 🛑 **AI training-bot policy** — `GPTBot` and `ClaudeBot` are **Allow: /** in
|
||||||
production values. (The live WP contact page carries placeholders `123-456-8912` /
|
`public/robots.txt`, with a ready-to-uncomment `Disallow: /`. The site is now crawlable.
|
||||||
`info@gmail.com`, deliberately **not** reproduced; see `docs/content-deviations.md`.)
|
|
||||||
- [ ] 🛑 **AI training-bot policy** — `GPTBot` and `ClaudeBot` are currently **Allow: /** in
|
|
||||||
`public/robots.txt`, with a ready-to-uncomment `Disallow: /`. Search and answer crawlers
|
|
||||||
stay allowed either way. Decide before the site is publicly crawlable.
|
|
||||||
- [ ] 🛑 **Brad Rodgers bio + credentials + portrait** — gates `Person` schema, the About
|
- [ ] 🛑 **Brad Rodgers bio + credentials + portrait** — gates `Person` schema, the About
|
||||||
buildout, and principal-led E-E-A-T/GEO positioning. Core asset, still missing.
|
buildout, and principal-led E-E-A-T/GEO positioning.
|
||||||
- [ ] 🛑 **Real insights articles** — no fabricated or unreviewed AI content. Until one exists,
|
- [ ] 🛑 **Real insights articles** — no fabricated or unreviewed AI content (see §1.4).
|
||||||
`/insights/` is an empty section in the nav (see §1.5).
|
- [ ] 🛑 **Social profiles** — provide real profiles or omit the footer block.
|
||||||
- [ ] 🛑 **Social profiles** — the live site only has per-page share links and empty `#` hrefs.
|
- [ ] 🛑 **Proof modules / representative engagements** — no fabricated client work.
|
||||||
Provide real profiles or omit the footer block.
|
|
||||||
- [ ] 🛑 **`/test/` page** — the live site has a staging duplicate of the homepage. Default: omit.
|
|
||||||
- [ ] 🛑 **Proof modules / representative engagements** — no fabricated client work or dollar
|
|
||||||
figures. Provide real ones or ship without.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. Post-cutover (not blocking)
|
## 4. Post-launch improvements (not blocking)
|
||||||
|
|
||||||
- [ ] Google Search Console + Bing Webmaster verified; `sitemap-index.xml` submitted.
|
- [ ] Bing Webmaster Tools — import from Search Console (covers Bing, DuckDuckGo, ChatGPT search).
|
||||||
- [ ] Per-page 1200×630 OG images. Every page currently shares one fallback; `BaseLayout.astro:49`
|
- [ ] Check Search Console in ~1 week: sitemap "Success", pages indexed, no 5xx in Coverage.
|
||||||
emits a single `og:image`.
|
- [ ] Per-page 1200×630 OG images; `BaseLayout.astro` emits one shared fallback `og:image`.
|
||||||
- [ ] `robots` meta wired to a `noindex` frontmatter flag. `/thank-you/` is excluded from the
|
- [ ] `robots` meta wired to a `noindex` frontmatter flag. `/thank-you/` is excluded from the
|
||||||
sitemap but emits **no** `noindex` meta tag.
|
sitemap but emits no `noindex` meta tag.
|
||||||
- [ ] `article:published_time` / `article:modified_time` OG tags on insights posts (absent today).
|
- [ ] `article:published_time` / `article:modified_time` OG tags on insights posts.
|
||||||
- [ ] `lastmod` in the sitemap from content dates.
|
- [ ] `lastmod` in the sitemap from content dates.
|
||||||
- [ ] Lighthouse ≥ 95 on SEO / Perf / Best-Practices / Accessibility; watch CLS from unsized images.
|
- [ ] Lighthouse ≥ 95 on SEO / Perf / Best-Practices / Accessibility.
|
||||||
- [ ] Trailing-slash audit — `trailingSlash: "always"`; confirm canonicals and internal links agree.
|
- [ ] Umami/GA custom event for contact-form submits (today the `/thank-you/` pageview is the
|
||||||
- [ ] Decommission plan for the WordPress/Bluehost instance (keep it live and reachable through
|
conversion signal).
|
||||||
cutover + a rollback window; see the "keep old site live" constraint).
|
- [ ] Optional: move `bluecapstrategies.com` into Pangolin's `config.yml` `domains:` so it shows
|
||||||
|
"Config Managed" like the other three (needs a Pangolin restart; purely cosmetic).
|
||||||
|
- [ ] Deferred: VPS static failover for homelab outages (see memory note / earlier discussion).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5. Already done — do not redo
|
## 5. Launch-day record (2026-10-23)
|
||||||
|
|
||||||
Recorded because the older checklists still list several of these as open.
|
What happened, so it does not have to be rediscovered.
|
||||||
|
|
||||||
- [x] Full copy parity: home, about, services, focus areas, all 8 service detail pages, all 4
|
- **Pangolin certs need a real config change to retry.** HTTP-01 failed while `www` was still
|
||||||
focus-area detail pages — verbatim live copy in Markdown content collections (ADR-0002).
|
proxied (Let's Encrypt reached Cloudflare's IPv6, got 404) and hit the 5-failures/hour limit.
|
||||||
- [x] Real imagery imported to `public/assets/live-site/` (+ `/icons`); placeholders deleted.
|
Clicking Save with no changes does nothing; toggling **Enable Resource** off/on (saving between)
|
||||||
- [x] Zod `seoSchema` in `src/content.config.ts` — build **fails** on title/description/OG
|
re-adds the router and triggers issuance.
|
||||||
violations.
|
- **Pangolin health checks** probe the target IP with no hostname, which k3s traefik answers with
|
||||||
- [x] `scripts/seo-lint.mjs` + `npm run seo:lint` — single-h1, heading order, canonical, alt text,
|
404. Set the health check's Host header and TLS server name to the resource's hostname, path
|
||||||
JSON-LD parse. Green on 21 pages.
|
`/` (or `/api/heartbeat` for Umami), expect 200.
|
||||||
- [x] `BreadcrumbList` schema (`src/utils/schema.ts`) applied across detail pages — the SEO doc
|
- **nginx 404 loop** — `error_page` pointed at `/404/index.html` but Astro emits `404.html`, so
|
||||||
still lists this as to-do; it is done.
|
every unknown URL returned 500. Fixed in `d1c1726`; unknown URLs now return a real 404.
|
||||||
|
- **CI tag-bump push rejected** when `main` moved during a build (or on a re-run of an older
|
||||||
|
commit). The step now rebases onto `origin/main` and retries (`93415c6`). Do not "re-run" old
|
||||||
|
runs to deploy — push instead.
|
||||||
|
- **Cloudflare cleanup** — removed all Bluehost/cPanel leftovers (autoconfig/autodiscover,
|
||||||
|
cpanel/whm/webdisk/ftp, cal/carddav SRV, `localhost`), replaced Bluehost's `default._domainkey`
|
||||||
|
with MXroute's `x._domainkey`, added `_dmarc`.
|
||||||
|
- **Monitoring safety** — the monylog playbook only renders Pushover routing when `VAULT_TOKEN`
|
||||||
|
is set; without it Alertmanager silently falls back to ntfy. Always run it with the token.
|
||||||
|
- **Umami** ships with `admin` / `umami`. It was brought up LAN-only
|
||||||
|
(`umami.int.7tl-homelab.com`), the password changed, and only then exposed via Pangolin.
|
||||||
|
Manifests live in `homelab-gitops` (`apps/umami.yaml`, `analytics/umami/`); credentials in
|
||||||
|
OpenBao `cluster/umami`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Already done — do not redo
|
||||||
|
|
||||||
|
- [x] Full copy parity with the WordPress site, verbatim in content collections (ADR-0002).
|
||||||
|
- [x] Real imagery in `public/assets/live-site/`; placeholders deleted.
|
||||||
|
- [x] Zod `seoSchema`; `scripts/seo-lint.mjs` (`npm run seo:lint`).
|
||||||
|
- [x] `BreadcrumbList` schema across detail pages.
|
||||||
- [x] Per-agent `public/robots.txt`; `@astrojs/sitemap` excluding `/thank-you/`.
|
- [x] Per-agent `public/robots.txt`; `@astrojs/sitemap` excluding `/thank-you/`.
|
||||||
- [x] Deviations logged in `docs/content-deviations.md`.
|
- [x] Draft insights excluded from routes and sitemap.
|
||||||
- [x] GitOps: Gitea Actions CI, Gitea Packages registry, ArgoCD Application, External-Secrets
|
- [x] Contact form: build-time Formspree endpoint, background submit, honeypot, failure fallback.
|
||||||
registry pull auth over public HTTPS (ADR-0007, ADR-0008). Proven end-to-end in July —
|
- [x] GitOps: Gitea Actions CI → Gitea Packages → tag-bump commit → ArgoCD (ADR-0007, ADR-0008).
|
||||||
currently stalled, see §1.2.
|
- [x] CI job image `node:22-bookworm`; docker CLI + kustomize as pinned static binaries.
|
||||||
- [x] Monitoring approach decided: reuse `monylog`, not Uptime Kuma (ADR-0004). The SEO doc's
|
- [x] Production TLS in-cluster (DNS-01) and at the edge (HTTP-01).
|
||||||
"Uptime Kuma" line is superseded.
|
- [x] Canonical www + apex 301; real 404s.
|
||||||
|
- [x] Monitoring: blackbox probes + `WebsiteDown` / `WebsiteCertExpiringSoon` / `WebsiteSlow` /
|
||||||
|
`BlackboxExporterDown` rules in the monylog role (ADR-0004: monylog, not Uptime Kuma).
|
||||||
|
- [x] Analytics: Umami + GA4, production-host only; `/privacy/` page.
|
||||||
|
- [x] Search Console verified; sitemap submitted.
|
||||||
|
- [x] Mail authentication: SPF, DKIM, DMARC passing.
|
||||||
Reference in new issue
Block a user