bmr_bluecap
450bba8bea
Records the live architecture, launch-day findings (Pangolin cert retry, health-check Host header, nginx 404 loop, CI rebase, homelab DNS interception, runner socket) and the remaining open items.
197 lines
11 KiB
Markdown
197 lines
11 KiB
Markdown
# Launch Checklist — Post-Cutover Status
|
||
|
||
`www.bluecapstrategies.com` went **live on the Astro rebuild on 2026-10-23**. This file is now the
|
||
single source of truth for what is running, how it is wired, and what is left. The pre-launch
|
||
runbook it replaced is in git history (`git log -- docs/launch-checklist.md`, before 2026-10-23).
|
||
|
||
Status legend: `[ ]` open · `[x]` done · `[?]` needs verification · 🛑 owner decision
|
||
|
||
---
|
||
|
||
## 0. As-built state (2026-10-23)
|
||
|
||
| Thing | State |
|
||
|-------|-------|
|
||
| Public site | **https://www.bluecapstrategies.com** — canonical. Apex 301s to www, path preserved |
|
||
| Old WordPress / Bluehost | **Gone.** Access lost and it was returning 404 before cutover; nothing to roll back to |
|
||
| DNS | Cloudflare, **DNS-only (grey cloud)** for apex, `www`, `stats` → `23.94.111.133` |
|
||
| Edge | Pangolin VPS 23.94.111.133 (Traefik + CrowdSec), org `rhh-homelab`, site `7tl_lab` |
|
||
| Pangolin resources | `Bluecap-www`, `Bluecap-base`, `stats` — 3 https targets `10.66.15.30/.31/.32:443` each |
|
||
| Edge certs | Let's Encrypt HTTP-01 via Pangolin's `letsencrypt` resolver, issued 2026-10-23 |
|
||
| Cluster certs | cert-manager `letsencrypt-prod` DNS-01: `bluecap-website-tls` (www + apex), `bcs-int-tls` |
|
||
| ArgoCD | `bcs-website` and `umami` — Synced / Healthy |
|
||
| Deployed image | tracks `main` via CI tag-bump (`ci: deploy <sha>` commits) |
|
||
| Staging | https://bcs.int.7tl-homelab.com — LAN only |
|
||
| Contact form | Formspree, verified end to end 2026-10-09 |
|
||
| Analytics | **Umami** (self-hosted, `stats.bluecapstrategies.com`) + **GA4** `G-Z273QL1F0N`, both verified recording 2026-10-23 |
|
||
| Monitoring | monylog blackbox probes on www + apex → Alertmanager → **Pushover** |
|
||
| Search | Google Search Console domain property verified; sitemap submitted and fetched by Googlebot |
|
||
| Mail | MXroute. SPF pass, DKIM pass (selector `x`), DMARC `p=none` — mail-tester 9.5/10 |
|
||
| `npm run build` / `typecheck` / `seo:lint` | green, 21 pages |
|
||
| `npm run test:e2e` | 44 passed / 4 failed — stale assertions, see §2.1 |
|
||
|
||
### Traffic path
|
||
```
|
||
visitor → Cloudflare DNS (DNS-only) → Pangolin VPS 23.94.111.133 (TLS #1, CrowdSec)
|
||
→ gerbil/newt tunnel → k3s traefik 10.66.15.30/.31/.32 :443 (TLS #2)
|
||
→ bluecap-website (nginx, 2 replicas) ns bluecap-strategies
|
||
→ umami (+ umami-postgres) ns umami
|
||
```
|
||
|
||
---
|
||
|
||
## 1. Open — do soon
|
||
|
||
### 1.1 External uptime check 🛑
|
||
monylog runs inside the homelab, so a homelab power or internet outage takes the site **and** its
|
||
monitor down together, and nothing pages. Needs a vantage point outside the homelab.
|
||
- [ ] Decide: hosted (UptimeRobot / Healthchecks.io) or Uptime Kuma on the Pangolin VPS.
|
||
- [ ] Probe `https://www.bluecapstrategies.com/` and route alerts to Pushover.
|
||
|
||
### 1.2 Homelab intercepts port-53 DNS
|
||
Found 2026-10-23. Queries from the homelab — including k3s nodes — to *any* DNS server, even
|
||
Cloudflare's authoritative NS, are answered by a local recursive resolver (`dig @arely.ns.cloudflare.com`
|
||
returns `rd ra` with no `aa`). A name looked up before its record exists gets **NXDOMAIN cached for
|
||
30 minutes** (the zone's SOA negative TTL).
|
||
|
||
This, not Cloudflare, explains the 4-hour cert-manager DNS-01 stall of 2026-10-09 and the
|
||
`stats` NXDOMAIN at launch. Consequences:
|
||
- Pointing cert-manager at `1.1.1.1/8.8.8.8` (the fix previously suggested here) **will not help**
|
||
while port 53 is intercepted.
|
||
- To see true public DNS, query from the VPS or via DoH:
|
||
`curl -H 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=<name>&type=A'`
|
||
- Create DNS records *before* anything in the homelab (cert-manager included) looks the name up.
|
||
|
||
- [ ] 🛑 Owner decision: leave as is, or exempt the k3s nodes from the redirect. The redirect rule
|
||
is not in `~/Homelab/mikrotik/*.rsc`; find it on the router (or Pi-hole/AdGuard, if any).
|
||
- [?] `umami-stats-tls` (internal hop for `stats`) was still pending on the stale NXDOMAIN at
|
||
18:30 UTC on launch day; it should issue by itself once the cache expires. Visitor-facing
|
||
TLS for `stats` is Pangolin's and is fine. Check with `kubectl -n umami get certificate`.
|
||
|
||
### 1.3 Gitea runner loses Docker after a dockerd restart
|
||
`dockerd` on `gitea-1` runs with `live-restore=true`. When it restarts, containers keep running
|
||
but `act_runner`'s bind mount of `/var/run/docker.sock` points at the dead socket, so every CI
|
||
job fails in its first second with `Cannot connect to the Docker daemon`. Happened 2026-10-21 →
|
||
discovered 2026-10-23.
|
||
- **Workaround:** `sudo docker restart act_runner` on gitea-1.
|
||
- [ ] Durable fix in the `gitea` ansible role: disable live-restore on that VM, or restart the
|
||
runner after dockerd restarts (systemd `PartOf=`/`ExecStartPost` hook).
|
||
|
||
### 1.4 `/insights/` ships empty, with developer-facing copy 🛑
|
||
`src/pages/insights/index.astro` shows two cards written for a developer ("Publishing path",
|
||
"RSS support") and a hero calling the section a "Future publishing home". The section is in the
|
||
primary nav with zero articles.
|
||
- [ ] 🛑 Ship it or not.
|
||
- [ ] If yes: delete both note-cards and rewrite the hero as reader-facing copy.
|
||
- [ ] If no: drop it from the primary nav and exclude the route from the sitemap.
|
||
|
||
### 1.5 Privacy policy review 🛑
|
||
`/privacy/` went live 2026-10-23, drafted from what the site actually does (contact form via
|
||
Formspree, Umami, GA4 with cookies, server/security logs).
|
||
- [ ] 🛑 Owner review of the text. It is a legal document.
|
||
- [ ] 🛑 Consent banner: GA4 sets cookies. Strictly, EU/UK visitors need opt-in consent; most
|
||
US-focused small-business sites run with a privacy policy only. Owner's call. Umami needs
|
||
neither.
|
||
- Anything added to `src/components/Analytics.astro` must be disclosed on `/privacy/`.
|
||
|
||
---
|
||
|
||
## 2. Should fix
|
||
|
||
### 2.1 e2e suite is red — stale assertions, not a regression
|
||
4 of 48 fail (2 tests × desktop/mobile). Both assert **copy the exact-copy work correctly removed**:
|
||
- `tests/site.spec.ts:63` waits for a link named `Explore services`; the homepage CTA is `All services`.
|
||
- `tests/site.spec.ts:93` waits for a heading matching `/Focused expertise/`, gone from `/focus-areas/`.
|
||
- [ ] Update both tests to current copy; get back to green so the suite is trustworthy.
|
||
|
||
### 2.2 Repo hygiene
|
||
Commit `9f1cbca` committed ~50 `.playwright-mcp/` screenshots/snapshots plus a 2.8 MB
|
||
`home-full.png` at the repo root.
|
||
- [ ] Add `.playwright-mcp/` to `.gitignore`; remove the tracked artifacts and `home-full.png`.
|
||
|
||
### 2.3 DMARC tightening
|
||
- [ ] Tighten DMARC from `p=none` to `p=quarantine` after a few weeks of clean reports
|
||
to info@bluecapstrategies.com.
|
||
|
||
---
|
||
|
||
## 3. Owner decisions (Brad)
|
||
|
||
- [x] **Canonical host** — `www` (2026-10-23). Apex 301s via Traefik middleware in `k8s/ingress.yaml`.
|
||
- [x] **Analytics** — Umami + GA4 side by side (2026-10-23).
|
||
- [ ] 🛑 **Contact details** — confirm `804-332-6618` / `info@bluecapstrategies.com` are final.
|
||
- [ ] 🛑 **AI training-bot policy** — `GPTBot` and `ClaudeBot` are **Allow: /** in
|
||
`public/robots.txt`, with a ready-to-uncomment `Disallow: /`. The site is now crawlable.
|
||
- [ ] 🛑 **Brad Rodgers bio + credentials + portrait** — gates `Person` schema, the About
|
||
buildout, and principal-led E-E-A-T/GEO positioning.
|
||
- [ ] 🛑 **Real insights articles** — no fabricated or unreviewed AI content (see §1.4).
|
||
- [ ] 🛑 **Social profiles** — provide real profiles or omit the footer block.
|
||
- [ ] 🛑 **Proof modules / representative engagements** — no fabricated client work.
|
||
|
||
---
|
||
|
||
## 4. Post-launch improvements (not blocking)
|
||
|
||
- [ ] Bing Webmaster Tools — import from Search Console (covers Bing, DuckDuckGo, ChatGPT search).
|
||
- [ ] Check Search Console in ~1 week: sitemap "Success", pages indexed, no 5xx in Coverage.
|
||
- [ ] Per-page 1200×630 OG images; `BaseLayout.astro` emits one shared fallback `og:image`.
|
||
- [ ] `robots` meta wired to a `noindex` frontmatter flag. `/thank-you/` is excluded from the
|
||
sitemap but emits no `noindex` meta tag.
|
||
- [ ] `article:published_time` / `article:modified_time` OG tags on insights posts.
|
||
- [ ] `lastmod` in the sitemap from content dates.
|
||
- [ ] Lighthouse ≥ 95 on SEO / Perf / Best-Practices / Accessibility.
|
||
- [ ] Umami/GA custom event for contact-form submits (today the `/thank-you/` pageview is the
|
||
conversion signal).
|
||
- [ ] Optional: move `bluecapstrategies.com` into Pangolin's `config.yml` `domains:` so it shows
|
||
"Config Managed" like the other three (needs a Pangolin restart; purely cosmetic).
|
||
- [ ] Deferred: VPS static failover for homelab outages (see memory note / earlier discussion).
|
||
|
||
---
|
||
|
||
## 5. Launch-day record (2026-10-23)
|
||
|
||
What happened, so it does not have to be rediscovered.
|
||
|
||
- **Pangolin certs need a real config change to retry.** HTTP-01 failed while `www` was still
|
||
proxied (Let's Encrypt reached Cloudflare's IPv6, got 404) and hit the 5-failures/hour limit.
|
||
Clicking Save with no changes does nothing; toggling **Enable Resource** off/on (saving between)
|
||
re-adds the router and triggers issuance.
|
||
- **Pangolin health checks** probe the target IP with no hostname, which k3s traefik answers with
|
||
404. Set the health check's Host header and TLS server name to the resource's hostname, path
|
||
`/` (or `/api/heartbeat` for Umami), expect 200.
|
||
- **nginx 404 loop** — `error_page` pointed at `/404/index.html` but Astro emits `404.html`, so
|
||
every unknown URL returned 500. Fixed in `d1c1726`; unknown URLs now return a real 404.
|
||
- **CI tag-bump push rejected** when `main` moved during a build (or on a re-run of an older
|
||
commit). The step now rebases onto `origin/main` and retries (`93415c6`). Do not "re-run" old
|
||
runs to deploy — push instead.
|
||
- **Cloudflare cleanup** — removed all Bluehost/cPanel leftovers (autoconfig/autodiscover,
|
||
cpanel/whm/webdisk/ftp, cal/carddav SRV, `localhost`), replaced Bluehost's `default._domainkey`
|
||
with MXroute's `x._domainkey`, added `_dmarc`.
|
||
- **Monitoring safety** — the monylog playbook only renders Pushover routing when `VAULT_TOKEN`
|
||
is set; without it Alertmanager silently falls back to ntfy. Always run it with the token.
|
||
- **Umami** ships with `admin` / `umami`. It was brought up LAN-only
|
||
(`umami.int.7tl-homelab.com`), the password changed, and only then exposed via Pangolin.
|
||
Manifests live in `homelab-gitops` (`apps/umami.yaml`, `analytics/umami/`); credentials in
|
||
OpenBao `cluster/umami`.
|
||
|
||
---
|
||
|
||
## 6. Already done — do not redo
|
||
|
||
- [x] Full copy parity with the WordPress site, verbatim in content collections (ADR-0002).
|
||
- [x] Real imagery in `public/assets/live-site/`; placeholders deleted.
|
||
- [x] Zod `seoSchema`; `scripts/seo-lint.mjs` (`npm run seo:lint`).
|
||
- [x] `BreadcrumbList` schema across detail pages.
|
||
- [x] Per-agent `public/robots.txt`; `@astrojs/sitemap` excluding `/thank-you/`.
|
||
- [x] Draft insights excluded from routes and sitemap.
|
||
- [x] Contact form: build-time Formspree endpoint, background submit, honeypot, failure fallback.
|
||
- [x] GitOps: Gitea Actions CI → Gitea Packages → tag-bump commit → ArgoCD (ADR-0007, ADR-0008).
|
||
- [x] CI job image `node:22-bookworm`; docker CLI + kustomize as pinned static binaries.
|
||
- [x] Production TLS in-cluster (DNS-01) and at the edge (HTTP-01).
|
||
- [x] Canonical www + apex 301; real 404s.
|
||
- [x] Monitoring: blackbox probes + `WebsiteDown` / `WebsiteCertExpiringSoon` / `WebsiteSlow` /
|
||
`BlackboxExporterDown` rules in the monylog role (ADR-0004: monylog, not Uptime Kuma).
|
||
- [x] Analytics: Umami + GA4, production-host only; `/privacy/` page.
|
||
- [x] Search Console verified; sitemap submitted.
|
||
- [x] Mail authentication: SPF, DKIM, DMARC passing.
|