0 Fork 0

Files

T

bmr_bluecap 5c825b08e9 docs: stats internal cert issued after DNS cache expiry

2026-10-23 14:32:54 -04:00

11 KiB

Raw Blame History

Launch Checklist — Post-Cutover Status

www.bluecapstrategies.com went live on the Astro rebuild on 2026-10-23. This file is now the single source of truth for what is running, how it is wired, and what is left. The pre-launch runbook it replaced is in git history (git log -- docs/launch-checklist.md, before 2026-10-23).

Status legend: [ ] open · [x] done · [?] needs verification · 🛑 owner decision

0. As-built state (2026-10-23)

Thing	State
Public site	https://www.bluecapstrategies.com — canonical. Apex 301s to www, path preserved
Old WordPress / Bluehost	Gone. Access lost and it was returning 404 before cutover; nothing to roll back to
DNS	Cloudflare, DNS-only (grey cloud) for apex, `www`, `stats` → `23.94.111.133`
Edge	Pangolin VPS 23.94.111.133 (Traefik + CrowdSec), org `rhh-homelab`, site `7tl_lab`
Pangolin resources	`Bluecap-www`, `Bluecap-base`, `stats` — 3 https targets `10.66.15.30/.31/.32:443` each
Edge certs	Let's Encrypt HTTP-01 via Pangolin's `letsencrypt` resolver, issued 2026-10-23
Cluster certs	cert-manager `letsencrypt-prod` DNS-01: `bluecap-website-tls` (www + apex), `bcs-int-tls`
ArgoCD	`bcs-website` and `umami` — Synced / Healthy
Deployed image	tracks `main` via CI tag-bump (`ci: deploy <sha>` commits)
Staging	https://bcs.int.7tl-homelab.com — LAN only
Contact form	Formspree, verified end to end 2026-10-09
Analytics	Umami (self-hosted, `stats.bluecapstrategies.com`) + GA4 `G-Z273QL1F0N`, both verified recording 2026-10-23
Monitoring	monylog blackbox probes on www + apex → Alertmanager → Pushover
Search	Google Search Console domain property verified; sitemap submitted and fetched by Googlebot
Mail	MXroute. SPF pass, DKIM pass (selector `x`), DMARC `p=none` — mail-tester 9.5/10
`npm run build` / `typecheck` / `seo:lint`	green, 21 pages
`npm run test:e2e`	44 passed / 4 failed — stale assertions, see §2.1

Traffic path

visitor → Cloudflare DNS (DNS-only) → Pangolin VPS 23.94.111.133 (TLS #1, CrowdSec)
        → gerbil/newt tunnel → k3s traefik 10.66.15.30/.31/.32 :443 (TLS #2)
        → bluecap-website (nginx, 2 replicas)        ns bluecap-strategies
        → umami (+ umami-postgres)                   ns umami

1. Open — do soon

1.1 External uptime check 🛑

monylog runs inside the homelab, so a homelab power or internet outage takes the site and its monitor down together, and nothing pages. Needs a vantage point outside the homelab.

Decide: hosted (UptimeRobot / Healthchecks.io) or Uptime Kuma on the Pangolin VPS.
Probe https://www.bluecapstrategies.com/ and route alerts to Pushover.

1.2 Homelab intercepts port-53 DNS

Found 2026-10-23. Queries from the homelab — including k3s nodes — to any DNS server, even Cloudflare's authoritative NS, are answered by a local recursive resolver (dig @arely.ns.cloudflare.com returns rd ra with no aa). A name looked up before its record exists gets NXDOMAIN cached for 30 minutes (the zone's SOA negative TTL).

This, not Cloudflare, explains the 4-hour cert-manager DNS-01 stall of 2026-10-09 and the stats NXDOMAIN at launch. Consequences:

Pointing cert-manager at 1.1.1.1/8.8.8.8 (the fix previously suggested here) will not help while port 53 is intercepted.
To see true public DNS, query from the VPS or via DoH: curl -H 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=<name>&type=A'
Create DNS records before anything in the homelab (cert-manager included) looks the name up.
🛑 Owner decision: leave as is, or exempt the k3s nodes from the redirect. The redirect rule is not in ~/Homelab/mikrotik/*.rsc; find it on the router (or Pi-hole/AdGuard, if any).
umami-stats-tls (internal hop for stats) stalled on the stale NXDOMAIN at launch and issued by itself once the cache expired (~18:35 UTC, 2026-10-23) — the expected pattern.

1.3 Gitea runner loses Docker after a dockerd restart

dockerd on gitea-1 runs with live-restore=true. When it restarts, containers keep running but act_runner's bind mount of /var/run/docker.sock points at the dead socket, so every CI job fails in its first second with Cannot connect to the Docker daemon. Happened 2026-10-21 → discovered 2026-10-23.

Workaround: sudo docker restart act_runner on gitea-1.
Durable fix in the gitea ansible role: disable live-restore on that VM, or restart the runner after dockerd restarts (systemd PartOf=/ExecStartPost hook).

1.4 `/insights/` ships empty, with developer-facing copy 🛑

src/pages/insights/index.astro shows two cards written for a developer ("Publishing path", "RSS support") and a hero calling the section a "Future publishing home". The section is in the primary nav with zero articles.

🛑 Ship it or not.
If yes: delete both note-cards and rewrite the hero as reader-facing copy.
If no: drop it from the primary nav and exclude the route from the sitemap.

1.5 Privacy policy review 🛑

/privacy/ went live 2026-10-23, drafted from what the site actually does (contact form via Formspree, Umami, GA4 with cookies, server/security logs).

🛑 Owner review of the text. It is a legal document.
🛑 Consent banner: GA4 sets cookies. Strictly, EU/UK visitors need opt-in consent; most US-focused small-business sites run with a privacy policy only. Owner's call. Umami needs neither.
Anything added to src/components/Analytics.astro must be disclosed on /privacy/.

2. Should fix

2.1 e2e suite is red — stale assertions, not a regression

4 of 48 fail (2 tests × desktop/mobile). Both assert copy the exact-copy work correctly removed:

tests/site.spec.ts:63 waits for a link named Explore services; the homepage CTA is All services.
tests/site.spec.ts:93 waits for a heading matching /Focused expertise/, gone from /focus-areas/.
Update both tests to current copy; get back to green so the suite is trustworthy.

2.2 Repo hygiene

Commit 9f1cbca committed ~50 .playwright-mcp/ screenshots/snapshots plus a 2.8 MB home-full.png at the repo root.

Add .playwright-mcp/ to .gitignore; remove the tracked artifacts and home-full.png.

2.3 DMARC tightening

Tighten DMARC from p=none to p=quarantine after a few weeks of clean reports to info@bluecapstrategies.com.

3. Owner decisions (Brad)

Canonical host — www (2026-10-23). Apex 301s via Traefik middleware in k8s/ingress.yaml.
Analytics — Umami + GA4 side by side (2026-10-23).
🛑 Contact details — confirm 804-332-6618 / info@bluecapstrategies.com are final.
🛑 AI training-bot policy — GPTBot and ClaudeBot are Allow: / in public/robots.txt, with a ready-to-uncomment Disallow: /. The site is now crawlable.
🛑 Brad Rodgers bio + credentials + portrait — gates Person schema, the About buildout, and principal-led E-E-A-T/GEO positioning.
🛑 Real insights articles — no fabricated or unreviewed AI content (see §1.4).
🛑 Social profiles — provide real profiles or omit the footer block.
🛑 Proof modules / representative engagements — no fabricated client work.

4. Post-launch improvements (not blocking)

Bing Webmaster Tools — import from Search Console (covers Bing, DuckDuckGo, ChatGPT search).
Check Search Console in ~1 week: sitemap "Success", pages indexed, no 5xx in Coverage.
Per-page 1200×630 OG images; BaseLayout.astro emits one shared fallback og:image.
robots meta wired to a noindex frontmatter flag. /thank-you/ is excluded from the sitemap but emits no noindex meta tag.
article:published_time / article:modified_time OG tags on insights posts.
lastmod in the sitemap from content dates.
Lighthouse ≥ 95 on SEO / Perf / Best-Practices / Accessibility.
Umami/GA custom event for contact-form submits (today the /thank-you/ pageview is the conversion signal).
Optional: move bluecapstrategies.com into Pangolin's config.yml domains: so it shows "Config Managed" like the other three (needs a Pangolin restart; purely cosmetic).
Deferred: VPS static failover for homelab outages (see memory note / earlier discussion).

5. Launch-day record (2026-10-23)

What happened, so it does not have to be rediscovered.

Pangolin certs need a real config change to retry. HTTP-01 failed while www was still proxied (Let's Encrypt reached Cloudflare's IPv6, got 404) and hit the 5-failures/hour limit. Clicking Save with no changes does nothing; toggling Enable Resource off/on (saving between) re-adds the router and triggers issuance.
Pangolin health checks probe the target IP with no hostname, which k3s traefik answers with 404. Set the health check's Host header and TLS server name to the resource's hostname, path / (or /api/heartbeat for Umami), expect 200.
nginx 404 loop — error_page pointed at /404/index.html but Astro emits 404.html, so every unknown URL returned 500. Fixed in d1c1726; unknown URLs now return a real 404.
CI tag-bump push rejected when main moved during a build (or on a re-run of an older commit). The step now rebases onto origin/main and retries (93415c6). Do not "re-run" old runs to deploy — push instead.
Cloudflare cleanup — removed all Bluehost/cPanel leftovers (autoconfig/autodiscover, cpanel/whm/webdisk/ftp, cal/carddav SRV, localhost), replaced Bluehost's default._domainkey with MXroute's x._domainkey, added _dmarc.
Monitoring safety — the monylog playbook only renders Pushover routing when VAULT_TOKEN is set; without it Alertmanager silently falls back to ntfy. Always run it with the token.
Umami ships with admin / umami. It was brought up LAN-only (umami.int.7tl-homelab.com), the password changed, and only then exposed via Pangolin. Manifests live in homelab-gitops (apps/umami.yaml, analytics/umami/); credentials in OpenBao cluster/umami.

6. Already done — do not redo

Full copy parity with the WordPress site, verbatim in content collections (ADR-0002).
Real imagery in public/assets/live-site/; placeholders deleted.
Zod seoSchema; scripts/seo-lint.mjs (npm run seo:lint).
BreadcrumbList schema across detail pages.
Per-agent public/robots.txt; @astrojs/sitemap excluding /thank-you/.
Draft insights excluded from routes and sitemap.
Contact form: build-time Formspree endpoint, background submit, honeypot, failure fallback.
GitOps: Gitea Actions CI → Gitea Packages → tag-bump commit → ArgoCD (ADR-0007, ADR-0008).
CI job image node:22-bookworm; docker CLI + kustomize as pinned static binaries.
Production TLS in-cluster (DNS-01) and at the edge (HTTP-01).
Canonical www + apex 301; real 404s.
Monitoring: blackbox probes + WebsiteDown / WebsiteCertExpiringSoon / WebsiteSlow / BlackboxExporterDown rules in the monylog role (ADR-0004: monylog, not Uptime Kuma).
Analytics: Umami + GA4, production-host only; /privacy/ page.
Search Console verified; sitemap submitted.
Mail authentication: SPF, DKIM, DMARC passing.

11 KiB Raw Blame History Unescape Escape