Brad Rodgers
cd91c4b97c
Static Astro 5 site for BlueCap Strategies with exact live-site content parity, deployed to k3s via GitOps (ArgoCD). - Page copy in Markdown content collections (services, focus-areas, pages, insights) with a Zod SEO schema enforced at build time - Homepage + About restored to exact live copy; real live-site imagery - Build-time SEO linter (scripts/seo-lint.mjs) and Playwright e2e suite - Multi-stage Dockerfile (nginx serves dist/) and Kustomize manifests (k8s/) - Per-agent robots.txt; config-driven PostHog + Umami analytics scaffold - ADRs and engineering docs under docs/ Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
238 lines
15 KiB
Markdown
238 lines
15 KiB
Markdown
# SEO, AEO & AI-Crawler Optimization Plan
|
||
|
||
Requirements and rules for optimizing the BlueCap Strategies rebuild for classic search (SEO),
|
||
answer engines / AI citation (AEO/GEO), and AI-crawler acceptance. Written **2026-07-15** against
|
||
mid-2026 best practices (see Sources). Where a rule can be machine-enforced, it is defined as a
|
||
**Zod schema constraint** or a **build-time check** so the build fails when copy violates it —
|
||
this is the TypeScript/Astro equivalent of Pydantic validation (do **not** add Python/Pydantic to
|
||
a JS build; Astro Content Collections already run on Zod).
|
||
|
||
## 0. Guiding principles (2026 reality check)
|
||
|
||
1. **HTML is the source of truth, not `llms.txt`.** AI search crawlers (GPTBot, ClaudeBot,
|
||
PerplexityBot, OAI-SearchBot, Google-Extended) overwhelmingly **skip `/llms.txt` and crawl
|
||
rendered HTML directly**; Google confirmed it does not support llms.txt. Keep our `llms.txt`
|
||
(low cost, ~10% of sites have one) but invest first in semantic HTML, structured data, and
|
||
crawlable copy. Our Astro output is **fully static** — 100% of meaningful copy is in initial
|
||
HTML, which is the single biggest AEO advantage over the old JS-heavy WordPress/Elementor site.
|
||
2. **Structured data drives citation.** Schema lets answer engines verify *who we are*, *what a
|
||
page covers*, and *which facts are reliable to cite*. FAQ schema has among the highest AI
|
||
citation rates (question/answer mirrors how engines present answers).
|
||
3. **Answer-ready structure wins.** Clear headings, direct definitional first sentences, and
|
||
self-contained paragraphs get extracted; buried or paraphrased marketing prose does not.
|
||
4. **E-E-A-T via the principal.** Boutique-advisory credibility is entity-driven — a real
|
||
`Person` entity (Brad Rodgers) with `knowsAbout`, linked to the `Organization`, is a core GEO
|
||
asset. (Gated on owner-approved bio — see parity plan Phase 5.)
|
||
|
||
## 1. Programmatic enforcement (Zod + build linter)
|
||
|
||
### 1a. Content-collection SEO schema (fails the build)
|
||
|
||
Extend `src/content.config.ts`. Every collection (pages, services, focusAreas, insights) shares
|
||
an `seo` shape with enforced limits:
|
||
|
||
```ts
|
||
import { z } from "astro:content";
|
||
|
||
export const seo = z.object({
|
||
// <title> renders as "`title` | BlueCap Strategies" — keep the full SERP title ≤ ~60 chars
|
||
title: z.string().min(10).max(55),
|
||
// meta description + OG description; 120–158 is the safe SERP/AI-snippet window
|
||
description: z.string().min(120).max(158),
|
||
// canonical is derived from slug, but allow override for cross-posted insights
|
||
canonicalUrl: z.string().url().optional(),
|
||
// every page must ship a social share image (absolute path under /assets)
|
||
ogImage: z.string().startsWith("/assets/").default("/assets/og/default.jpg"),
|
||
// primary entity/topic — powers internal linking + schema keywords
|
||
primaryTopic: z.string(),
|
||
noindex: z.boolean().default(false),
|
||
});
|
||
```
|
||
|
||
Rules encoded here (build fails on violation): title length, description length window, OG image
|
||
present, canonical is a valid URL. This is the "rules are being followed" guarantee you asked
|
||
about — enforced on every `npm run build` in CI before ArgoCD ever sees the image.
|
||
|
||
### 1b. Build-time SEO linter (what Zod can't see)
|
||
|
||
Zod validates frontmatter, not rendered HTML. Add a small check (a Node script run in CI, and/or
|
||
Playwright assertions in `tests/`) that fails the build if any built page in `dist/` violates:
|
||
|
||
- Exactly **one `<h1>`** per page.
|
||
- Heading hierarchy has no skipped levels (no `<h4>` before an `<h2>`).
|
||
- `<title>`, `<meta name="description">`, `<link rel="canonical">` all present and non-empty.
|
||
- Every `<img>` has non-empty `alt` (parity plan already requires this).
|
||
- At least one JSON-LD block present, and **every** JSON-LD block `JSON.parse`s.
|
||
- Internal links resolve (no 404s to our own routes).
|
||
- Canonical host matches the chosen production host (apex vs `www` — see §5).
|
||
|
||
Recommended: a `scripts/seo-lint.mjs` that walks `dist/**/*.html` with a lightweight parser, plus
|
||
keep the existing Playwright suite for link/CTA coverage.
|
||
|
||
## 2. Per-page technical SEO requirements
|
||
|
||
Already implemented in `src/layouts/BaseLayout.astro` (keep + enforce):
|
||
|
||
- `<title>`, `<meta name="description">`, `<link rel="canonical">`
|
||
- Open Graph (`og:type/title/description/url/image`) + Twitter `summary_large_image`
|
||
- JSON-LD injection slot (Organization + WebSite globally, page schema per route)
|
||
- `lang="en"`, viewport, skip-link, RSS `<link rel="alternate">`
|
||
|
||
To add/verify:
|
||
|
||
- [ ] **Per-page OG images** — currently every page falls back to `/assets/water-strategy.jpg`.
|
||
Generate topic-specific OG images (services, focus areas, insights) at 1200×630.
|
||
- [ ] **`robots` meta** wired to the `noindex` frontmatter flag (for `/thank-you/`, staging).
|
||
- [ ] **`article:published_time` / `modified_time`** OG tags on insights posts.
|
||
- [ ] **Breadcrumbs** (visible + `BreadcrumbList` schema) on services/focus/insights detail pages.
|
||
- [ ] **Sitemap** already via `@astrojs/sitemap` (excludes `/thank-you/`); add `lastmod` from
|
||
content dates for insights.
|
||
- [ ] **Trailing-slash consistency** — config uses `trailingSlash: "always"`; ensure canonical +
|
||
internal links all match to avoid duplicate-URL dilution.
|
||
|
||
## 3. Structured data (JSON-LD) matrix
|
||
|
||
| Page / type | Required schema | Notes |
|
||
|-------------|-----------------|-------|
|
||
| Global | `Organization`, `WebSite` | Already in `src/utils/schema.ts`. Add `sameAs` (real socials), `logo`, `contactPoint` (phone/email), `areaServed`. |
|
||
| Homepage | `Organization` + `WebSite` (+ optional `SearchAction`) | — |
|
||
| `/services/` + each service | `Service` (with `provider` → Organization, `serviceType`, `areaServed`) | One per detail page. |
|
||
| `/focus-areas/` + each focus | `Service` or `DefinedTerm`/`Article` | Focus areas double as topic-authority hubs. |
|
||
| About / Brad Rodgers | `Person` + `AboutPage`; `Person.worksFor` → Organization; `knowsAbout[]` | **Owner-approved facts only.** Core GEO/E-E-A-T asset. |
|
||
| Insights posts | `Article`/`BlogPosting` (author=Person, publisher=Organization, dates, `headline`, `image`) | Powers AI citation of thought leadership. |
|
||
| Insights index | `Blog` / `CollectionPage` | — |
|
||
| Contact | `ContactPage` + `Organization.contactPoint` | Use real values, never the WP placeholders. |
|
||
| FAQ sections | `FAQPage` — **only where the same Q&A is visibly rendered** | Highest AI-citation rate. Never add invisible FAQ schema. |
|
||
| Breadcrumbs | `BreadcrumbList` | Detail pages. |
|
||
|
||
Rule: **structured data must reflect content that is visibly present on the page.** No hidden-text
|
||
schema, no unsupported claims, no fabricated review/rating markup.
|
||
|
||
## 4. AEO / GEO content rules
|
||
|
||
Applied during Phase 2 copy restoration and Phase 5+ enhancements (must not violate the
|
||
no-paraphrasing parity rule on restored copy):
|
||
|
||
- **Answer-first**: each section's opening sentence should stand alone as a direct answer /
|
||
definition (e.g., "Marine conservation finance is …"). Extractable by answer engines.
|
||
- **Semantic headings** phrased as the questions/topics people ask ("What makes PFAS remediation a
|
||
capital formation problem?").
|
||
- **FAQ modules** on high-intent authority pages (PFAS economics, marine conservation finance) —
|
||
visible Q&A + `FAQPage` schema. Owner-approved answers only.
|
||
- **Entity consistency**: use canonical terminology everywhere — *catalytic capital*, *aquatic
|
||
resources*, *marine & freshwater ecosystems*, *PFAS remediation*, *marine conservation finance*,
|
||
*environmental markets*, *coastal resilience*. Feeds `primaryTopic` + internal linking.
|
||
- **Internal linking graph**: services ↔ focus areas ↔ insights ↔ principal bio. Dense, relevant
|
||
internal links are a top GEO signal.
|
||
- **Freshness**: insights carry `publishDate`/`modifiedDate`; surface "Updated" dates.
|
||
- **Statistics & specifics**: the homepage metrics (27 years, $10B+, $3.5B+, 275+) are exactly the
|
||
kind of concrete, citable facts answer engines favor — keep them prominent and, where true,
|
||
attributable.
|
||
|
||
## 5. AI-crawler policy (`robots.txt`) — **owner decision required**
|
||
|
||
Current `public/robots.txt` = `Allow: /` for everyone. 2026 best practice is **explicit per-agent
|
||
rules**, splitting *search/answer* bots (that can cite and send traffic) from *training* bots
|
||
(that ingest content into model weights). Decision matrix:
|
||
|
||
| Bot | Job | Recommendation |
|
||
|-----|-----|----------------|
|
||
| `OAI-SearchBot`, `ChatGPT-User` | ChatGPT search index + live citation fetch | **Allow** |
|
||
| `Claude-SearchBot` | Claude web-search citations | **Allow** |
|
||
| `PerplexityBot` | Perplexity answer citations | **Allow** |
|
||
| `Google-Extended` | Gemini + Google AI Overviews | **Allow** (maximizes AI visibility) |
|
||
| `Googlebot`, `Bingbot` | Classic search | **Allow** |
|
||
| `GPTBot` | OpenAI model *training* | Owner choice — allow = max reach, disallow = opt out of training |
|
||
| `ClaudeBot` | Anthropic model *training* | Owner choice (same trade-off) |
|
||
| `Bytespider`, stealth scrapers | Known to ignore robots.txt | Consider **edge/WAF block** (Cloudflare) — robots.txt won't stop them |
|
||
|
||
**Recommended default for an authority-seeking advisory firm: allow all search/answer + classic
|
||
bots (maximize citation and discoverability); training bots are a values call.** Because BlueCap
|
||
wants to be cited as an authority, allowing everything is the simplest high-visibility posture.
|
||
Non-compliant scrapers are handled at the Cloudflare edge (see gitops doc §4.2), not robots.txt.
|
||
|
||
Keep `llms.txt` (already present, low cost) but treat it as supplemental, not load-bearing.
|
||
|
||
## 6. Measurement & Monitoring — **recommendation, needs sign-off**
|
||
|
||
Three distinct layers; today none is decided.
|
||
|
||
### 6a. Uptime / infrastructure — reuse the existing `monylog` stack (do NOT deploy Uptime Kuma)
|
||
|
||
The homelab already runs a full self-hosted **LGTM + Alertmanager** stack on the `monylog` box
|
||
(`10.66.15.21`, Ansible role `pglta`): Prometheus (15d retention, remote-write receiver), Grafana,
|
||
Loki, Tempo, Alertmanager → **Pushover** (creds from OpenBao) with **ntfy** fallback → phone. It
|
||
runs **outside k8s on purpose** so it still alerts when the cluster is down. Grafana Alloy runs as
|
||
a k3s DaemonSet and remote-writes cluster metrics + ships all pod logs to monylog.
|
||
|
||
**Already flowing for BlueCap (zero work):**
|
||
- **Pod/deployment health** — Alloy scrapes kube-state-metrics + annotated pods → Prometheus.
|
||
`kube_deployment_status_replicas_unavailable`, restarts, OOMKills are queryable now.
|
||
- **nginx logs** — Alloy ships all pod logs to Loki. 5xx rate / errors available via LogQL now.
|
||
|
||
**To add (small):**
|
||
1. **blackbox_exporter** — the one gap (not in the monylog compose). Add as a compose service +
|
||
a Prometheus scrape job. Probe the in-cluster Service (app health) **and** the public URL (full
|
||
edge-path uptime + TLS expiry). monylog is off-cluster → probing the public URL is a genuine
|
||
external vantage. Metrics: `probe_success`, `probe_http_status_code`,
|
||
`probe_ssl_earliest_cert_expiry`, latency.
|
||
2. **`web-bluecap.yml` alert rules** (mirror the existing host-infra rules pattern): `SiteDown`
|
||
(`probe_success==0` for 2m), `CertExpiringSoon` (<14d), `High5xx` (Loki/blackbox),
|
||
`DeploymentDegraded`, `PodRestarting`.
|
||
3. **Alertmanager** already routes to Pushover — just add a `service: bluecap` label; no new
|
||
channel. **Grafana** already present → build a BlueCap uptime/latency/5xx/pod-health dashboard.
|
||
- **ArgoCD** health/sync status as an additional deploy-level signal.
|
||
|
||
### 6b. Visitor & behavior analytics — PostHog + Umami side-by-side trial (ADR-0005)
|
||
|
||
Decided: **run PostHog and Umami side-by-side initially** and keep whichever fits best. This is
|
||
**product/visitor analytics**, distinct from the `monylog` ops telemetry in §6a. Trial setup:
|
||
|
||
- **PostHog — Cloud free tier** (1M events/mo) for the trial, not self-host (self-hosting needs
|
||
ClickHouse/Kafka/etc.; PostHog's docs steer small users to Cloud). Reverse-proxy ingest through
|
||
our domain to reduce ad-blocker loss. Gives autocapture, funnels, session replay.
|
||
- **Umami — self-hosted** in k3s: cookieless, ~2 KB, no consent banner, always-on baseline
|
||
(pageviews, sources, devices, custom events). Aligns with the Digital-ESG positioning.
|
||
- Both wired through one config-driven `Analytics.astro` in `BaseLayout`, gated by env vars, so
|
||
either can be toggled and the winner kept without code churn. Same event set in both:
|
||
consultation-CTA clicks, per-service/focus CTA source (ties to the segmented-CTA strategy),
|
||
insights reads, contact submits.
|
||
- **Consent/privacy must cover the superset** — PostHog (esp. session replay) likely triggers
|
||
cookie-consent + privacy-policy obligations; add these **before** enabling PostHog in production.
|
||
Umami alone needs no banner.
|
||
- **Avoid GA4** unless Google Ads attribution is later required.
|
||
- Evaluate on insight value vs. weight/consent/ESG fit; record the winner in a follow-up ADR.
|
||
|
||
### 6c. Search & AI-citation visibility
|
||
- **Google Search Console** + **Bing Webmaster Tools** (free, essential) — submit sitemap, monitor
|
||
impressions/queries/coverage, Core Web Vitals.
|
||
- **AI-citation tracking** (emerging, optional): periodically test brand/topic prompts across
|
||
ChatGPT/Perplexity/Google AI Overviews and log whether BlueCap is cited. Lightweight manual
|
||
tracker to start; dedicated GEO tools are immature — don't over-invest yet.
|
||
- **Performance budget**: Lighthouse in CI; target 95+ SEO/Perf/Best-Practices/Accessibility.
|
||
Static Astro should hit this easily; guard against regressions from images (use Astro `Image`).
|
||
|
||
## 7. Pre-launch SEO checklist
|
||
|
||
- [ ] Zod SEO schema in place; build fails on title/description/OG/canonical violations.
|
||
- [ ] `scripts/seo-lint.mjs` (single H1, heading order, canonical, alt text, JSON-LD parses) green in CI.
|
||
- [ ] Every page: unique title, description in window, canonical, OG image, valid JSON-LD.
|
||
- [ ] Organization/WebSite/Service/Person/Article/Breadcrumb schema per §3, all visible-content-backed.
|
||
- [ ] `robots.txt` per §5 (owner-decided); `sitemap-index.xml` correct; `/thank-you/` + staging `noindex`.
|
||
- [ ] Canonical host decided (apex vs `www`) + redirect configured at the edge.
|
||
- [ ] Search Console + Bing verified, sitemap submitted.
|
||
- [ ] Analytics deployed (Umami) + consultation/CTA events firing.
|
||
- [ ] Uptime Kuma + external check live and alerting.
|
||
- [ ] Lighthouse ≥ 95 across the board; no CLS from unsized images.
|
||
|
||
## Sources (verified 2026-07-15)
|
||
|
||
- [The State of llms.txt in 2026 — aeo.press](https://www.aeo.press/ai/the-state-of-llms-txt-in-2026)
|
||
- [llms.txt Explained (May 2026) — Codersera](https://codersera.com/blog/llms-txt-complete-guide-2026/)
|
||
- [Structured Data for AEO and GEO: Schema Markup Guide 2026 — Kurieta](https://kurieta.com/schema-for-aeo-geo/)
|
||
- [Answer Engine Optimization: Complete AEO Guide [2026] — Frase](https://www.frase.io/blog/what-is-answer-engine-optimization-the-complete-guide-to-getting-cited-by-ai)
|
||
- [Are FAQ Schemas Important for AI Search, GEO & AEO? — Frase](https://www.frase.io/blog/faq-schema-ai-search-geo-aeo)
|
||
- [The AI User-Agent Landscape in 2026: A Complete Reference — No Hacks](https://nohacks.co/blog/ai-user-agents-landscape-2026)
|
||
- [AI Crawlers Explained: GPTBot, ClaudeBot, PerplexityBot (2026) — Anagram](https://www.anagram.ai/blog/ai-crawlers-explained-gptbot-claudebot-perplexitybot-and-how-to-let-them-in-2026)
|
||
- [Robots.txt for AI Crawlers in 2026 — Cubitrek](https://cubitrek.com/blog/robots-txt-2026-managing-ai-crawler-budgets)
|