Brad Rodgers
cd91c4b97c
Static Astro 5 site for BlueCap Strategies with exact live-site content parity, deployed to k3s via GitOps (ArgoCD). - Page copy in Markdown content collections (services, focus-areas, pages, insights) with a Zod SEO schema enforced at build time - Homepage + About restored to exact live copy; real live-site imagery - Build-time SEO linter (scripts/seo-lint.mjs) and Playwright e2e suite - Multi-stage Dockerfile (nginx serves dist/) and Kustomize manifests (k8s/) - Per-agent robots.txt; config-driven PostHog + Umami analytics scaffold - ADRs and engineering docs under docs/ Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
52 lines
1.4 KiB
Plaintext
52 lines
1.4 KiB
Plaintext
# BlueCap Strategies robots.txt — see docs/seo-aeo-optimization.md §5
|
|
# Policy: allow classic search + AI answer/search crawlers for maximum discoverability and
|
|
# answer-engine citation. Training-crawler policy (GPTBot, ClaudeBot, Google-Extended) is an
|
|
# owner decision — currently ALLOWED for maximum reach. To opt out of model training, uncomment
|
|
# the "Disallow: /" lines in the training-crawler block below.
|
|
|
|
# --- Classic search engines ---
|
|
User-agent: Googlebot
|
|
Allow: /
|
|
|
|
User-agent: Bingbot
|
|
Allow: /
|
|
|
|
# --- AI answer / search crawlers (these can cite us and send referral traffic) ---
|
|
User-agent: OAI-SearchBot
|
|
Allow: /
|
|
|
|
User-agent: ChatGPT-User
|
|
Allow: /
|
|
|
|
User-agent: Claude-SearchBot
|
|
Allow: /
|
|
|
|
User-agent: Claude-User
|
|
Allow: /
|
|
|
|
User-agent: PerplexityBot
|
|
Allow: /
|
|
|
|
# Google-Extended controls Gemini + Google AI Overviews (this is also a training signal for
|
|
# Google — treated as answer-visibility here; move to the training block to opt out).
|
|
User-agent: Google-Extended
|
|
Allow: /
|
|
|
|
# --- AI model-training crawlers (owner decision — allowed by default) ---
|
|
User-agent: GPTBot
|
|
Allow: /
|
|
# Disallow: /
|
|
|
|
User-agent: ClaudeBot
|
|
Allow: /
|
|
# Disallow: /
|
|
|
|
# --- Everything else ---
|
|
User-agent: *
|
|
Allow: /
|
|
|
|
# Note: non-compliant scrapers (e.g. Bytespider, stealth crawlers) ignore robots.txt; block those
|
|
# at the Cloudflare edge, not here (see docs/gitops-deployment-strategy.md §4.2).
|
|
|
|
Sitemap: https://www.bluecapstrategies.com/sitemap-index.xml
|