How to Build the Technical Foundation for a Discoverable Documentation Site

Build the technical foundation for a discoverable documentation site by delivering one trustworthy answer at one stable canonical URL to people, search engines, and AI agents. Work through seven layers in order: canonical identity, crawlable HTML, an accurate URL inventory, search-ready page signals, clean machine representations, explicit crawler policy, and version and language handling. Test the deployed output, not only the publishing configuration. A generated sitemap cannot repair a dead canonical host, llms.txt cannot replace missing page content, and crawler permission cannot make client-rendered or private text retrievable. Stop and repair the first broken layer before adding features to a later one.

Layer 0: establish one canonical documentation identity

Choose one public docs origin, normally https://docs.example.com. Before publishing any metadata that points there:

Create the exact DNS record supplied by the docs platform.
Confirm the hostname resolves through more than one public resolver.
Confirm a valid TLS certificate is served.
Confirm the root, a guide, a reference page, and discovery files return the intended content.
Make every identity signal use the same host.

Identity signals include:

redirects;
<link rel="canonical">;
Open Graph URLs;
XML sitemap locations;
robots.txt sitemap declarations;
JSON-LD url and @id values;
llms.txt and Markdown links;
OpenAPI servers;
Agent Cards, skills, MCP cards, and package metadata;
links from the marketing site and product.

A preview host that canonicalizes to an unresolvable custom domain is worse than an unfinished custom domain hidden from launch: it tells every consumer that the live copy is not authoritative and points them to a dead address.

Move a documentation domain without splitting canonical signals

Redirect the old host and old paths permanently to matching canonical paths with 301 or 308.
Preserve the path and query where appropriate; avoid redirecting every old page to the homepage.
Keep redirects for as long as old links, packages, or model priors may exist.
Update internal links, sitemaps, cards, contracts, and external profiles.
Use one current brand name; explain a former name on one dated migration/entity page when needed.
Crawl both old and new hosts after deployment to find chains, loops, split signals, and leaks.

Exit: one host resolves and serves the corpus; every declared URL is reachable or intentionally redirected; no canonical or machine link points to a dead or preview host.

Layer 1: serve crawlable canonical documentation HTML

Every indexable page should:

return 200 for the canonical URL;
return meaningful article text in the initial HTML response;
use a self-referencing canonical;
have one visible article H1;
use real <a href> links for navigation and contextual links;
avoid noindex, login, consent, challenge, or client-rendering barriers;
expose the same core facts to users and crawlers;
have a stable URL that does not depend on a session or temporary token.

Inspect the raw response, not only the browser after JavaScript runs. A crawler can render JavaScript, but requiring it adds cost and more failure modes. Essential documentation should be directly present.

Return correct HTTP statuses for documentation pages

Use a real 404 or 410 for missing/retired pages, not a friendly error page returning 200.
Use a permanent redirect for moved pages.
Avoid redirect chains and locale/device redirects based only on guesses.
Make root behavior deliberate: serve a useful index or permanently redirect to the canonical introduction/locale.
Never expose private drafts because a platform’s “index hidden pages” setting was enabled globally.

Layer 2: publish an accurate documentation URL inventory

Publish only canonical documentation URLs in the XML sitemap

Publish a canonical sitemap and list it in robots.txt. The sitemap should contain:

absolute HTTPS URLs on the canonical host;
every intended indexable HTML page;
only canonical 200 pages;
meaningful lastmod values when the page changed materially.

Exclude:

.md alternates;
preview/staging hosts;
redirected, missing, or noindex pages;
parameter duplicates;
incomplete translations and private drafts.

A sitemap helps discovery and diagnosis; it does not guarantee indexing. Submit it to verified Google Search Console and Bing Webmaster Tools properties and inspect coverage there. Google’s sitemap guidance specifically warns that lastmod should describe a significant page change. Do not stamp every URL on every build.

Build crawl paths with contextual internal links

The sitemap is not a replacement for a usable crawl path. Link from:

the marketing site to relevant guides, not only the docs homepage;
the quickstart to the next implementation task;
concept and decision pages to each implementation branch;
reference operations to task guides and exact errors;
errors back to their operation;
changelog entries to migrations and affected pages.

Keep the top user jobs within roughly three meaningful navigation choices from the docs root.

Layer 3: make documentation HTML search-ready

Align titles, descriptions, headings, and URLs around one question

Every indexable page needs:

a unique title that states the task/error/decision and product context;
a unique factual description, often around 140–160 characters;
one article H1 aligned with the target question;
sequential, descriptive H2/H3 headings;
descriptive link text and meaningful image alt text.

Character ranges are presentation guardrails, not ranking laws. Clarity and differentiation matter more than hitting an exact count. Plan descriptive folders and slugs before publishing, and add permanent redirects when a path changes. Use the patterns and worked URL examples in Optimize Documentation Pages for SEO, GEO, and Agent Discovery.

Consolidate duplicate documentation URLs with canonical signals

Redirects and rel=canonical are strong canonical signals; sitemap inclusion is weaker. Keep all signals aligned. A self-canonical page should not appear in the sitemap under another host, language, or trailing-slash variant. See Google’s canonical guidance.

Add structured data only for visible documentation content

Publish JSON-LD only when it describes visible, current content. Useful types may include:

Organization with stable identity and evidence-backed profiles;
WebSite;
WebPage;
BreadcrumbList;
TechArticle or another accurate article type.

Include stable @id values, canonical URL, headline, description, publisher, mainEntityOfPage, and a truthful dateModified where applicable. Mark up product, FAQ, review, rating, or HowTo data only when the visible page qualifies and the markup can be maintained. Structured data clarifies meaning and may enable search features. It does not guarantee ranking or an LLM citation. Follow Google’s structured-data policies.

Keep documentation fast and accessible

Documentation must work on mobile, keyboard, and assistive technology. At minimum:

landmarks, skip links, focus states, labeled controls, and a logical heading outline;
readable contrast and zoom behavior;
text alternatives for meaningful images;
tables with accessible headers;
no essential information locked inside hover, animation, or a screenshot;
stable rendering without intrusive overlays.

Use field Core Web Vitals at the 75th percentile when enough data exists: LCP no more than 2.5 seconds, INP no more than 200 milliseconds, and CLS no more than 0.1. Use repeatable lab budgets only as a provisional check for a new or low-traffic site. See web.dev’s current metrics.

Layer 4: expose clean Markdown and machine-readable documentation

Preserve the complete answer in per-page Markdown

For technical documentation, expose a predictable Markdown representation such as /guide.md or content negotiation with Accept: text/markdown. A good Markdown response:

returns 200 with an accurate text/Markdown content type;
contains the same answer, version, warnings, links, code, and dates as HTML;
keeps tables and image descriptions readable;
removes navigation chrome, CSS, hydration state, and raw component implementation;
identifies the canonical HTML URL through an HTTP Link relation or a clear preamble;
is kept out of the search index with an appropriate X-Robots-Tag when it would otherwise compete with the HTML canonical;
is not added to the canonical HTML sitemap.

An illustrative Markdown preamble makes the representation self-describing:

---
canonical: https://docs.example.com/payments/verify-webhooks
docsIndex: https://docs.example.com/llms.txt
productVersion: 3.4
lastReviewed: 2026-08-08
pageType: how-to
prerequisites:
  - https://docs.example.com/authentication
related:
  - https://docs.example.com/errors/webhook-signature
---

Use the field names supported by the publishing system; the invariant is that an isolated Markdown page identifies its canonical answer, scope, review date, prerequisites, and next retrieval routes. When the platform supports a reusable preamble, make every deep-linked page disclose the corpus index:

> Documentation index: use https://docs.example.com/llms.txt to find the complete, current page list
> before searching for another answer.

This does not replace navigation or contextual links. It gives a client that entered through one retrieved page a deterministic route to the rest of the corpus. If one URL varies by Accept, send correct Vary headers and test cache behavior. If content varies by user agent, include the relevant variation too; a separate .md URL is simpler when edge caching is hard to control. Do not accept Markdown merely because the endpoint exists. Read samples from every page template. Interactive components often leak raw MDX/JSX or lose important content.

Use `llms.txt` as a documentation routing index

Treat llms.txt as a small routing file, not an SEO ranking trick and not a substitute for navigation. Use:

# Acme API

> Acme lets developers <specific category and outcome>. These docs cover <scope>.

## Get started

- [Make your first sandbox request](https://docs.acme.com/quickstart.md): Authenticate and receive a verified test response

## Core tasks

- [Verify webhook signatures](https://docs.acme.com/webhooks/verify.md): Validate the raw payload before processing an event

## Reference and trust

- [API contract](https://docs.acme.com/openapi.json): Current operations, schemas, errors, and authentication
- [Changelog](https://docs.acme.com/changelog.md): Breaking changes, migrations, and release dates

Requirements:

one clear H1 and a precise product definition;
H2 groups that help a retriever route by job;
a distinct task description for every link;
canonical, working Markdown destinations;
current version/deprecation instructions only when they are precise and maintainable;
links to relevant product, locale, contract, changelog, pricing, limits, and trust surfaces;
generated link and integrity checks on deploy.

Keep the root file inspectable. Use section or locale indexes when the corpus is large. An optional llms-full.txt may help bulk ingestion, but it should be generated from the same source, size-monitored, and never be the only route to a page. The llms.txt specification is a proposal. Major search and model providers do not currently document it as a general ranking factor. Measure target-client use instead of assuming it.

Publish validated API contracts and capability catalogs

For API/event products, publish a validated OpenAPI or AsyncAPI contract at a stable URL. It should match production:

server URLs and environments;
authentication and authorization;
operation lifecycle;
parameter types and formats;
request and response examples;
errors and retry behavior;
pagination, idempotency, rate limits, and webhooks/events.

Run validation and sampled sandbox/contract calls in CI. Generate reference pages from the contract, then add task guides that explain real workflows. Well-known API catalogs, Agent Cards, Agent Skills, and MCP cards can make capabilities addressable. Every advertised URL, digest, version, and transport must be validated after deployment. A manifest that links to dead resources is worse than no manifest for strict clients.

Layer 5: set documentation crawler policy by purpose

Do not use one “AI bots” switch. Decide what the business wants for:

conventional search indexing;
answer-engine search/citation;
user-requested fetches;
model training.

Record legal/product approval and test the actual CDN/WAF path. A permissive robots.txt does not help if the firewall blocks the request; a robots rule may not control every user-initiated fetch. Provider behavior checked 2026-08-07; recheck primary documentation before changing policy.

Purpose	Provider token	Documented behavior and action
Google Search	`Googlebot`	Allow public canonical docs that should appear in Search.
Gemini training/certain grounding uses	`Google-Extended`	Separate control; Google says it does not affect Search inclusion or ranking.
ChatGPT search/citations	`OAI-SearchBot`	Allow if pages should be eligible for ChatGPT search answers.
OpenAI model training	`GPTBot`	Separate training decision. It can be blocked while search is allowed.
ChatGPT user-requested fetch	`ChatGPT-User`	OpenAI says robots rules may not apply; enforce private access with auth/WAF and test directly.
Perplexity search index	`PerplexityBot`	Perplexity says it honors robots. Test CDN/WAF and published IP data where available.
Perplexity user-requested fetch	`Perplexity-User`	Perplexity says it generally ignores robots; use access controls and direct tests.
Claude search	`Claude-SearchBot`	Set separately for search visibility. Anthropic says it honors robots.
Anthropic model training	`ClaudeBot`	Separate training decision. Anthropic says it honors robots.
Claude user-requested fetch	`Claude-User`	User retrieval token. Anthropic says it honors robots; test the end-to-end request path.

For example, this robots.txt allows conventional and answer-engine search while blocking one training crawler. It is an example of the output shape, not a universal policy:

User-agent: *
Allow: /
Disallow: /login
Disallow: /private/

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://docs.example.com/sitemap.xml

Add or remove named groups only after legal and product owners approve the corresponding search, user-fetch, and training purposes. Test the public CDN response for each user agent; the file cannot override a firewall challenge or protect private content. Primary references:

Do not trust a user-agent string alone for privileged access. Public documentation should be public; private documentation should require real authentication.

Layer 6: handle documentation versions and languages explicitly

Give materially different documentation versions stable URLs

Give supported major versions stable URL namespaces when procedures differ.
State the current/default version in entry pages and machine indexes.
Keep old-version pages accessible while supported and clearly labeled.
Redirect only when the replacement is behaviorally compatible.
Link changelog, migration, old reference, and new reference together.
Test that search does not select an obsolete version for current queries.

Give each complete documentation translation a stable locale URL

Use separate stable URLs such as /en/... and /ja/.... For every complete translation pair:

set the correct page language;
use a self-canonical within the locale;
publish reciprocal hreflang values and an x-default where appropriate;
link language alternatives without forced browser-language redirects;
keep sitemap and machine indexes locale-aware;
distinguish a missing translation from an accidental orphan.

For an English/Japanese pair, each page should emit the same reciprocal set with locale-specific self-canonicals:

<link
  rel="alternate"
  hreflang="en"
  href="https://docs.example.com/en/quickstart"
/>
<link
  rel="alternate"
  hreflang="ja"
  href="https://docs.example.com/ja/quickstart"
/>
<link
  rel="alternate"
  hreflang="x-default"
  href="https://docs.example.com/en/quickstart"
/>

Do not emit a Japanese alternate that redirects, returns an untranslated page, or canonicals back to English. Check every pair rather than sampling one locale landing page. See Google’s multilingual guidance.

Verify SEO and GEO output from the documentation platform

Verify Mintlify’s generated SEO and agent surfaces

Mintlify’s SEO configuration can generate semantic HTML, metadata, canonical signals, robots, sitemap, and schema. Its llms.txt support can generate per-page Markdown, indexes, and full-corpus output. The hosted docs MCP adds a retrieval surface. Use the platform, but verify the deployed result:

configure and verify the custom domain before making it canonical;
ensure every public page that should be discovered is intentionally in navigation/indexing;
hand-write page descriptions so generated indexes have useful routing text;
inspect Markdown from interactive components;
verify locale alternates and identity across every generated file;
link-check machine files and digests after deploy;
keep full-corpus and MCP output optional unless target clients use them.

Configuration names change. Check the current schema and run the platform validator rather than copying a stale config fragment.

Verify ReadMe’s generated SEO and agent surfaces

ReadMe may provide server-rendered pages, canonical metadata, Markdown alternates, content negotiation, and generated LLM surfaces depending on the project and version. Do not assume it generated a sitemap, complete descriptions, clean Markdown, or correct schema because another ReadMe site did. For each project:

inspect every expected discovery endpoint;
verify platform SEO settings and custom head/schema support;
read sample .md output from every component-heavy template;
check whether generated indexes describe every link;
test preview/custom-domain and deploy behavior;
escalate platform limitations only after confirming the issue in the production response.

Generate custom documentation surfaces from one content graph

A custom Next.js, Docusaurus, Astro, or similar site can meet the same invariants. Prefer generation from one content graph so navigation, sitemap, canonicals, Markdown, LLM indexes, and redirects cannot silently drift apart.

Run copyable checks against the production documentation site

Run these against the public origin after deployment. Replace the two variables once; the remaining commands use the same host and representative page.

export DOCS_ORIGIN="https://docs.example.com"
export DOCS_SAMPLE_PATH="/quickstart"

# The canonical host resolves through independent public resolvers.
dig @8.8.8.8 +short docs.example.com
dig @1.1.1.1 +short docs.example.com

# The representative page and discovery files return from the public edge.
curl -fsSIL "$DOCS_ORIGIN$DOCS_SAMPLE_PATH"
curl -fsSL "$DOCS_ORIGIN/robots.txt" | head -40
curl -fsSL "$DOCS_ORIGIN/sitemap.xml" | head -20
curl -fsSL "$DOCS_ORIGIN/llms.txt" | head -40

# HTML contains an article title, canonical, and useful server-delivered text.
curl -fsSL "$DOCS_ORIGIN$DOCS_SAMPLE_PATH" \
  | rg -o '<title>[^<]+|rel="canonical"[^>]+|<h1[^>]*>[^<]+'

# The explicit Markdown URL is non-empty text rather than HTML or a soft 404.
curl -fsSIL "$DOCS_ORIGIN$DOCS_SAMPLE_PATH.md"
curl -fsSL "$DOCS_ORIGIN$DOCS_SAMPLE_PATH.md" | head -40

# Content negotiation varies only for clients that request it.
for agent in "Claude-User/1.0" "ChatGPT-User/1.0" "curl/8.0"; do
  curl -sS -o /dev/null -L -A "$agent" -w "$agent  %{http_code}  %{content_type}\n" \
    "$DOCS_ORIGIN$DOCS_SAMPLE_PATH"
done
curl -sS -o /dev/null -L -H 'Accept: text/markdown' \
  -w 'Accept markdown  %{http_code}  %{content_type}\n' \
  "$DOCS_ORIGIN$DOCS_SAMPLE_PATH"

# Report how many llms.txt links contain a routing description.
curl -fsSL "$DOCS_ORIGIN/llms.txt" \
  | awk '/^- \[/{total++} /^- \[.*\]\(.*\): .+/{described++} END {print described " described of " total}'

# Catch changelog soft 404s and unrendered component leakage.
curl -fsSL "$DOCS_ORIGIN/changelog.md" | head -c 100
curl -fsSL "$DOCS_ORIGIN/llms-full.txt" \
  | rg -n '^import .+ from|<[A-Z][A-Za-z]+[ />]' \
  | head

Interpret the results literally:

no DNS answer means stop; every canonical and machine link on that host is dead;
200 plus an HTML document at a Markdown or changelog URL is a soft failure, not a pass;
different HTML and Markdown facts fail parity even when both responses return 200;
fewer described links than total links means the routing index is incomplete;
raw component imports or tags mean the machine representation leaked implementation syntax.

Block documentation deployment when a technical discovery gate fails

Fail production promotion when an applicable blocker fails:

DNS/TLS and canonical host consistency;
intended status, redirect, and noindex behavior;
sitemap parse, URL status, canonical, and meaningful lastmod rules;
unique titles/descriptions and one article H1;
broken internal links and image references;
schema parse and visible-content consistency;
HTML/Markdown factual parity and raw-component leakage;
every llms.txt, card, catalog, contract, skill, MCP, and digest link;
OpenAPI/AsyncAPI validation and current example tests;
locale alternate reciprocity;
accessibility and performance regression budgets;
secret/private-content scans.

Run representative checks in pull requests and a full public-origin check after deploy. Cache, DNS, CDN, WAF, platform rendering, and generated digests can fail only after deployment.

​Layer 0: establish one canonical documentation identity

​Move a documentation domain without splitting canonical signals

​Layer 1: serve crawlable canonical documentation HTML

​Return correct HTTP statuses for documentation pages

​Layer 2: publish an accurate documentation URL inventory

​Publish only canonical documentation URLs in the XML sitemap

​Build crawl paths with contextual internal links

​Layer 3: make documentation HTML search-ready

​Align titles, descriptions, headings, and URLs around one question

​Consolidate duplicate documentation URLs with canonical signals

​Add structured data only for visible documentation content

​Keep documentation fast and accessible

​Layer 4: expose clean Markdown and machine-readable documentation

​Preserve the complete answer in per-page Markdown

​Use llms.txt as a documentation routing index

​Publish validated API contracts and capability catalogs

​Layer 5: set documentation crawler policy by purpose

​Layer 6: handle documentation versions and languages explicitly

​Give materially different documentation versions stable URLs

​Give each complete documentation translation a stable locale URL

​Verify SEO and GEO output from the documentation platform

​Verify Mintlify’s generated SEO and agent surfaces

​Verify ReadMe’s generated SEO and agent surfaces

​Generate custom documentation surfaces from one content graph

​Run copyable checks against the production documentation site

​Block documentation deployment when a technical discovery gate fails