SSD-0007 · Status: stable · Version 1.0 · Reviewed 12 November 2026 · Wikidata Q80776
Definition
robots.txt is a plain-text file at a site’s root that tells automated crawlers which paths they may fetch — enforced by convention and by crawler cooperation, not by the server.
robots.txt 是放在網站根目錄的純文字檔案，用來告訴自動化檢索器哪些路徑可以抓取——依靠慣例與檢索器自願遵守，而不是靠伺服器強制執行。
In practice
It is a request, not a lock. A well-behaved crawler reads it before fetching; a scraper may ignore it entirely, and nothing in the protocol stops that. Two distinctions matter. First, robots.txt controls crawling, not indexing: a URL disallowed here can still appear in results if other pages link to it, because the engine is forbidden from fetching the page in order to read a noindex instruction. Second, blocking a crawler removes the page from that crawler’s index as well as its training set — which is why the same line is now used both for SEO hygiene and for AI opt-out, with consequences for both.
它是請求，不是鎖。守規矩的檢索器會先讀它；爬蟲程式可以完全不理，協議本身無法阻止。有兩個區別很重要。第一，robots.txt 控制的是檢索，不是索引：在這裡被禁止的網址，若有其他頁面連結，仍可能出現在結果中，因為引擎被禁止抓取該頁去讀取 noindex 指令。第二，封鎖某個檢索器，會同時把頁面從該檢索器的索引與其訓練資料中移除——這就是為何同一行指令現在既用於 SEO 衛生，也用於 AI 退出授權，而兩者都有代價。
What it is not
- Not a security control. Never use it to protect private data. Use authentication and server-side access control.
- Not a substitute for noindex. Different stage: robots.txt prevents fetching, noindex prevents listing after a fetch.
- Not guaranteed to be honoured. Reputable search crawlers follow it. Compliance by AI and third-party crawlers varies, and the protocol cannot compel anyone.
Status
Stable as a protocol: standardised as RFC 9309 in 2022, after decades of de facto use. What is not stable is who obeys it, which now varies by crawler operator and by commercial incentive.
Identity
- Wikidata: Q80776 — robots.txt
- English aliases: robots exclusion protocol; robots file
- 中文正名：robots.txt（業界通用原名，不翻譯）；亦稱「網站機器人排除協定檔案」
- Primary official source: RFC 9309, Robots Exclusion Protocol
Related terms
Sources
- IETF, RFC 9309: Robots Exclusion Protocol, rfc-editor.org/rfc/rfc9309.html (accessed 12 November 2026).
- Wikidata, item Q80776 “robots.txt” (accessed 12 November 2026).