SSD-0005 · Status: stable · Version 1.0 · Reviewed 12 November 2026 · Wikidata Q45842
Definition
A web crawler is a program that fetches web pages automatically, mostly by following links from page to page, so that a search engine or other system can discover and store what exists on the web.
網路檢索器（又稱網路蜘蛛）是一種自動抓取網頁的程式，主要透過沿著頁面之間的連結逐頁前進，讓搜尋引擎或其他系統能夠發現並儲存網路上存在的內容。
In practice
Crawling is discovery, not ranking, and the two are routinely confused. A crawler that fetches a page has learned nothing about its quality; it has only moved the URL into a queue for processing and indexing. Three things decide whether a crawl happens at all: whether a link points to the page, whether the site’s robots rules allow it, and whether the server answers fast enough to be worth the crawl’s time. A page that is never crawled cannot rank, no matter how good it is — which is why crawl diagnostics come before content work on any site that has lost visibility.
檢索是「發現」，不是「排名」，而兩者經常被混為一談。檢索器抓取了一個頁面，對它的質素一無所知；它只是把該網址移入處理與索引的隊列。有三件事決定一次檢索會不會發生：有沒有連結指向該頁、網站的 robots 規則是否允許、伺服器回應是否夠快而值得檢索器花時間。一個從未被檢索的頁面不可能有排名，無論它多好——這就是為何在失去曝光度的網站上，檢索診斷要先於內容工作。
What it is not
- Not an indexer. Crawling fetches; indexing decides what to store and how to understand it. The two run as separate stages with separate failures.
- Not a scraper by definition. Scrapers extract content for reuse; crawlers map and store for retrieval. The same technology, different intent.
- Not one crawler. Search, social previews, archivers, AI training and AI answer retrieval each use their own agents, with their own rules and their own respect for robots.txt.
Status
Stable. The mechanism has been unchanged in principle since the 1990s. What has changed is who operates crawlers and why: retrieval for generated answers now sends agents to pages that no human will ever see.
Identity
- Wikidata: Q45842 — web crawler
- English aliases: crawler; spider; bot; web spider; robot
- 中文正名：網路檢索器／網路蜘蛛（香港、台灣）；中國大陸作「网络爬虫」
- Primary official source: Google crawler and fetcher overview
Related terms
Sources
- Google, “Google crawlers and fetchers”, developers.google.com/search/docs/crawling-indexing/overview-google-crawlers (accessed 12 November 2026).
- Wikidata, item Q45842 “web crawler” (accessed 12 November 2026).