Crawling your site

A crawl walks your site from its home page, following its links and its sitemap, and fetches what it finds, recording what each page is and what is wrong with it. It is paid for by the page, which is why a plan states a number of audit pages a month, and a number of pages one crawl may read, rather than a number of crawls.

How big one crawl is

Pressing “Crawl now” sizes the crawl to whatever is left of this month’s audit pages, up to the plan’s per-crawl ceiling, and refuses when there is nothing left — or when the workspace is read-only, or its monthly lookup budget is spent. The screen states both beside the button, the ceiling and the pages left this month — unless the workspace is read-only, when it says crawls are paused instead — so the size is known before you press it, and the crawl counts that whole size against the month when it starts, however few pages the site turns out to have. A site larger than the ceiling is read as far as the ceiling, from the home page, every time — a second crawl reads the same first pages again rather than the next ones — and the screen says when a crawl stopped there.

A crawl is work done by somebody else’s crawler and is not instant: a large site can take several hours, and one whose pages are slow to load more than half a day. The screen says when one is still running, and how many pages it has fetched so far, rather than showing you last week’s score as though it were this one.

The weekly crawl is an instruction, not a schedule you manage

Switching the weekly crawl on asks for one crawl a week, sized the same way out of whatever that month has left. It is refused only when this month is already fully spent, and a month that runs dry in week three leaves the switch exactly as it was — an allowance running out is not you withdrawing a standing instruction. Switching it off is never refused.

What it cannot see

A crawl starts from your site and follows what is linked, so a page nothing links to is found only when your sitemap lists it, and without a sitemap the count of such pages reads as a dash rather than as none. The block of findings at the foot of the Issues tab is a different instrument with a different blind spot: it is taken from the handful of your own pages that already earn traffic, named by a run rather than discovered by a crawl, so it cannot find a page that does not exist either. The two are kept apart on the screen and each says which it is.

What it reports

Every check the crawl ran, with how many pages each one found — the ones that found none included — grouped by how serious each one is. The pages themselves, by the answer each gave, with each page’s title, where a redirect goes and the checks it failed, and, for a page that loaded, its depth from the home page and how many pages link to it. Beside them, the broken links it followed, its redirect chains, the addresses kept out of search results and why, and a handful of facts about the whole site such as its certificate, its sitemap and its robots.txt file. A crawl that fetched no pages has no score and says so, which is a different state from a site with nothing wrong.