Introduction

Spire is a crawler and scraper framework for Rust, built on tokio. You describe what to do with each kind of page, and Spire handles the rest of the crawl: the queue of pages to visit, fetching them concurrently and politely, retrying what fails, and stopping once there is nothing left to do.

use spire::backend::ReqwestBackend;
use spire::extract::html::{Html, Links};
use spire::frontier::queue::Queue;
use spire::frontier::seed::Seed;
use spire::{Crawler, Result, Router};

async fn page(html: Html, links: Links, queue: Queue) -> Result<()> {
    println!("{}", html.title().unwrap_or_default());
    queue.follow_all(links).await?;
    Ok(())
}

async fn crawl() -> Result<()> {
let router = Router::new().fallback(page);
let crawler = Crawler::new(ReqwestBackend::default(), router)
    .with_seed(Seed::get("https://example.com/")?);
crawler.run().await?;
Ok(())
}

Handlers are async functions, and their arguments are extractors: typed values taken from the page, such as its parsed HTML or the queue to add links to. If you have used axum, this will feel familiar.

What Spire does for you

• Routing: every request carries a tag, such as listing or product, that picks its handler. Tags can be an enum, checked by the compiler.
• Extraction: parse pages into structs with CSS selectors, using #[derive(Select)], or read JSON, text and raw bytes.
• A real frontier: deduplication, depth limits, retries with backoff, and pauses that honour Retry-After. With redb, the crawl survives a crash and resumes where it stopped.
• Politeness: per-host concurrency and delays, robots.txt, and limits on requests, time and each request’s duration.
• Two backends: plain HTTP through reqwest, and real browsers over WebDriver for pages that need JavaScript.
• Middleware, identities and testing: change every request, send each as one consistent visitor with its own proxy, user agent and cookies, and test handlers against fixed pages without a network.

The crates

Most projects depend only on spire, which re-exports the rest behind feature flags:

The API reference is on docs.rs. This book is the guide: it explains how the pieces fit together, chapter by chapter.

Installation

Spire needs Rust 1.95 or later. Add it with the features you need, and tokio to run it:

[dependencies]
spire = { version = "0.6", features = ["reqwest", "macros"] }
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

Feature flags

Enable testing in your dev-dependencies only; see Testing.

The HTTP backend’s own features

spire-reqwest has features of its own for compression, cookies and more. Depend on it next to spire to turn them on:

[dependencies]
spire-reqwest = { version = "0.6", features = ["gzip", "brotli", "cookies"] }

Your first crawl

This chapter builds a small crawler for an online shop: it walks the product listing page by page, and extracts every product it links to.

Tags

Every request carries a tag that says what kind of page it is, and the router picks the handler for it by that tag. Tags can be plain strings, but an enum deriving Tags lets the compiler check them:

use spire::request::Tags;

/// The kinds of page the crawl visits.
#[derive(Debug, Clone, Copy, Tags)]
enum Page {
    Listing,
    Product,
}

What to extract

#[derive(Select)] turns CSS selectors into a struct. Here, a product is the main.product element of its page, with a name and an optional price:

use spire::extract::html::Select;

#[derive(Debug, Select)]
#[select("main.product")]
struct Product {
    #[select("h1")]
    name: String,
    #[select(".price")]
    price: Option<f64>,
}

The crawl

use spire::backend::ReqwestBackend;
use spire::dataset::InMemDataset;
use spire::extract::html::Select;
use spire::frontier::seed::Seed;
use spire::frontier::{FrontierExt, InMemFrontier};
use spire::handler::{Collect, FollowLinks};
use spire::request::Tags;
use spire::{Crawler, Result, Router};

#[derive(Debug, Clone, Copy, Tags)]
enum Page {
    Listing,
    Product,
}

#[derive(Debug, Select)]
#[select("main.product")]
struct Product {
    #[select("h1")]
    name: String,
    #[select(".price")]
    price: Option<f64>,
}

#[tokio::main]
async fn main() -> Result<()> {
    let router = Router::new()
        // Two handlers on one listing page, fetched once: follow the next
        // page and every product.
        .route(Page::Listing, (
            FollowLinks::new(Page::Listing).select("a.next"),
            FollowLinks::new(Page::Product).select("article.product a"),
        ))
        .route(Page::Product, Collect::<Product>::new())
        .require::<Page>();

    let crawler = Crawler::new(ReqwestBackend::default(), router)
        .with_frontier(InMemFrontier::fifo().dedup().max_depth(5).retry(3).tag_backoff())
        .with_dataset(InMemDataset::<Product>::new())
        .with_concurrency(16)
        .with_seed(Seed::get("https://example.com/products")?.with_tag(Page::Listing));

    let stats = crawler.run().await?;
    let products = crawler.dataset::<Product>()?.read_all().await?;
    println!("{} products in {} requests", products.len(), stats.requests);
    Ok(())
}

Step by step:

1. The router. Listing pages run two built-in handlers in turn: FollowLinks queues the links matching a selector, with the tag to give them. Product pages run Collect, which extracts a Product and stores it. require fails at startup if a variant of Page has no route.
2. The frontier holds the requests waiting to be crawled, in first-in, first-out order. On top of it, dedup skips URLs already seen, max_depth stops five links away from the seed, retry tries failed requests three more times, and tag_backoff pauses a tag when a handler backs off.
3. The dataset stores what Collect extracts, in memory.
4. The seed is the first request: the listing, tagged as one.
5. run crawls until nothing is queued or in flight, then returns statistics. The products are then read back from the dataset.

Relative links resolve against the page they were found on, after redirects. Error statuses fail the request: 5xx and 429 are retried, honouring Retry-After, and other 4xx are not.

The rest of this book goes through each of these pieces in turn.

How a crawl works

A crawl moves seeds from a frontier, through a backend and a worker, into datasets:

            ┌──────────┐  pop   ┌─────────┐ connect ┌─────────┐
  seeds ──▶ │ Frontier │ ─────▶ │ Crawler │ ──────▶ │ Backend │
            └──────────┘        └─────────┘         └─────────┘
                 ▲                   │ Context          │ Client
          follow │                   ▼                  │
                 │      ┌──────────────────────────┐    │
                 └───── │ Worker: Router, handlers │ ◀──┘ fetch
                        └──────────────────────────┘
                              │ write       │ Outcome
                              ▼             ▼
                          Datasets    reported back to the frontier

• A seed is a request waiting to be crawled: a method and URI, with a tag, a depth (how many links away from the first seeds it is), a priority, and the attempt it is on. The crawler turns it into a request right before fetching.
• The frontier holds the seeds. It decides their order, and facades on top of it add deduplication, depth limits, retries and backoff. See The frontier.
• The crawler pops seeds and runs up to with_concurrency of them at once, within the limits of each host. See Limits and politeness.
• A backend hands out a client for each request. ReqwestBackend fetches over HTTP, and ThirtyfourBackend loads the page in a browser. See Browsers.
• The worker handles each request. It is usually a Router, which picks handlers by the request’s tag. See Routing.
• Datasets store what the handlers extract, one per item type. See Storing results.

One request, step by step

1. The crawler pops a seed, and waits if its host is busy.
2. The backend hands out a client, and the crawler builds a Context: the request, the client, a queue for new links, and the datasets.
3. Middleware, if any, runs first. See Middleware.
4. The router picks a handler by the tag, or by the URL’s path for an untagged request.
5. The handler’s extractors read what they need. The first one that needs the response fetches it, and the others share it: a page is fetched once however many handlers and extractors read it.
6. The handler queues links, writes items, and returns an Outcome.
7. The crawler reports the outcome to the frontier: done, retry, pause a tag, skip a tag, stop the crawl, or failed.

The crawl ends when the frontier is empty and no request is in flight, when a handler returns Outcome::Stop, when a limit is reached, or when it is shut down.

Outcomes

A handler returns anything that converts into an Outcome: () and Ok(()) mean the request is done, and an error fails it. Outcomes also retry a request, pause or skip a tag, or stop the crawl: see Errors and retries.

spire and spire-core

The crawler, frontier, backends, workers and datasets live in spire-core. spire adds the router, handlers and extractors on top, and re-exports everything. Depend on spire-core directly only to implement your own backend, frontier or dataset.

Routing

A Router picks the handler for each request. It is the crawl’s worker:

use spire::backend::ReqwestBackend;
use spire::{Crawler, Router};
async fn listing() {}
let router = Router::new().route("listing", listing);
let crawler = Crawler::new(ReqwestBackend::default(), router);

Handlers

A handler is an async function whose arguments are extractors, up to 12 of them, returning anything that converts into an Outcome: nothing, a Result, or an Outcome itself.

use spire::Result;
use spire::extract::html::{Html, Links};
use spire::frontier::queue::Queue;
use spire::request::Depth;
use spire::request::http::Uri;

async fn page(uri: Uri, depth: Depth, html: Html, links: Links, queue: Queue) -> Result<()> {
    println!("{uri} at depth {depth}: {:?}", html.title());
    queue.follow_all(links).await?;
    Ok(())
}

If an extractor fails, for example because the page is not valid JSON, the handler does not run and the request fails. Extractors lists every extractor.

Routes

Routes are tried in this order:

1. By tag: route(tag, handler) handles the requests with that tag.
2. By path: route_path(pattern, handler) handles untagged requests whose path matches the pattern. * matches within one path segment and ** across segments; the query is ignored. Path routes are tried in the order they were added.
3. The fallback: fallback(handler) handles the rest.

A request no route takes fails. A tag always wins over a path, and tags are not inherited: a link queued without a tag is routed by its path.

use spire::Router;
use spire::backend::Noop;

async fn listing() {}
async fn product() {}
async fn review() {}
async fn other() {}

let router: Router<Noop> = Router::new()
    .route("listing", listing)
    .route_path("/products/*", product)
    .route_path("/products/*/reviews/**", review)
    .fallback(other);

Routing a tag or path twice panics, as does merge, which adds another router’s routes, when both routers route the same one.

Typed tags

#[derive(Tags)], with the macros feature, makes an enum of tags:

use spire::Router;
use spire::request::Tags;
use spire::backend::Noop;

#[derive(Debug, Clone, Copy, Tags)]
enum Page {
    List,
    ItemReviews,
    #[tag(rename = "item")]
    Detail,
}

async fn list() {}
async fn reviews() {}

// A handler serving several variants takes the enum as an argument.
async fn detail(page: Page) {
    println!("{page:?}");
}

let router: Router<Noop> = Router::new()
    .route(Page::List, list)
    .route(Page::ItemReviews, reviews)
    .route(Page::Detail, detail)
    .require::<Page>();

• Variants become snake_case tags: ItemReviews is item_reviews. #[tag(rename_all = "kebab-case")] on the enum picks another case, and #[tag(rename = "...")] names one variant.
• require::<Page>() panics at startup, naming every variant without a route, so a forgotten route never surfaces mid-crawl.
• Persistent frontiers store tags as strings. When renaming a variant, keep its old name with #[tag(rename = "old_name")] so its stored seeds still route.

Several handlers on one page

A tuple of up to 4 handlers runs them in turn on the same page, which is fetched once. They stop at the first one that does not return Outcome::Continue:

use spire::handler::{Collect, FollowLinks};
use spire::Router;
use spire::extract::html::Select;
use spire::backend::Noop;
#[derive(Debug, Select)]
#[select("article")]
struct Product {
    #[select("h2")]
    name: String,
}
let router: Router<Noop> = Router::new().route("listing", (
    Collect::<Product>::new(),
    FollowLinks::new("listing").select("a.next"),
));

Built-in handlers

• FollowLinks::new(tag) queues every link on the page with tag, or only those matching .select(css), and without rel="nofollow" ones with .skip_nofollow().
• Collect::<T>::new() extracts every T: Select on the page and writes them to the dataset for T.
• FollowSitemap::new(tag), with the sitemap feature, follows a sitemap. See Sitemaps.

State

Router::with_state shares a value with every handler, which takes it, or a part of it, with the State extractor: see State.

Extractors

Extractors are a handler’s arguments. They borrow the request’s context, so any of them may read the response: the first that needs it fetches it, and the others share it.

Wrapping an extractor in Option or Result makes it infallible: the handler runs with None or the error instead of failing the request.

The response head and the query

Head gives the response’s status, headers and the URI it was served from, for example to read Last-Modified, or a status the backend’s StatusPolicy::Accept let through. Query<T> deserializes the request URI’s query string, for example the current page of a paginated API:

use serde::Deserialize;
use spire::Result;
use spire::extract::{Head, Query};
use spire::frontier::queue::Queue;

#[derive(Deserialize)]
struct Page {
    #[serde(default)]
    page: u32,
}

async fn listing(head: Head, Query(Page { page }): Query<Page>, queue: Queue) -> Result<()> {
    println!("page {page}: {} from {}", head.status, head.uri);
    queue.follow(format!("?page={}", page + 1)).await?;
    Ok(())
}

A URI without a query deserializes from an empty one, so a struct of optional or defaulted fields always extracts.

JSON and text

Json<T> deserializes the body with serde, which suits APIs. Deriving Deserialize needs serde’s derive feature in your own dependencies:

use serde::Deserialize;
use spire::Result;
use spire::extract::Json;
use spire::frontier::queue::Queue;

#[derive(Deserialize)]
struct Page {
    items: Vec<String>,
    next: Option<String>,
}

async fn api(Json(page): Json<Page>, queue: Queue) -> Result<()> {
    println!("{} items", page.items.len());
    if let Some(next) = page.next {
        queue.follow(next).await?;
    }
    Ok(())
}

Text is the body as a string, and Bytes the raw body.

Writing your own

An extractor implements FromContext: it gets the context, with the request and a way to fetch the response, and the router’s state. Its Rejection is what the handler returns instead of running when extraction fails, anything that converts into an Outcome, usually an Error:

use spire::backend::Client;
use spire::extract::FromContext;
use spire::request::http::header::CONTENT_LANGUAGE;
use spire::{Context, Error, Result};

/// The response's `Content-Language`, rejecting pages without one.
struct Language(String);

impl<C: Client, S: Sync> FromContext<C, S> for Language {
    type Rejection = Error;

    async fn from_context(cx: &Context<C>, _: &S) -> Result<Self> {
        let response = cx.response().await?;
        let language = response
            .headers()
            .get(CONTENT_LANGUAGE)
            .and_then(|value| value.to_str().ok())
            .ok_or_else(|| Error::worker("the page has no language"))?;
        Ok(Self(language.to_owned()))
    }
}

async fn page(Language(language): Language) {
    println!("{language}");
}

cx.response() fetches the request on its first call and shares the response with every other extractor, and cx.body() reads the body as it is now, e.g. a browser page as currently rendered. An extractor that only reads the request, like Uri, never fetches it.

State

Router::with_state shares a value with every handler of a router, such as a database pool, an API client or settings. A handler takes it with the State extractor, which clones it, so keep large or shared parts behind an Arc.

Parts of the state

FromRef takes a part of the state instead of all of it, so a handler, or an extractor of your own, asks only for what it needs:

use spire::Router;
use spire::extract::{FromRef, State};
use spire::backend::Noop;

#[derive(Clone)]
struct AppState {
    api_key: String,
}

struct ApiKey(String);

impl FromRef<AppState> for ApiKey {
    fn from_ref(state: &AppState) -> Self {
        Self(state.api_key.clone())
    }
}

async fn page(State(ApiKey(key)): State<ApiKey>) {
    println!("{key}");
}

let state = AppState { api_key: "secret".into() };
let router: Router<Noop, AppState> = Router::with_state(state).route("page", page);

Deriving it

With the macros feature, #[derive(FromRef)] writes these impls: every field’s type gets one, cloning the field, so each handler takes only the parts it needs. #[from_ref(skip)] leaves a field out, e.g. one of two fields of the same type, which would otherwise conflict:

use std::sync::Arc;

use spire::Router;
use spire::extract::{FromRef, State};
use spire::backend::Noop;

#[derive(Clone)]
struct Db;

#[derive(Clone)]
struct Config {
    user_agent: String,
}

#[derive(Clone, FromRef)]
struct AppState {
    db: Db,
    config: Arc<Config>,
}

async fn page(State(_db): State<Db>, State(config): State<Arc<Config>>) {
    println!("{}", config.user_agent);
}

let state = AppState {
    db: Db,
    config: Arc::new(Config { user_agent: "MyBot/1.0".into() }),
};
let router: Router<Noop, AppState> = Router::with_state(state).route("page", page);

HTML

The html feature, on by default, parses pages with scraper. Everything for HTML is in spire::extract::html: the Html extractor and its methods, extraction with Select, a page’s Meta and JsonLd, and scraper’s own API.

Parsing pages

Html is the parsed page. It finds the title and links, and extracts anything implementing Select:

use spire::Result;
use spire::extract::html::{Html, Links};

async fn page(html: Html, links: Links) -> Result<()> {
    println!("{:?}", html.title());
    for link in &links {
        println!("{} {}", link.uri(), link.text());
    }
    Ok(())
}

Links holds the page’s links, resolved as a browser resolves them, with their rel values and text: see Following links.

anchors() gives every link together with its <a> element, e.g. to read its text or attributes.

visible_text() is the text a reader sees: the body without scripts and styles, one line per paragraph or list item. main_text() keeps only the main content, the first <main>, <article> or role="main" element, without navigation and asides, or else the body without its header and footer as well, for indexing or summarizing pages. To walk the tree yourself, Html wraps scraper’s document, and spire::extract::html::scraper re-exports scraper at the version spire uses.

Markdown

With the markdown feature, the Markdown extractor converts the page’s main content, as main_text() finds it, to Markdown: headings, lists, emphasis, links, images, tables, block quotes and code are kept, which suits a model’s context, a vector store or a search index. Links and images resolve against the URI the page was served from, and its <base href>:

use spire::extract::html::Markdown;

async fn page(Markdown(markdown): Markdown) {
    println!("{markdown}");
}

Markdown::from_html converts with options: the whole body instead of the main content, without images, or with links as plain text:

use spire::extract::html::{Html, Markdown, MarkdownOptions};
use spire::request::http::Uri;

fn convert(html: &Html, uri: Uri) -> spire::Result<Markdown> {
    let options = MarkdownOptions::new().with_base(uri).whole_body().without_images();
    Markdown::from_html(html, &options)
}

The conversion is done by html-to-markdown-rs, which logs an html_to_markdown::convert span for every page.

Selecting with a derive

#[derive(Select)], with the macros feature, extracts a struct with CSS selectors:

use spire::extract::html::{Element, Select};

#[derive(Debug, Select)]
#[select("article.product")]
struct Product {
    #[select("h2")]
    name: String,
    #[select(".price")]
    price: Option<f64>,
    #[select("a.tag")]
    tags: Vec<String>,
    #[select("a.more", attr = "href")]
    link: String,
}

async fn listing(Element(products): Element<Vec<Product>>) {
    for product in products {
        println!("{}: {:?}", product.name, product.price);
    }
}

• The #[select("...")] on the struct is optional. With it, each element it matches is one value; without it, the whole page is one.
• Every field needs a selector: #[select("css")] for an element’s text, #[select("css", attr = "name")] for an attribute of it, or #[select(attr = "name")] for an attribute of the matched element itself.
• The field’s type decides how many matches it takes: Option<T> the first if any, Vec<T> all of them, and any other T exactly the first, failing if there is none. Text is trimmed, with its whitespace collapsed, and parsed with FromStr.

Element<T> takes one value, Element<Option<T>> the first if any, and Element<Vec<T>> all of them.

Metadata and structured data

Meta reads a page’s <head> without selectors: its title, description, canonical URL, language, <meta name="robots">, and Open Graph and Twitter fields. JsonLd<T> reads its schema.org data, which many product, article and event pages publish in <script type="application/ld+json"> blocks, often the most reliable data on the page:

use serde::Deserialize;
use spire::Result;
use spire::extract::html::{JsonLd, Meta};
use spire::frontier::queue::Queue;

/// The nodes to keep, by their `@type`.
#[derive(Deserialize)]
#[serde(tag = "@type")]
enum Node {
    Product { name: String, sku: Option<String> },
    #[serde(other)]
    Other,
}

async fn product(meta: Meta, JsonLd(nodes): JsonLd<Node>, queue: Queue) -> Result<()> {
    // Respect pages asking not to be followed.
    if meta.robots.as_deref().is_some_and(|robots| robots.contains("nofollow")) {
        return Ok(());
    }
    for node in nodes {
        if let Node::Product { name, sku } = node {
            println!("{name} ({sku:?})");
        }
    }
    if let Some(canonical) = meta.canonical {
        queue.follow(canonical).await?;
    }
    Ok(())
}

Both parse the page; a handler that also takes Html can call html.meta() and html.json_ld() instead, to parse it once.

Walking the tree yourself

Html wraps scraper’s document, so a handler can reach everything scraper offers: elements, nodes and CSS selectors. spire::extract::html::scraper re-exports scraper at the version spire uses, so no dependency of your own has to match it:

use spire::extract::html::Html;
use spire::extract::html::scraper::Selector;

async fn page(html: Html) {
    let headings = Selector::parse("h2").unwrap();
    for heading in html.0.select(&headings) {
        println!("{}", heading.text().collect::<String>());
    }
}

Storing results

Handlers write what they extract to datasets, one per item type. Register a dataset on the crawler, then write to it through the Data<T> extractor:

use spire::backend::ReqwestBackend;
use spire::dataset::{Data, InMemDataset};
use spire::extract::html::Html;
use spire::frontier::seed::Seed;
use spire::request::http::Uri;
use spire::{Crawler, Result, Router};

#[derive(Debug)]
struct Title {
    uri: String,
    title: String,
}

async fn page(uri: Uri, html: Html, titles: Data<Title>) -> Result<()> {
    let title = html.title().unwrap_or_default();
    titles.write(Title { uri: uri.to_string(), title }).await
}

async fn crawl() -> Result<()> {
let crawler = Crawler::new(ReqwestBackend::default(), Router::new().fallback(page))
    .with_dataset(InMemDataset::<Title>::new())
    .with_seed(Seed::get("https://example.com/")?);
crawler.run().await?;

for title in crawler.dataset::<Title>()?.read_all().await? {
    println!("{}: {}", title.uri, title.title);
}
Ok(())
}

A handler taking Data<T> for a type without a dataset fails. Collect<T> writes to the dataset for T too.

In memory

InMemDataset keeps items in memory. new and queue read them in the order they were written, and stack reads the most recent first. Reading removes them: read takes one, and read_all takes every item.

JSON Lines

JsonlDataset, with the json feature, appends every item to a file as one line of JSON, for items that implement Serialize:

use serde::Serialize;
use spire::dataset::JsonlDataset;

#[derive(Serialize)]
struct Title {
    uri: String,
    title: String,
}

async fn example() -> spire::Result<()> {
// `create` truncates the file, and `append` adds to it.
let titles = JsonlDataset::<Title>::create("titles.jsonl").await?;
Ok(())
}

Each item is written and flushed as it is stored, so it survives a crash of the crawl, though not of the machine. The file is locked while the dataset is open, so a second dataset on the same file, in this crawl or another, fails to open. A crawl resumed after a crash may crawl a page again, writing its items twice: deduplicate by a key such as the URL when reading the file.

CSV

CsvDataset, with the csv feature, appends every item to a CSV file as one row, after a header row of its field names:

use serde::Serialize;
use spire::dataset::CsvDataset;

#[derive(Serialize)]
struct Product {
    name: String,
    price: Option<f64>,
}

async fn example() -> spire::Result<()> {
// `create` truncates the file; `append` adds to it, with a header row only
// if it is empty.
let products = CsvDataset::<Product>::create("products.csv").await?;
Ok(())
}

Items must be flat: fields of strings, numbers, booleans, or Options of them. An item with a nested struct or a list fails to write. Like JSON Lines, rows are flushed as they are written, and a resumed crawl may write a page’s rows twice.

In the crawl’s database

With the redb feature, store.dataset::<T>(name) keeps items of type T in a table of the crawl’s own redb file, as JSON, for types that implement both Serialize and Deserialize. Items survive a crash with the rest of the crawl: see Crash safety with redb.

Reading in batches

Data::read_all takes every item, a batch at a time, and into_batches streams them, which suits large datasets. Datasets that can take several items at once do: a redb dataset takes a batch in one transaction instead of one per item. Writing works the same way: Collect writes a page’s items as one batch, which the file and redb datasets store all at once or not at all.

use futures::TryStreamExt;
use spire::dataset::Data;
struct Product;

async fn export(products: Data<Product>) -> spire::Result<()> {
let mut batches = std::pin::pin!(products.into_batches(1000));
while let Some(batch) = batches.try_next().await? {
    println!("{} products", batch.len());
}
Ok(())
}

Your own

Implement spire::dataset::Dataset<T> to store items anywhere else, such as a database: see Custom storage.

Following links

Pages lead to other pages. A handler queues the links it wants crawled, and the frontier decides whether to take them: see The frontier.

Queue adds requests to the crawl. Links resolve against the page they were found on, after redirects, and are one level deeper than it:

use spire::Result;
use spire::frontier::queue::{Link, Queue};
use spire::frontier::seed::Seed;

async fn page(queue: Queue) -> Result<()> {
    // A relative link, untagged: routed by its path.
    queue.follow("/about").await?;
    // A link with a tag.
    queue.follow(Link::new("/products/1").with_tag("product")).await?;
    // A whole seed, e.g. a POST request.
    let search = Seed::get("https://example.com/search?q=lamp")?.with_tag("search");
    queue.push(search).await?;
    Ok(())
}

follow and push return an Admission: whether the request was queued, or refused as a duplicate, too deep, outside the crawl’s scope, or not an http(s) link. Fragments are dropped, and follow_all follows several links at once.

The FollowLinks handler does this for a whole page: every link, or those matching a CSS selector, with the tag to give them, and without rel="nofollow" ones with skip_nofollow, see Routing.

Link::with_priority sets how soon a link is crawled, on a frontier that orders by priority, and FollowLinks scores the links it follows with score_page and score_links: see Best-first crawling.

The links of a page

The Links extractor, with the html feature, gives the links a visitor can follow, <a> and <area>, already resolved, each with its rel values and the text it shows. Filters narrow them, and they queue as they are:

use spire::Result;
use spire::extract::html::Links;
use spire::frontier::queue::{Link, Queue};

async fn page(links: Links, queue: Queue) -> Result<()> {
    for link in &links {
        println!("{} {:?} {}", link.uri(), link.rel(), link.text());
    }
    // The next page of a listing, on this site, if the site marks it.
    let next = links.clone().followable().same_host().with_rel("next");
    queue.follow_all(next).await?;
    // Every other link, tagged by its text.
    let products = links
        .into_iter()
        .filter(|link| link.text().contains("lamp"))
        .map(|link| Link::from(link).with_tag("product"));
    queue.follow_all(products).await?;
    Ok(())
}

• followable() leaves out rel="nofollow" links, same_host() links to other hosts, subdomains included, with_rel(value) keeps those with a rel value, and of_kind those from <a> or <area>.
• Links::within(&html, &uri, "article") reads only the links inside matching elements, e.g. an article without its navigation.
• Html::links(&uri) does the same for a page parsed by hand, e.g. a browser tab’s snapshot after interacting with it.
• Links that are not http(s) URLs, such as mailto: or javascript:, are left out, and fragments are dropped.

Where links resolve

A relative link resolves against the page it was found on, as a browser resolves it: against the page’s <base href>, if it has one, itself resolved against the URI the page was served from. When the request was redirected, that is the final URI: a link a on a page requested as /old/ and redirected to /new/ becomes /new/a. Head::uri, or ResponseExt::final_uri on a response, gives that URI. Links and FollowLinks resolve this way; a relative href passed to Queue::follow resolves against the final URI alone. A link is one level deeper than its page, which max_depth limits.

Queueing from outside a handler

Crawler::queue returns a queue for the crawl’s frontier, e.g. to add seeds while the crawl runs, from a task of your own:

use spire::backend::ReqwestBackend;
use spire::frontier::seed::Seed;
use spire::{Crawler, Result, Router};
async fn page() {}

async fn crawl() -> Result<()> {
let crawler = Crawler::new(ReqwestBackend::default(), Router::new().fallback(page));
let queue = crawler.queue();
queue.push(Seed::get("https://example.com/")?).await?;
crawler.run().await?;
Ok(())
}

Best-first crawling

A focused crawl reaches the pages most likely to matter first, by giving every link a priority: a score. It needs a frontier that orders by priority, InMemFrontier::priority() or a redb store’s, see The frontier. Scores come from the URL, in the frontier, and from the page, in the handler, and add up.

Scoring by URL

FrontierExt::score adds a scorer’s score to every seed pushed. A scorer is a Score, or a closure from &Seed to i32; Keywords counts keywords in the URL’s path and query, and ShallowFirst scores minus the depth. Stacked scorers add up:

use spire::frontier::score::{Keywords, ShallowFirst};
use spire::frontier::seed::Seed;
use spire::frontier::{FrontierExt, InMemFrontier};

let frontier = InMemFrontier::priority()
    .dedup()
    .score(Keywords::new(["laptop", "review"]).with_weight(10))
    .score(ShallowFirst)
    // Product pages over everything else.
    .score(|seed: &Seed| if seed.uri().path().starts_with("/p/") { 50 } else { 0 });

Scoring by page

A handler has the page a link was found on. FollowLinks scores the links it follows: score_page scores the page once, for every link on it, and score_links scores each link from its URI, text and rel. The two add up:

use spire::frontier::score::Keywords;
use spire::handler::FollowLinks;

let keywords = Keywords::new(["laptop", "review"]);
let page_keywords = keywords.clone();

let follow = FollowLinks::new("page")
    // Links on relevant pages first...
    .score_page(move |page| 10 * page_keywords.matches(&page.title().unwrap_or_default()))
    // ...then by their own text.
    .score_links(move |link| keywords.matches(link.text()));

Scoring in your own handler

When a link’s score needs more than the page, such as the router’s state or the request, a handler of your own scores the links: Html::anchors gives every link with its element, and Link::with_priority its priority.

use std::sync::Arc;

use spire::Result;
use spire::extract::State;
use spire::extract::html::{Html, Meta};
use spire::frontier::queue::{Link, Queue};
use spire::frontier::score::Keywords;

struct App {
    keywords: Keywords,
}

async fn page(html: Html, meta: Meta, State(app): State<Arc<App>>, queue: Queue) -> Result<()> {
    // Once per page: how relevant it is.
    let relevance = 10 * app.keywords.matches(&meta.title.unwrap_or_default());
    // Once per link: that, plus its own text.
    let links = html.anchors().map(|(href, anchor)| {
        let text: String = anchor.text().collect();
        Link::from(href).with_priority(relevance + app.keywords.matches(&text))
    });
    let links: Vec<Link> = links.collect();
    queue.follow_all(links).await?;
    Ok(())
}

How scores combine

A link’s priority is its base, which the frontier’s scorers add to. Seeds are scored once, when pushed: retries and revisits keep their priority. Scores only order a frontier that orders by priority; a FIFO or LIFO one ignores them.

Sitemaps

With the sitemap feature, the FollowSitemap handler follows sitemaps: every page is queued with one tag, and every sitemap of an index with the tag of the sitemap itself. Seed sitemaps yourself, or let Robots::with_sitemaps queue the ones each site’s robots.txt lists:

use spire::backend::ReqwestBackend;
use spire::crawler::Robots;
use spire::frontier::seed::Seed;
use spire::frontier::{FrontierExt, InMemFrontier};
use spire::handler::FollowSitemap;
use spire::request::Tags;
use spire::{Crawler, Result, Router};

#[derive(Debug, Clone, Copy, Tags)]
enum Page {
    Home,
    Sitemap,
    Product,
}

async fn product() {}

async fn crawl() -> Result<()> {
    let router = Router::new()
        .route(Page::Home, || async {})
        .route(Page::Sitemap, FollowSitemap::new(Page::Product))
        .route(Page::Product, product);

    let crawler = Crawler::new(ReqwestBackend::default(), router)
        .with_frontier(InMemFrontier::fifo().dedup().allow_hosts(["example.com"]))
        .with_robots(Robots::new("MyBot").with_sitemaps(Page::Sitemap))
        .with_seed(Seed::get("https://example.com/")?.with_tag(Page::Home));
    crawler.run().await?;
    Ok(())
}

The Sitemap extractor parses a sitemap for a handler of your own. Sitemaps must be fetched over HTTP: a browser renders XML as a page.

HTTP

With the reqwest feature, ReqwestBackend fetches pages over HTTP with reqwest. Every request of a crawl shares one connection pool.

use std::time::Duration;

use spire::backend::ReqwestBackend;
use spire::backend::reqwest::Client;

fn main() -> Result<(), Box<dyn std::error::Error>> {
// reqwest's defaults:
let backend = ReqwestBackend::default();

// Or a configuration of your own, e.g. with a user agent and a timeout.
let backend = ReqwestBackend::from_builder(|| {
    Client::builder()
        .user_agent("MyBot/1.0 (+https://example.com/bot)")
        .timeout(Duration::from_secs(30))
})?;
Ok(())
}

from_builder takes a function that makes the configuration, rather than a built client, because a crawl with identities builds a client per identity from it, dropped once the identity is gone.

spire::backend::reqwest re-exports reqwest at the version spire uses. Compression, cookies, SOCKS proxies and DNS through hickory are features of spire-reqwest, see Installation.

Error statuses

ReqwestBackend applies a StatusPolicy to every response, before its body is downloaded:

• StatusPolicy::Classify, the default, turns error statuses into errors, so the crawler retries or fails them without a handler checking: 408, 429 and 5xx are retried, 429 and 503 honouring Retry-After, and other 4xx are not. See Errors and retries.
• StatusPolicy::Accept hands every response to the handlers, whatever its status, e.g. to record 404s. The Head extractor reads the status.

use spire::backend::{ReqwestBackend, StatusPolicy};

let backend = ReqwestBackend::new().with_status_policy(StatusPolicy::Accept);

Failed transfers

A request that never gets a response fails with an error whose kind says whether trying again may help:

Redirects are followed by reqwest, and the response records the URI it was served from, which links resolve against: see Following links.

Proxies

To send requests through proxies, give the crawl identities: each one a proxy with its own user agent and cookies. ReqwestBackend builds a reqwest client per identity, from its configuration, with the identity’s proxy and, with the cookies feature, a cookie store of its own, and the crawl sets aside proxies that keep failing. A socks5 proxy needs the socks feature.

Without identities, reqwest takes proxies from the environment by default (HTTP_PROXY, HTTPS_PROXY, NO_PROXY), and a configuration of your own can route through any proxy reqwest supports, e.g. only some sites through one with Proxy::custom. reqwest picks such a proxy per connection, and does not notice one that keeps failing.

Browsers

ThirtyfourBackend drives real browsers over WebDriver BiDi, with the page’s real status and headers, any request method, and a fresh browser profile per request. The page is loaded once, by the first extractor that needs it, and body extractors such as Html read it as currently rendered. Tab is the request’s tab, to interact with the page; tab.snapshot() then parses it again, including what the interaction changed:

use spire::backend::ThirtyfourBackend;
use spire::backend::thirtyfour::By;
use spire::extract::{Head, Tab};
use spire::frontier::queue::Queue;
use spire::frontier::seed::Seed;
use spire::{Crawler, Result, Router};

async fn rendered(head: Head, queue: Queue, tab: Tab) -> Result<()> {
    if let Ok(more) = tab.find(By::Css("button.more")).await {
        let _ = more.click().await;
    }
    // The page as it is after the click.
    let page = tab.snapshot().await?;
    println!("{:?} at {}", page.title(), head.uri);
    queue.follow_all(page.links(&head.uri)).await?;
    Ok(())
}

async fn crawl() -> Result<()> {
    let backend = ThirtyfourBackend::builder()
        .remote("http://localhost:4444")
        .max_sessions(4)
        .build()?;

    let router = Router::new().route("page", rendered);
    let crawler = Crawler::new(backend, router)
        .with_seed(Seed::get("https://example.com/")?.with_tag("page"));
    crawler.run().await?;

    // Quit the browser sessions instead of dropping them.
    crawler.backend().close().await;
    Ok(())
}

Running a browser

The backend talks to a WebDriver server that supports WebDriver BiDi, as current chromedriver and geckodriver do. Start one, for example chromedriver --port=4444, and point the backend at it with remote. Several remote endpoints are used in turn. With the manager feature of spire-thirtyfour, ThirtyfourBuilder::managed downloads and starts the driver itself instead; the browser must be installed.

Sessions start headless Chrome by default. Other browsers, and browser flags, take capabilities:

use spire::backend::ThirtyfourBackend;
use spire::backend::thirtyfour::DesiredCapabilities;

let builder = ThirtyfourBackend::builder()
    .remote("http://localhost:4444")
    .capabilities(DesiredCapabilities::firefox());

Sessions

The backend keeps a pool of up to max_sessions browser sessions, and every request gets a tab in a fresh profile, so no cookies or storage carry over between requests. keep_session_state shares one profile instead, for example for a crawl that logs in first.

With identities, each identity gets a profile of its own in every browser it uses: a user context with its proxy, user agent and headers, which every request from it carries, the page’s scripts and images included. The profile is kept while the identity is in use, so its cookies and storage carry over between its requests, never to another identity’s. Proxies asking for credentials get those of the identity’s proxy URL. A browser cannot use a SOCKS proxy with credentials, and drivers without a proxy per profile fail such requests as Invalid.

An identity’s profile lives in one browser, so with several sessions, its requests in different browsers keep separate cookies.

Close the backend after the crawl with close, which quits the sessions before going on. A backend dropped without it quits them on background tasks, which may not finish if the program exits right after.

Responses

Navigation goes through WebDriver BiDi, so a response has the page’s real status and headers, after redirects, and error statuses are retried or rejected as over HTTP. The body is the rendered document serialized as HTML, so its content type is always text/html.

Sitemaps and robots.txt should be fetched over HTTP: a browser renders XML as a page. Robots::with_backend fetches robots.txt with an ReqwestBackend while the pages use the browser.

Middleware

Middleware runs around a router for every request: before it, to change the request or answer without fetching; around it, to check the response’s head, which is fetched once and shared with the handlers; and after it, to change the outcome. SetHeader sets a header on every request, and middleware::from_fn turns an async function into middleware. Headers that say who a request comes from, such as a user agent that should match a proxy, belong to an identity instead, see Identities. The middleware added last runs first:

use std::time::Duration;

use spire::backend::ReqwestBackend;
use spire::request::http::HeaderValue;
use spire::request::http::header::USER_AGENT;
use spire::worker::middleware::{self, Next, SetHeader};
use spire::worker::{Outcome, WorkerExt};
use spire::{Context, Crawler, Result, Router};

/// Retries pages answering with a captcha, before any handler runs.
async fn captcha<C: spire::backend::Client>(cx: Context<C>, next: Next<C>) -> Result<Outcome> {
    if cx.response().await?.headers().contains_key("x-captcha") {
        return Ok(Outcome::Retry { after: Some(Duration::from_secs(60)) });
    }
    Ok(next.run(cx).await)
}

async fn crawl() -> Result<()> {
    let router = Router::new()
        .route("page", || async {})
        .layer(middleware::from_fn(captcha))
        .layer(SetHeader::new(USER_AGENT, HeaderValue::from_static("MyBot/1.0")));

    Crawler::new(ReqwestBackend::default(), router).run().await?;
    Ok(())
}

Revalidate is middleware too, for conditional requests: see Recrawls.

Identities

A request reveals who appears to send it in several places at once: its IP, through a proxy; its user agent and the headers that go with it; and its cookies. Chosen separately, they disagree: one IP sends many user agents, and a session cookie set through one proxy is sent through another. An Identity keeps them together, one visitor with one proxy, its headers and its own cookies, and Crawler::with_identities sends every request as one.

use spire::Result;
use spire::identity::{Identities, Identity};
use spire::request::http::HeaderValue;
use spire::request::http::header::{ACCEPT_LANGUAGE, USER_AGENT};

fn main() -> Result<()> {
let identities = Identities::new([
    Identity::new("de")
        .with_proxy("http://proxy-de.example:8080")?
        .with_header(USER_AGENT, HeaderValue::from_static("MyBot/1.0 (de)"))
        .with_header(ACCEPT_LANGUAGE, HeaderValue::from_static("de-DE,de;q=0.9")),
    Identity::new("us")
        .with_proxy("http://proxy-us.example:8080")?
        .with_header(USER_AGENT, HeaderValue::from_static("MyBot/1.0 (us)")),
])
.sticky_per_host();
Ok(())
}

An identity’s headers are added to every request it sends, unless the request sets them itself. A proxy URL may carry credentials, as the gateway below does; they never show up in logs or Debug output.

Browser user agents

Enable user-agents (also included in full) to give an identity a uagent browser profile: a user agent and matching navigation headers, including Chromium client hints.

use spire::identity::{Browser, Identities, Identity, UserAgents};

let agents = UserAgents::new().browsers([Browser::Chrome, Browser::Firefox]);
let identities = Identities::from_fn(move |n| {
    Identity::new(format!("session-{n}"))
        .with_proxy(format!("http://user-session-{n}:secret@gate.example:7000"))
        .map(|identity| identity.with_profile(agents.pick(n)))
});

Profile and Platform are also re-exported from spire::identity. pick(n) selects the same profile for the same number, configuration and uagent version. Keep the lockfile when resuming a crawl: uagent capture updates can change the selected headers.

Profile headers are defaults: the identity’s explicit headers override them, and the request’s headers override both. Identity::headers() returns only explicit headers; Identity::profile() returns the separate profile. Profiles omit Accept-Encoding; reqwest advertises the encodings enabled by its decoders, unless you explicitly override that header.

ThirtyfourBackend::connect_as refuses an identity with a user-agent profile as Invalid before opening a session. A real browser supplies its own user agent and client hints, matching its installed version.

Many identities

Listing a thousand identities by hand does not scale. Identities::from_fn builds identity n on demand, the first time the crawl needs another one, up to max_active at once, so a large pool costs nothing until it is used:

use spire::Result;
use spire::identity::{Identities, Identity};

fn main() -> Result<()> {
// A rotating gateway: each session id is a different exit IP.
let gateway = Identities::from_fn(|n| {
    Identity::new(format!("session-{n}"))
        .with_proxy(format!("http://user-session-{n}:secret@gate.example:7000"))
})
.max_active(1000);

// A list of proxies, e.g. read from a file or a provider's API.
let proxies: Vec<String> = vec!["http://10.0.0.1:3128".into(), "http://10.0.0.2:3128".into()];
let count = proxies.len();
let listed = Identities::from_fn(move |n| {
    let proxy = &proxies[usize::try_from(n).unwrap_or_default() % proxies.len()];
    Identity::new(format!("proxy-{n}")).with_proxy(proxy.as_str())
})
.max_active(count);
Ok(())
}

Names should follow from n, as above, so a resumed crawl builds the same identities again. A function that fails, e.g. on a malformed proxy URL, fails the request that needed the identity.

Picking one per request

• in_turn(), the default, gives each request the next identity.
• at_random() picks one at random.
• sticky_per_host() keeps each host on one identity, as one visitor would be, for as long as that identity stays in use. A new host gets a new identity while fewer than max_active are in use.

A seed pinned with Seed::with_identity(name) always goes out as that identity, e.g. a logged-in account. A handler reads the identity its request went out as with RequestExt::identity.

Failing identities

A request counts against its identity when it gets no answer at all, a connection failure or a timeout, or an answer that refuses it: 403, 407 or 429. Any other answer resets the count. After max_failures in a row, 3 by default, the identity is taken out of use:

• a generated identity is retired, and replaced by the next n: a new IP on a gateway, the next proxy on a list;
• a listed identity is set aside for rest, one minute by default, and twice as long each time it fails again.

retire_after_requests and retire_after retire generated identities after a number of requests or an age, to spread a crawl over more IPs. When every identity is set aside, requests are retried once the first comes back.

robots.txt is fetched as an identity too, the one its host is kept on, so it does not go out from your own IP. A Robots with a backend of its own fetches without identities.

Backends

A backend connects each request as its identity, with Backend::connect_as: through the identity’s proxy, keeping each identity’s cookies and session apart. A backend that cannot route through a proxy refuses an identity that has one, failing its requests rather than sending them from your own IP: that is what connect_as does unless a backend overrides it.

ReqwestBackend overrides it with a reqwest client per identity, built from its configuration with the identity’s proxy and, with the cookies feature, a cookie store of its own, dropped once the identity is gone; see HTTP:

use std::time::Duration;

use spire::backend::ReqwestBackend;
use spire::backend::reqwest::Client;
use spire::identity::{Identities, Identity};
use spire::{Crawler, Result, Router};
async fn page() {}

async fn crawl() -> Result<()> {
let backend = ReqwestBackend::from_builder(|| Client::builder().timeout(Duration::from_secs(30)))?;
let identities = Identities::from_fn(|n| {
    Identity::new(format!("session-{n}"))
        .with_proxy(format!("http://user-session-{n}:secret@gate.example:7000"))
})
.max_active(100)
.sticky_per_host();

let crawler = Crawler::new(backend, Router::new().fallback(page)).with_identities(identities);
crawler.run().await?;
Ok(())
}

ThirtyfourBackend overrides it with a browser profile per identity, a user context with its proxy, user agent and headers, see Browsers.

MockSite overrides it too, to test crawls with identities: MockSite::sent lists each request with its identity and headers, and unreachable_through(name) makes an identity’s proxy refuse every connection.

What identities are not

Identities keep a visitor consistent; they are not a disguise. A user agent names a browser, but a request sent over reqwest’s TLS stack does not have that browser’s TLS fingerprint, and sites that check it will see the difference.

Profiles supply matching header values, but reqwest does not preserve the browser’s header order or emulate its TLS or HTTP/2 fingerprint. See issue #60. Profiles describe a direct page navigation; they do not reproduce a browser’s changing headers for every navigation or subresource request.

The frontier

The frontier holds the seeds waiting to be crawled. It decides their order, and facades on top of it decide which seeds are let in and what happens when one fails.

Seeds

A seed is a request waiting to be crawled:

use spire::frontier::seed::Seed;
use spire::request::http::{HeaderValue, Method};
use spire::request::http::header::ACCEPT;

fn main() -> spire::Result<()> {
let page = Seed::get("https://example.com/products")?
    .with_tag("listing")
    .with_priority(10);

let search = Seed::new(Method::POST, "https://example.com/search".parse().unwrap())?
    .with_header(ACCEPT, HeaderValue::from_static("application/json"))
    .with_body(r#"{"q": "lamp"}"#);
Ok(())
}

Seeds are deduplicated by their fingerprint: the method, the URI in a canonical form, and the body. The canonical form lowercases the scheme and host, drops default ports and the fragment, sorts the query, and removes tracking parameters such as utm_source, so HTTPS://Example.com:443/a?b=2&a=1&utm_source=x and https://example.com/a?a=1&b=2 are the same page.

Order

InMemFrontier keeps seeds in memory:

• InMemFrontier::fifo() crawls in the order seeds were queued, breadth first. This is the default.
• InMemFrontier::lifo() crawls the most recent first, depth first.
• InMemFrontier::priority() crawls the highest with_priority first, then in the order queued.

A redb store’s frontier orders by priority too, see Crash safety with redb.

Priorities can be scored for every link, so the crawl reaches the pages that matter first: see Best-first crawling.

Facades

FrontierExt adds facades, each wrapping the frontier before it:

use spire::frontier::{FrontierExt, InMemFrontier};

let frontier = InMemFrontier::fifo()
    .dedup()
    .allow_hosts(["example.com"])
    .filter(|seed| !seed.uri().path().ends_with(".pdf"))
    .max_depth(5)
    .retry(3)
    .tag_backoff();

A crawler without with_frontier uses InMemFrontier::fifo() with the default retries, and no deduplication. Add dedup to any crawl that follows links, or it will crawl pages again whenever they are linked twice.

Seen sets

dedup remembers every fingerprint it has seen, in memory. Long crawls have other options, through dedup_with:

use spire::frontier::seen::BloomSeen;
use spire::frontier::{FrontierExt, InMemFrontier};

// Ten million URLs in about 12 MB, skipping one new URL in a hundred.
let frontier = InMemFrontier::fifo().dedup_with(BloomSeen::new(10_000_000, 0.01));

A Bloom filter past its expected size makes more mistakes; it logs a warning once.

Stopping and resuming

A redb store keeps the frontier and the seen set across runs, crashes included: see Crash safety with redb.

Without it, a stopped crawl’s state can be saved and restored by hand. Crawler::snapshot copies the seeds the frontier still holds, including those waiting for a retry and those the stop cancelled, and InMemSeen and BloomSeen have snapshots of their own. With the serde feature, both serialize:

use std::sync::Arc;

use spire::backend::ReqwestBackend;
use spire::frontier::seen::InMemSeen;
use spire::frontier::{FrontierExt, InMemFrontier};
use spire::{Crawler, Result, Router};
async fn page() {}

async fn crawl() -> Result<()> {
let seen = Arc::new(InMemSeen::default());
let first = Crawler::new(ReqwestBackend::default(), Router::new().fallback(page))
    .with_frontier(InMemFrontier::fifo().dedup_with(Arc::clone(&seen)))
    .with_max_requests(1000);
first.run().await?;
let (seeds, fingerprints) = (first.snapshot().unwrap_or_default(), seen.snapshot());
// ... save both with serde, and load them in the next run ...

// Restored as is: the queue without facades, the seen set from its
// fingerprints.
let seen = Arc::new(fingerprints.into_iter().collect::<InMemSeen>());
let frontier = seeds.into_iter().collect::<InMemFrontier>().dedup_with(seen);
let next = Crawler::new(ReqwestBackend::default(), Router::new().fallback(page))
    .with_frontier(frontier);
next.run().await?;
Ok(())
}

Errors and retries

Every request ends with an Outcome, which the crawler reports to the frontier. Failures carry an Error, whose kind decides whether the request is tried again.

Outcomes

A handler returns anything that converts into an Outcome: () and Ok(()) mean the request is done, and an error fails it.

Helpers build the common ones: Outcome::retry(), Outcome::retry_after(delay), Outcome::backoff(duration) and Outcome::abort(), plus backoff_tag and abort_tag for another tag.

Where errors come from

• A handler returns an Error, through Result, or Outcome::Fail.
• An extractor that cannot extract fails the request before the handler runs, with its rejection: an Html of a page that cannot be fetched, a Json<T> of a body that is not a T. Wrapping it in Option or Result hands the handler None or the error instead.
• The backend fails a fetch: a transfer that broke, a timeout, or an error status under the default StatusPolicy, see HTTP.
• No route takes a request: it fails.

Kinds

Error::worker("...") is the usual way for a handler to fail; Error::new takes any kind, and with_source keeps the cause:

use spire::extract::Text;
use spire::{Error, ErrorKind, Result};

async fn page(Text(text): Text) -> Result<()> {
    if text.contains("Please try again later") {
        // Retried: the site is busy, not wrong.
        return Err(Error::new(ErrorKind::Http).with_message("soft error page"));
    }
    let count: u32 = text
        .trim()
        .parse()
        .map_err(|error| Error::worker("not a count").with_source(error))?;
    println!("{count}");
    Ok(())
}

Retries

retry(3) retries a failed request up to 3 times: after 500 ms, then doubling the delay up to 30 s, each delay shortened by up to 20% so requests failing together are not retried together. RetryPolicy changes those:

use std::time::Duration;

use spire::frontier::facade::RetryPolicy;
use spire::frontier::{FrontierExt, InMemFrontier};

let policy = RetryPolicy::new(5)
    .with_base_delay(Duration::from_secs(1))
    .with_max_delay(Duration::from_secs(60))
    .with_jitter(50);
let frontier = InMemFrontier::fifo().retry_with(policy);

Only errors that may succeed next time are retried: timeouts, connection errors, and 408, 429 and 5xx statuses. Other 4xx statuses, invalid requests and failed extractors are not. A handler can ask for a retry itself with Outcome::retry().

A request that fails for good, or runs out of retries, is counted in Stats::failed and logged as a warning; see Logs and statistics.

Pausing a tag

When a site answers 429 Too Many Requests or 503 Service Unavailable with a Retry-After header, the crawler pauses every request to that host for as long as it asks, then tries the request again; see Limits and politeness. A handler pauses a tag itself with Outcome::backoff(duration), which tag_backoff applies to every request with that tag, and Outcome::abort() skips every further request of its tag.

Limits and politeness

A crawler can easily overload a site, or run for longer than intended. Spire bounds both.

Concurrency

with_concurrency(n) runs up to n requests at once, 8 by default. On top of it, limits per host keep a single site from taking them all:

use std::time::Duration;

use spire::backend::ReqwestBackend;
use spire::{Crawler, Router};
async fn page() {}
let crawler = Crawler::new(ReqwestBackend::default(), Router::new().fallback(page))
    .with_concurrency(32)
    // At most 2 requests in flight to the same host,
    .with_host_concurrency(2)
    // and at least half a second between starting two of them.
    .with_host_delay(Duration::from_millis(500));

Hosts are told apart by their authority, so http:// and https:// of the same host share their limits. A request whose host is busy waits without holding up requests to other hosts.

Slowing down

A host is throttled when it answers 429 Too Many Requests or 503 Service Unavailable, or when a request to it times out. Every crawl slows a throttled host down, without being asked:

• A Retry-After header pauses the host for exactly as long as it asks: no request starts to it until then, whatever its tag, and it resumes at the same pace after.
• Once its concurrency cannot go lower, each round of throttling doubles the gap between its requests: from the delay’s minimum, or one second without one, up to a maximum of one minute. Each success halves it back.

with_host_delay takes the gap’s range. A Duration is its minimum, with the one-minute maximum; a range sets both ends, and the same value at both keeps the gap fixed:

use std::time::Duration;

use spire::backend::ReqwestBackend;
use spire::{Crawler, Router};
async fn page() {}
let crawler = Crawler::new(ReqwestBackend::default(), Router::new().fallback(page))
    // At least 200 ms between requests, up to 5 minutes for a struggling host.
    .with_host_delay(Duration::from_millis(200)..=Duration::from_secs(300));

Adaptive concurrency

A fixed limit is either too timid for a fast site or too much for a slow one. HostConcurrency::adaptive gives each host its own limit instead, up to a maximum, adapted to how it responds:

use spire::backend::ReqwestBackend;
use spire::crawler::HostConcurrency;
use spire::{Crawler, Router};
async fn page() {}

let crawler = Crawler::new(ReqwestBackend::default(), Router::new().fallback(page))
    .with_host_concurrency(HostConcurrency::adaptive(8));

• Each host starts at one request at a time. Its limit grows by one for every round of responses that come back about as fast as the host’s best: one more after one success at a limit of one, after two at two, and so on, up to the maximum.
• It halves when the host asks to slow down, with a 429 or a 503, when a request times out, or when responses take more than twice the host’s best time, which is how an overloaded server shows before it fails. It halves at most once per round, never below one. Other failures leave it alone.
• Response times are how long the fetch took, not the handlers, so slow parsing does not slow the crawl down.
• At a limit of one, a throttled host’s gap widens instead, as above. Recovering, its gap closes first, then its limit grows again.
• A robots.txt Crawl-delay raises a host’s minimum gap, whatever the limit allows.

robots.txt

With the robots feature, with_robots fetches robots.txt once per origin (scheme, host and port) before its first request, skips what it disallows for your user agent, and honours its Crawl-delay:

use std::time::Duration;

use spire::backend::ReqwestBackend;
use spire::crawler::Robots;
use spire::{Crawler, Router};
async fn page() {}

let robots = Robots::new("MyBot")
    // Kept for 6 hours instead of the default 24, then fetched again.
    .with_max_age(Duration::from_secs(6 * 60 * 60));
let crawler = Crawler::new(ReqwestBackend::default(), Router::new().fallback(page))
    .with_robots(robots);

Requests robots.txt disallows are reported as skipped. A missing robots.txt, answering a 4xx other than 429, allows everything. One that cannot be reached, answering a 5xx or 429 or failing to connect, disallows the origin while it is retried. Once its retries are spent, 3 by default and set with with_unreachable_retries, the origin’s requests are skipped. By default, robots.txt is fetched through the crawl’s own backend; with_backend fetches it through another, for example plain HTTP while the pages are crawled with a browser. with_sitemaps queues the sitemaps it lists, see Sitemaps.

The user agent is only matched against robots.txt. To send it with every request, set it on the backend, or with the SetHeader middleware.

Budgets

When the crawl stops, for a limit, a handler’s Outcome::Stop or a shutdown, nothing new starts and the requests in flight finish, unless a grace period cancels them first.

Shutting down

shutdown_token returns a token that stops the crawl when cancelled, for example on Ctrl-C:

use spire::backend::ReqwestBackend;
use spire::{Crawler, Result, Router};
async fn page() {}
async fn crawl() -> Result<()> {
let crawler = Crawler::new(ReqwestBackend::default(), Router::new().fallback(page));

let token = crawler.shutdown_token();
tokio::spawn(async move {
    tokio::signal::ctrl_c().await.ok();
    token.cancel();
});

let stats = crawler.run().await?;
Ok(())
}

How far a crawl got is in its statistics, see Logs and statistics.

Recrawls

A crawl usually runs once: every page is fetched, and the crawl ends when nothing is left. Keeping a dataset fresh takes two more things: crawling pages again on a schedule, and skipping the ones that did not change.

Revisiting pages

FrontierExt::revisit crawls pages again after an interval that depends on their tag. The intervals belong to the tag enum, through the Revisit trait, which #[derive(Revisit)] implements from a #[revisit("...")] attribute on each variant to revisit:

use std::time::Duration;

use spire::backend::ReqwestBackend;
use spire::frontier::seed::Seed;
use spire::frontier::{FrontierExt, InMemFrontier};
use spire::request::{Revisit, Tags};
use spire::{Crawler, Result, Router};

#[derive(Debug, Clone, Copy, Tags, Revisit)]
enum Page {
    #[revisit("1h")]
    Listing,
    #[revisit("1d")]
    Product,
    // No attribute: crawled once.
    About,
}
async fn listing() {}
async fn product() {}
async fn about() {}

async fn crawl() -> Result<()> {
let router = Router::new()
    .route(Page::Listing, listing)
    .route(Page::Product, product)
    .route(Page::About, about);
let crawler = Crawler::new(ReqwestBackend::default(), router)
    .with_frontier(InMemFrontier::fifo().revisit::<Page>().dedup())
    // A crawl that revisits never runs out of work: stop it with a time
    // limit or its shutdown token.
    .with_time_limit(Duration::from_hours(24))
    .with_seed(Seed::get("https://example.com/")?.with_tag(Page::Listing));
crawler.run().await?;
Ok(())
}

• A seed reported done or failed is crawled again its tag’s interval later, on its first attempt again: a failed page may come back. Untagged seeds, tags outside the enum, and seeds skipped on purpose are crawled once.
• While only revisits wait, the frontier waits for the next one, so the crawl keeps running until it is stopped.
• revisit goes directly on the storage, which keeps the schedule, one revisit per page, so revisits come out of it like queued seeds: facades above it apply to them, and deduplication does not refuse them, as they are never pushed. Anywhere else, it does not compile.
• Intervals combine d, h, m and s, such as "90s" or "1h30m", and are checked at compile time. An interval read at runtime, e.g. from a configuration file, takes a hand-written impl Revisit.

With a RedbStore frontier, revisits are kept in the store, so a crawl that restarts picks up its schedule where it stopped: store.frontier().revisit::<Page>().dedup_with(store.seen()). A crawl over the store without revisit crawls the revisits that are due, and does not wait for the others.

Skipping unchanged pages

The Revalidate middleware sends conditional requests: for a page it has seen, If-None-Match with its ETag, and If-Modified-Since with its Last-Modified. A server answering 304 Not Modified says the page did not change, so its handlers are skipped:

use spire::backend::ReqwestBackend;
use spire::frontier::FrontierExt;
use spire::redb::RedbStore;
use spire::worker::WorkerExt;
use spire::worker::middleware::Revalidate;
use spire::{Crawler, Result, Router};
async fn page() {}

async fn crawl() -> Result<()> {
let store = RedbStore::open("crawl.redb").await?;
let router = Router::new()
    .fallback(page)
    .layer(Revalidate::new(store.validators()));
let crawler = Crawler::new(ReqwestBackend::default(), router)
    .with_frontier(store.frontier().dedup_with(store.seen()));

let stats = crawler.run().await?;
println!("{} pages unchanged", stats.unchanged);
Ok(())
}

• Validators are stored once a page’s handlers continued, so a page whose extraction failed is fetched in full next time.
• InMemValidators keeps them for one run; store.validators() keeps them in the redb file, so later runs are conditional too.
• Only GET and HEAD requests are revalidated, and only over HTTP: browsers handle conditional requests themselves.
• Stats::unchanged counts the 304s.

Together, revisits decide when a page is crawled again, and revalidation makes that cheap when it did not change.

Crash safety with redb

With the redb feature, a RedbStore keeps a whole crawl in one file on disk: its queue, the URLs it has seen, what it scraped, the validators of the pages it fetched, and its scheduled revisits. Reopening the file resumes the crawl where it stopped, after a clean stop or a crash.

use serde::{Deserialize, Serialize};
use spire::backend::ReqwestBackend;
use spire::frontier::FrontierExt;
use spire::frontier::seed::Seed;
use spire::redb::RedbStore;
use spire::worker::WorkerExt;
use spire::worker::middleware::Revalidate;
use spire::{Crawler, Result, Router};

#[derive(Serialize, Deserialize)]
struct Product {
    name: String,
}
async fn page() {}

async fn crawl() -> Result<()> {
let store = RedbStore::open("crawl.redb").await?;
let router = Router::new()
    .fallback(page)
    .layer(Revalidate::new(store.validators()));
let crawler = Crawler::new(ReqwestBackend::default(), router)
    .with_frontier(store.frontier().dedup_with(store.seen()).retry(3))
    .with_dataset(store.dataset::<Product>("products"))
    // Refused as already seen when the crawl resumes.
    .with_seed(Seed::get("https://example.com/")?);
crawler.run().await?;
Ok(())
}

What survives a crash

Every change is a committed transaction, and nothing is lost:

• Requests in flight are crawled again. A popped seed stays recorded until its outcome is reported, so the requests running at a crash are queued again on reopening: a request may run twice, never zero times.
• No link is lost. A page’s links are committed before the page is reported done, so a crash either keeps them, or keeps the page in flight, to be crawled again and find them again.
• Items may be written twice. Items and the frontier are committed separately, so a page whose items were committed just before a crash may be crawled again and write them again: deduplicate by a key such as the URL when reading.
• Revisits are kept, one per page. A revisit is scheduled before its page is reported, so a crash in between crawls the page once more rather than losing its revisit.

Syncing

Commits reach the disk at most once a second. A crash rolls the store back to its last synced commit, a consistent earlier state, so it repeats at most a second of work. RedbStore::builder().sync_always() syncs every commit instead, repeating only the requests in flight, at a large cost in throughput: in the redb benchmark on macOS, about 0.6 ms per seed batched against 11 ms synced each time.

Your own tables

The store’s file is the crawl’s own. redb locks its file and lets one transaction write at a time, so an application’s own tables, e.g. a search index, belong in a database of their own, where they do not wait on the frontier’s every commit. spire::redb::redb re-exports redb at the version spire uses.

Logs and statistics

Logs

Spire logs through tracing, with one target per crate, so each can be turned up or down on its own:

At info and above, a crawl logs what an operator wants to see: the crawl starting and stopping, tags aborted, hosts paused for Retry-After, and robots.txt it could not fetch. Requests that fail for good are warn. Every request and every decision, such as a host’s concurrency or gap changing, is debug.

Install a subscriber to see them, e.g. with tracing-subscriber and its env-filter feature:

tracing_subscriber::fmt()
    .with_env_filter("spire_core=info,spire=info")
    .init();

Each request runs in a request span with its method, URI, tag, depth and attempt, so a subscriber that prints spans shows which request a line belongs to.

Statistics

run returns Stats once the crawl is over, and stats() returns them while it runs, e.g. to report progress from another task:

use std::sync::Arc;
use std::time::Duration;

use spire::backend::ReqwestBackend;
use spire::{Crawler, Result, Router};
async fn page() {}

async fn crawl() -> Result<()> {
let crawler = Arc::new(Crawler::new(ReqwestBackend::default(), Router::new().fallback(page)));

let progress = Arc::clone(&crawler);
tokio::spawn(async move {
    loop {
        tokio::time::sleep(Duration::from_secs(10)).await;
        let stats = progress.stats();
        println!("{} requests, {} failed", stats.requests, stats.failed);
    }
});

let stats = crawler.run().await?;
println!("done in {:?}", stats.elapsed);
Ok(())
}

Testing

With the testing feature, MockSite serves fixed pages by URL, so handlers can be tested without a network. Enable it for tests only:

[dev-dependencies]
spire = { version = "0.6.0", features = ["testing"] }

MockSite is a backend, so a whole crawl runs against it, and visit runs one request through a router without crawling what it queues, to check what the handlers did:

use spire::dataset::InMemDataset;
use spire::extract::html::Select;
use spire::frontier::seed::Seed;
use spire::testing::{MockPage, MockSite};
use spire::worker::Outcome;
use spire::handler::{Collect, FollowLinks};
use spire::{Result, Router};

#[derive(Debug, Select)]
#[select("article")]
struct Product {
    #[select("h2")]
    name: String,
}

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<()> {
    let site = MockSite::new().page(
        "https://shop.test/list",
        MockPage::html(r#"<article><h2>Lamp</h2></article><a class="next" href="?page=2">next</a>"#),
    );
    let router = Router::new().route("listing", (
        Collect::<Product>::new(),
        FollowLinks::new("listing").select("a.next"),
    ));

    let visit = site
        .visit(Seed::get("https://shop.test/list")?.with_tag("listing"))
        .with_dataset(InMemDataset::<Product>::new());
    assert!(matches!(visit.run(&router).await, Outcome::Continue));

    let products = visit.dataset::<Product>()?.read_all().await?;
    assert_eq!(products[0].name, "Lamp");
    assert_eq!(visit.queued()[0].uri(), "https://shop.test/list?page=2");
    Ok(())
}

Pages are html, text, json, xml, a status, or a redirect, which is followed as an HTTP client would. URLs without a page get a 404, and error statuses fail the fetch as they do over HTTP.

Behaving like a server

MockSite answers as a real server would, so the parts of a crawl that depend on the server can be tested too:

• Error statuses go through the same StatusPolicy as the real backends: a 404 is rejected, and a 429 or 503 with Retry-After is retried after it. with_status_policy(StatusPolicy::Accept) hands every status to the handlers instead.
• Redirects with MockPage::redirect are followed, and the final URI is set, so links resolve as they would over HTTP.
• Slow pages with MockPage::with_latency take that long to answer, e.g. to test timeouts or adaptive host concurrency. With #[tokio::test(start_paused = true)], the wait takes no real time.
• Conditional requests: a page with an ETag or Last-Modified header answers 304 Not Modified to a request that still matches it, e.g. to test Revalidate.

use std::time::Duration;

use spire::request::http::header::{ETAG, RETRY_AFTER};
use spire::request::http::{HeaderValue, StatusCode};
use spire::testing::{MockPage, MockSite};

let site = MockSite::new()
    .page(
        "https://shop.test/busy",
        MockPage::status(StatusCode::TOO_MANY_REQUESTS)
            .with_header(RETRY_AFTER, HeaderValue::from_static("5")),
    )
    .page(
        "https://shop.test/slow",
        MockPage::html("<p>slow</p>").with_latency(Duration::from_secs(10)),
    )
    .page(
        "https://shop.test/cached",
        MockPage::html("<p>v1</p>").with_header(ETAG, HeaderValue::from_static("\"v1\"")),
    );

Custom backends

A backend fetches pages. ReqwestBackend and ThirtyfourBackend are two; any other source of pages, such as a local mirror, an API with its own client, or a service that renders pages for you, is a backend of your own.

Backend and Client

A Backend is shared by the whole crawl and hands out a Client per request, from connect: an HTTP client can hand out a handle on one shared pool, a browser backend a session from its pool. The client then serves its one request:

• fetch(&request) performs the request and returns the response’s head: its status and headers. It may keep the body for later.
• body() returns the body, after fetch: the bytes received, or a browser page as currently rendered.

Both are called through the request’s Context, at most once for fetch: the first extractor that needs the response fetches it, and every other extractor shares it.

use std::path::PathBuf;
use std::sync::{Arc, Mutex};

use bytes::Bytes;
use spire::backend::{Backend, Client};
use spire::request::Request;
use spire::response::Response;
use spire::{Crawler, Error, ErrorKind, Result, Router};

/// Serves pages from a directory, e.g. a site mirrored to disk.
#[derive(Clone)]
struct Mirror {
    root: Arc<PathBuf>,
}

/// A request to a [`Mirror`], keeping the page it read.
struct MirrorPage {
    root: Arc<PathBuf>,
    body: Mutex<Option<Bytes>>,
}

impl Backend for Mirror {
    type Client = MirrorPage;

    async fn connect(&self) -> Result<MirrorPage> {
        Ok(MirrorPage {
            root: Arc::clone(&self.root),
            body: Mutex::new(None),
        })
    }
}

impl Client for MirrorPage {
    async fn fetch(&self, request: &Request) -> Result<Response<()>> {
        let path = self.root.join(request.uri().path().trim_start_matches('/'));
        let body = tokio::fs::read(&path).await.map_err(|error| {
            // Not retried: the file will still be missing.
            Error::new(ErrorKind::Rejected)
                .with_message(format!("no page at {}", path.display()))
                .with_source(error)
        })?;
        *self.body.lock().unwrap() = Some(Bytes::from(body));
        Ok(Response::new(()))
    }

    async fn body(&self) -> Result<Bytes> {
        let body = self.body.lock().unwrap().clone();
        body.ok_or_else(|| Error::worker("the page was not fetched yet"))
    }
}
async fn page() {}

async fn crawl() -> Result<()> {
let mirror = Mirror { root: Arc::new(PathBuf::from("./mirror")) };
let crawler = Crawler::new(mirror, Router::new().fallback(page));
crawler.run().await?;
Ok(())
}

Doing it as the built-in backends do

• Classify failures by whether trying again may help: Http, Timeout and Backend errors are retried, Rejected and Invalid are not; see Errors and retries.
• Apply a StatusPolicy to error statuses with StatusPolicy::check, so they fail the request as over HTTP, with Retry-After delays.
• Record redirects with ResponseExt::set_final_uri, so links resolve against the page’s real URI.
• Test it with handlers, as Testing does with MockSite, itself a backend serving pages from memory.

Custom storage

Every place a crawl keeps state is a trait, so it can live anywhere: a database, a queue service, or memory shared with something else.

Datasets

A dataset stores items with write and hands them back with read, first in, first out, removing them. A write-only sink returns None:

use spire::Result;
use spire::dataset::Dataset;

/// Prints every item instead of storing it.
struct Printed;

impl Dataset<String> for Printed {
    async fn write(&self, item: String) -> Result<()> {
        println!("{item}");
        Ok(())
    }

    async fn read(&self) -> Result<Option<String>> {
        Ok(None)
    }
}

read_batch and write_batch have defaults that go one item at a time; storage that can do several at once, such as in one transaction, overrides them, and should store a batch all or nothing.

Facades

A facade wraps a frontier to change what it accepts or hands out, and a FrontierLayer adds it to a chain with FrontierExt::layer:

use std::future::Future;

use spire::Result;
use spire::frontier::seed::Seed;
use spire::frontier::{Ack, Admission, Frontier, FrontierExt, FrontierLayer, InMemFrontier, Pop, Report};

/// Logs every seed handed out for crawling.
struct Logged<F>(F);

impl<F: Frontier> Frontier for Logged<F> {
    fn push(&self, seed: Seed) -> impl Future<Output = Result<Admission>> + Send {
        self.0.push(seed)
    }

    async fn pop(&self) -> Result<Pop> {
        let next = self.0.pop().await?;
        if let Pop::Ready(seed) = &next {
            println!("crawling {}", seed.uri());
        }
        Ok(next)
    }

    fn report(&self, report: Report) -> impl Future<Output = Result<Ack>> + Send {
        self.0.report(report)
    }

    fn size_hint(&self) -> Option<usize> {
        self.0.size_hint()
    }

    fn snapshot(&self) -> Option<Vec<Seed>> {
        self.0.snapshot()
    }
}

struct Logging;

impl<F: Frontier> FrontierLayer<F> for Logging {
    type Frontier = Logged<F>;

    fn layer(self, inner: F) -> Logged<F> {
        Logged(inner)
    }
}

let frontier = InMemFrontier::fifo().dedup().layer(Logging);

A facade passes on what it does not handle: report, size_hint and snapshot have defaults that refuse retries and hide the queue, so forward them to the inner frontier, as above.

Frontiers

A storage frontier implements push, pop and report. pop returns Pop::Ready with a seed, Pop::Wait until a time when none is ready yet, or Pop::Empty, which ends the crawl once nothing is in flight. report tells it how a popped seed ended: a persistent frontier keeps a seed recorded until then, so a crash does not lose it. Implementing Schedule as well lets revisits sit on it.

Seen sets and validators

A SeenSet has one method, insert, returning whether the fingerprint is new. Validators has get and set, by the fingerprint of the request. Both are shared by every request, so they take &self and synchronize inside.