The frontier

The frontier holds the seeds waiting to be crawled. It decides their order, and facades on top of it decide which seeds are let in and what happens when one fails.

Seeds

A seed is a request waiting to be crawled:

use spire::frontier::seed::Seed;
use spire::http::http::{HeaderValue, Method};
use spire::http::http::header::ACCEPT;

fn main() -> spire::Result<()> {
let page = Seed::get("https://example.com/products")?
    .with_tag("listing")
    .with_priority(10);

let search = Seed::new(Method::POST, "https://example.com/search".parse().unwrap())?
    .with_header(ACCEPT, HeaderValue::from_static("application/json"))
    .with_body(r#"{"q": "lamp"}"#);
Ok(())
}

Seeds are deduplicated by their fingerprint: the method, the URI in a canonical form, and the body. The canonical form lowercases the scheme and host, drops default ports and the fragment, sorts the query, and removes tracking parameters such as utm_source, so HTTPS://Example.com:443/a?b=2&a=1&utm_source=x and https://example.com/a?a=1&b=2 are the same page.

Order

InMemFrontier keeps seeds in memory:

• InMemFrontier::fifo() crawls in the order seeds were queued, breadth first. This is the default.
• InMemFrontier::lifo() crawls the most recent first, depth first.
• InMemFrontier::priority() crawls the highest with_priority first, then in the order queued.

A redb store’s frontier orders by priority too, see Crash safety with redb.

Priorities can be scored for every link, so the crawl reaches the pages that matter first: see Best-first crawling.

Facades

FrontierExt adds facades, each wrapping the frontier before it:

use spire::frontier::{FrontierExt, InMemFrontier};

let frontier = InMemFrontier::fifo()
    .dedup()
    .allow_hosts(["example.com"])
    .filter(|seed| !seed.uri().path().ends_with(".pdf"))
    .max_depth(5)
    .retry(3)
    .tag_backoff();

A crawler without with_frontier uses InMemFrontier::fifo() with the default retries, and no deduplication. Add dedup to any crawl that follows links, or it will crawl pages again whenever they are linked twice.

Seen sets

dedup remembers every fingerprint it has seen, in memory. Long crawls have other options, through dedup_with:

use spire::frontier::seen::BloomSeen;
use spire::frontier::{FrontierExt, InMemFrontier};

// Ten million URLs in about 12 MB, skipping one new URL in a hundred.
let frontier = InMemFrontier::fifo().dedup_with(BloomSeen::new(10_000_000, 0.01));

A Bloom filter past its expected size makes more mistakes; it logs a warning once.

Stopping and resuming

A redb store keeps the frontier and the seen set across runs, crashes included: see Crash safety with redb.

Without it, a stopped crawl’s state can be saved and restored by hand. Crawler::snapshot copies the seeds the frontier still holds, including those waiting for a retry and those the stop cancelled, and InMemSeen and BloomSeen have snapshots of their own. With the serde feature, both serialize:

use std::sync::Arc;

use spire::backend::ReqwestBackend;
use spire::frontier::seen::InMemSeen;
use spire::frontier::{FrontierExt, InMemFrontier};
use spire::{Crawler, Result, Router};
async fn page() {}

async fn crawl() -> Result<()> {
let seen = Arc::new(InMemSeen::default());
let first = Crawler::new(ReqwestBackend::default(), Router::new().fallback(page))
    .with_frontier(InMemFrontier::fifo().dedup_with(Arc::clone(&seen)))
    .with_max_requests(1000);
first.run().await?;
let (seeds, fingerprints) = (first.snapshot().unwrap_or_default(), seen.snapshot());
// ... save both with serde, and load them in the next run ...

// Restored as is: the queue without facades, the seen set from its
// fingerprints.
let seen = Arc::new(fingerprints.into_iter().collect::<InMemSeen>());
let frontier = seeds.into_iter().collect::<InMemFrontier>().dedup_with(seen);
let next = Crawler::new(ReqwestBackend::default(), Router::new().fallback(page))
    .with_frontier(frontier);
next.run().await?;
Ok(())
}

Avoiding traps

An opt-in avoid_traps() facade bounds expanding URL branches. For how it works alongside hidden-link filtering, see Avoiding traps. This section describes its policy and admission behavior.

use std::sync::Arc;
use spire::frontier::{FrontierExt, InMemFrontier};
use spire::frontier::facade::TrapPolicy;

let policy = TrapPolicy::default()
    .with_max_urls(10_000)
    .with_max_fanout(1_000)
    .with_max_random_urls(usize::MAX); // This site's paths contain opaque IDs.
let traps = Arc::new(InMemFrontier::fifo()
    .dedup()
    .retry(3)
    .avoid_traps_with(policy));
let frontier = traps.clone().allow_hosts(["example.com"]);
// Keep the inner handle to inspect refusals after handing frontier to the crawler.
assert_eq!(traps.refused(), 0);

A prefix is an origin and first nonempty path segment: https://example.com/archive/ is independent of /products/, other hosts, ports and schemes. Opaque first segments share the origin’s root prefix so changing that segment cannot bypass the opaque-URL budget. Defaults are:

Opaque-looking means at least 16 ASCII alphanumeric characters, mixing letters and digits, with at least eight distinct characters. Real IDs can match. Adjust TrapPolicy for the site’s structure; use usize::MAX to disable a check. Date archives and ordinary pagination are not opaque, but can still reach explicit budgets. Fan-out is estimated from URL directories, not the number of links found on one page. Different query variants share the same directory budget. These are candidate budgets: inner facades may subsequently refuse a candidate, and it still counts; offering the same request fingerprint again does not count twice.

Crossing a limit blocks that prefix for the facade’s lifetime. Further pushes return its refusal reason. Already-queued seeds from it are skipped when popped and reported inwards as Report::Skipped; in-flight requests can finish. Other prefixes continue. AvoidTraps::refused() counts refused pushes and skipped queued seeds, separately from crawler request totals. Tracing logs identify the blocked prefix and the reason.

Place this facade outside retry/revisit facades to check what they pop, and inside scope filters to avoid tracking out-of-scope candidates. Keep dedup() too: the facade’s observations do not replace deduplication. Counters and blocked-prefix state are in memory and reset when the facade is recreated. Snapshots omit known blocked seeds but do not persist the heuristic state; pending seeds in persistent storage are skipped when popped after their prefix becomes blocked.