Errors and retries

Every request ends with an Outcome, which the crawler reports to the frontier. Failures carry an Error, whose kind decides whether the request is tried again.

Outcomes

A handler returns anything that converts into an Outcome: () and Ok(()) mean the request is done, and an error fails it.

Helpers build the common ones: Outcome::retry(), Outcome::retry_after(delay), Outcome::backoff(duration) and Outcome::abort(), plus backoff_tag and abort_tag for another tag.

Where errors come from

• A handler returns an Error, through Result, or Outcome::Fail.
• An extractor that cannot extract fails the request before the handler runs, with its rejection: an Html of a page that cannot be fetched, a Json<T> of a body that is not a T. Wrapping it in Option or Result hands the handler None or the error instead.
• The backend fails a fetch: a transfer that broke, a timeout, or an error status under the default StatusPolicy, see HTTP.
• No route takes a request: it fails.

Kinds

Error::worker("...") is the usual way for a handler to fail; Error::new takes any kind, and with_source keeps the cause:

use spire::extract::Text;
use spire::{Error, ErrorKind, Result};

async fn page(Text(text): Text) -> Result<()> {
    if text.contains("Please try again later") {
        // Retried: the site is busy, not wrong.
        return Err(Error::new(ErrorKind::Http).with_message("soft error page"));
    }
    let count: u32 = text
        .trim()
        .parse()
        .map_err(|error| Error::worker("not a count").with_source(error))?;
    println!("{count}");
    Ok(())
}

Retries

retry(3) retries a failed request up to 3 times: after 500 ms, then doubling the delay up to 30 s, each delay shortened by up to 20% so requests failing together are not retried together. RetryPolicy changes those:

use std::time::Duration;

use spire::frontier::facade::RetryPolicy;
use spire::frontier::{FrontierExt, InMemFrontier};

let policy = RetryPolicy::new(5)
    .with_base_delay(Duration::from_secs(1))
    .with_max_delay(Duration::from_secs(60))
    .with_jitter(50);
let frontier = InMemFrontier::fifo().retry_with(policy);

Only errors that may succeed next time are retried: timeouts, connection errors, and 408, 429 and 5xx statuses. Other 4xx statuses, invalid requests and failed extractors are not. A handler can ask for a retry itself with Outcome::retry().

A request that fails for good, or runs out of retries, is counted in Stats::failed and logged as a warning; see Logs and statistics.

Pausing a tag

When a site answers 429 Too Many Requests or 503 Service Unavailable with a Retry-After header, the crawler pauses every request to that host for as long as it asks, then tries the request again; see Limits and politeness. A handler pauses a tag itself with Outcome::backoff(duration), which tag_backoff applies to every request with that tag, and Outcome::abort() skips every further request of its tag.