Getting Started

Prerequisites

• Rust 1.95.0 or newer (the repository MSRV)
• Git

Build from Source

git clone https://github.com/WyattAu/crawlkit.git
cd crawlkit
cargo build --release

Binary output: target/release/crawlkit

First Crawl

crawlkit crawl https://example.com --max-pages 50

With custom settings:

crawlkit crawl https://example.com \
  --max-pages 500 \
  --delay 200 \
  --concurrency 8 \
  --output crawl-data

Generate an HTML report:

crawlkit report crawl-data/ --format html --output report.html

Reproducible Crawls

crawlkit crawl https://example.com --seed 42 --output crawl-seed42
crawlkit crawl https://example.com --seed 42 --output crawl-seed42-v2

Both runs produce identical output given the same configuration and seed.

Encrypted Storage

CRAWLKIT_ENCRYPTION_KEY=$(openssl rand -hex 32) crawlkit crawl https://example.com --encrypt

Encryption uses AES-256-GCM. The key is provided via the CRAWLKIT_ENCRYPTION_KEY environment variable (base64 or hex-encoded).

Metrics Export

crawlkit crawl https://example.com --metrics-json metrics.json

Exports an atomic metrics snapshot (pages crawled, bytes fetched, fetch/analysis times, throughput) to a JSON file.

REST API Mode

crawlkit-api --port 8080

Default API key: ck_dev_default_key_for_testing (development only).

Security note: The default API key is for local development only. Generate a unique key via POST /api/v1/keys and restrict access in production.

Programmatic Usage

use crawlkit_core::{CrawlConfig, HttpClient, HtmlParser};
use crawlkit_core::analyzers::{AnalyzerRegistry, AnalysisContext};

let config = CrawlConfig::default();
let client = HttpClient::from_crawl_config(&config)?;
let registry = AnalyzerRegistry::new(&config);

let url = url::Url::parse("https://example.com")?;
let result = client.fetch(&url).await?;
let parsed = HtmlParser::parse(&result.body, &url)?;
let ctx = AnalysisContext {
    page: &parsed,
    status_code: Some(result.status_code),
    headers: &result.headers,
    response_time: Some(result.response_time),
    redirect_chain: &[],
    robots_txt: None,
};
let findings = registry.analyze(&ctx, &config);