Configuration

Configuration File

Create crawlkit.toml in the working directory:

[crawl]
seed_urls = ["https://example.com"]
max_pages = 200
max_depth = 10
max_redirect_hops = 20
max_concurrent_requests = 64
request_timeout_secs = 30
crawl_delay_default_ms = 1000
user_agent = "crawlkit/0.4.0"
respect_robots_txt = true

[crawl.scope]
allowed_domains = ["example.com"]
blocked_patterns = ["/wp-admin/*", "/api/*"]

[analyzers]
enabled = ["meta", "links", "security", "accessibility", "content", "ai", "wasm"]

[output]
formats = ["json", "sqlite", "html"]
output_dir = "./crawl-results"

Feature Flags

Feature flags are configured in the [features] section of crawlkit.toml:

[features]
js_rendering = false
ai_analyzers = true
wasm_analyzers = true
backlink_analysis = true
encryption_at_rest = false
audit_trail = true
observability = true
rum_integration = false

CLI flags --enable-ai, --enable-wasm, and --javascript override configuration file values.

CLI Flags

All configuration file options can be overridden via CLI flags:

crawlkit crawl https://example.com \
  --max-pages 500 \
  --delay 200 \
  --concurrency 8 \
  --output crawl-data

Environment Variables

Variable	Description	Default
CRAWLKIT_DB_PATH	SQLite database path (API mode)	crawlkit-api.db
CRAWLKIT_ENCRYPTION_KEY	AES-256-GCM key (hex or base64)	none
RUST_LOG	Log level filter	crawlkit=info
NODE_PATH	Node.js module path (Playwright)	system default