Architecture
Crate Structure
Section titled “Crate Structure”crawlkit-core/ Core library: types, analyzers, storage, export, HTTP clientcrawlkit/ CLI binary: clap-based argument parsing, progress displaycrawlkit-api/ REST API: axum-based server with API key auth and rate limitingData Flow
Section titled “Data Flow”URL Queue -> DNS Prefetch -> Fetcher -> Parser -> Analyzers -> Storage -> Export ^ | +---- Link Discovery ---------+Key Components
Section titled “Key Components”Rate Limiter
Section titled “Rate Limiter”Token-bucket per domain with a global bucket. Concurrency semaphore. Mutex guards are dropped before await points for async safety.
Circuit Breaker
Section titled “Circuit Breaker”Three-state finite state machine: Closed -> Open -> HalfOpen. Configurable failure threshold, success threshold, and cooldown period. Per-domain fault isolation prevents cascading failures.
Backpressure
Section titled “Backpressure”Bounded pipeline with configurable channel capacity. Semaphore-based concurrency control. Producers block when the bounded channel is full, preventing memory exhaustion under high concurrency.
Analyzer Registry
Section titled “Analyzer Registry”28 analyzers dispatched in parallel. Each implements the Analyzer trait with analyze(&AnalysisContext, &CrawlConfig) -> Vec<Finding>.
Integrated Modules
Section titled “Integrated Modules”Observability
Section titled “Observability”Atomic metrics counters (AtomicU64) for pages crawled, bytes fetched, fetch time, analysis time, and throughput. Exported via --metrics-json.
Resource Monitor
Section titled “Resource Monitor”Enforces configurable limits on memory (RSS), CPU, disk usage, and page count. Replaces ad-hoc RSS checking with structured threshold enforcement.
Circuit Breaker
Section titled “Circuit Breaker”Per-domain fault isolation. Opens circuit after N consecutive failures, skips the domain until cooldown expires, then transitions to HalfOpen for a trial request.
Backpressure
Section titled “Backpressure”Bounded mpsc channels with configurable capacity. Prevents memory explosion when producer rate exceeds consumer rate.
Determinism
Section titled “Determinism”Seed-based reproducible crawls via --seed. Controls PRNG state for URL ordering, content hashing, and user-agent rotation. Identical seed + configuration yields identical output.
Feature Flags
Section titled “Feature Flags”Runtime feature toggles via TOML [features] section. CLI flags (--enable-ai, --enable-wasm, --javascript) override configuration file values.
JS Render Decision
Section titled “JS Render Decision”Heuristic SPA detection engine. Identifies React, Vue, Angular, Next.js, Nuxt, and SvelteKit pages by inspecting HTML markers and script patterns.
Playwright Integration
Section titled “Playwright Integration”JavaScript rendering for pages identified by the JS render decision engine. Requires --javascript flag and a Playwright installation. Falls back to static HTML when unavailable.
Alert Manager
Section titled “Alert Manager”Threshold-based monitoring. Evaluates crawl metrics against configured thresholds and triggers webhook notifications when exceeded.
Encryption at Rest
Section titled “Encryption at Rest”AES-256-GCM encryption for stored crawl data. Key sourced from CRAWLKIT_ENCRYPTION_KEY environment variable, configuration file, or system keyring (priority order).
Thread Safety
Section titled “Thread Safety”| Component | Synchronization Mechanism |
|---|---|
| RateLimiter | DashMap (domain buckets), parking_lot::Mutex (global, dropped before await) |
| UrlQueue | DashSet (dedup), parking_lot::Mutex (heap) |
| Storage | rusqlite::Connection (single-writer, WAL mode) |
| Metrics | AtomicU64 counters (lock-free) |
| AuditTrail | parking_lot::RwLock<Vec<AuditEvent>> |