Crawl Engine
Async HTTP/2 with configurable concurrency. 50+ pages/sec throughput. Redirect chain tracking, robots.txt compliance, token-bucket rate limiting.
crawlkit
Async HTTP/2 crawling engine with 28 analyzers. Supply chain auditing. Zero clippy warnings.
Core Capabilities
Section titled “Core Capabilities”28 Analyzers
SEO meta tags, structured data validation, content quality scoring, security headers, accessibility (WCAG 2.1 AA), AI search optimization, WASM pattern detection.
Storage and Export
SQLite with WAL mode. Export to JSON, CSV, Markdown, HTML (single-file reports), SQLite.
Enterprise
RBAC, multi-tenant isolation, SSO integration (OIDC/SAML), SLA monitoring, audit trail, AES-256-GCM encryption at rest, circuit breaker fault tolerance.