Uses Criterion for statistical rigor. Covers HTML parsing, analyzer execution, registry lookup, queue operations, and storage insert/query performance.
| Metric |
Target |
Measured |
| Pages/sec |
>= 50 |
50–100 |
| Memory (10k pages) |
< 500 MB |
~200 MB |
| Startup time |
< 100 ms |
~10 ms |
| Core binary size |
< 10 MB |
2.46 MiB stripped |
| Full binary size |
Reported separately |
24.39 MiB stripped |
| Full analyzer suite |
– |
~25 us/page |
| HTML parse (5 KB) |
– |
~45 us |
| PageRank (1K nodes) |
– |
~4 ms |
| Benchmark |
What It Measures |
| HTML parser |
Parse time for varying document sizes |
| Analyzer suite |
Per-analyzer execution latency |
| Registry lookup |
Analyzer dispatch overhead |
| Queue ops |
Push/pop/dedup throughput |
| Storage |
SQLite insert and query latency |
| Technique |
Description |
| Async HTTP/2 |
Concurrent fetching with connection pooling via reqwest |
| Token-bucket rate limiting |
Per-domain request throttling with DashMap buckets |
| LRU caching |
DNS resolution cache with TTL eviction |
| SQLite WAL mode |
Concurrent read/write access with write-ahead logging |
| SIMD-accelerated parsing |
HTML parsing via scraper crate (uses selectors with SIMD) |
| LTO + single codegen unit |
Profile-guided optimization in release builds |
