CAR (Content Addressed aRchive) format support for Helia

This crate provides comprehensive functionality for working with CAR (Content Addressable aRchive) files, a format used to bundle IPFS blocks into a single archive for efficient transport, distribution, and storage.

What are CAR Files?

CAR files are container formats that package IPFS content-addressed data into a single archive. They include:

• A header with root CIDs and format version
• A sequence of blocks (CID + data pairs)
• Optional metadata about the contained content

CAR files are particularly useful for:

• Bulk data transfer: Moving large IPFS datasets between systems
• Content distribution: Sharing complete IPFS graphs via HTTP, S3, etc.
• Archival storage: Creating snapshots of IPFS data for backup
• Offline data exchange: Transporting content without network connectivity
• Dataset publishing: Distributing large datasets in academic/scientific contexts

When to Use This Crate

✅ Use CAR Files When You Need To:

• Export entire IPFS graphs: Bundle all blocks reachable from root CIDs
• Transfer data between systems: Move content between different IPFS nodes
• Create portable archives: Package content for distribution via CDN, HTTP, or file transfer
• Backup IPFS content: Create snapshots of important data graphs
• Seed content networks: Pre-load data into new IPFS nodes
• Publish static datasets: Distribute research data, media collections, or archives

❌ Don't Use CAR Files When:

• Real-time streaming is needed → Use direct IPFS retrieval or streaming protocols
• Random access to individual blocks is required → Use native blockstore operations
• Live collaboration on mutable data → Use IPNS or other mutable references
• Small single-block operations → Use direct get()/put() operations

CAR Format Versions

This crate currently supports CAR v1 format:

• Simple, sequential format: header followed by blocks
• Suitable for streaming and sequential access
• Widely supported across IPFS ecosystem

Future support planned for CAR v2:

• Includes index for random access
• Better for large archives requiring frequent lookups

Usage Examples

Example 1: Export Blocks to CAR File

use helia_car::{SimpleCar, Car, CarBlock, ExportOptions};
use bytes::Bytes;
use cid::Cid;
use tokio::fs::File;

// Create a CAR instance and add some blocks
let mut car = SimpleCar::new();
let cid = Cid::default();
let data = Bytes::from("Hello, IPFS!");
car.add_block(cid, data);

// Export to a file
let file = File::create("output.car").await?;
let roots = vec![cid];
let options = ExportOptions {
    max_blocks: Some(1000),
    recursive: true,
};

car.export(file, &roots, Some(options)).await?;

Example 2: Import Blocks from CAR File

use helia_car::{SimpleCar, Car, ImportOptions};
use tokio::fs::File;

let car = SimpleCar::new();
let file = File::open("input.car").await?;

let options = ImportOptions {
    max_blocks: Some(5000),
    verify_blocks: true,  // Verify block integrity
};

// Import blocks and get list of imported CIDs
let imported_cids = car.import(file, Some(options)).await?;
println!("Imported {} blocks", imported_cids.len());

Example 3: Stream CAR Export

use helia_car::{SimpleCar, Car, ExportOptions};
use futures::stream::StreamExt;
use bytes::Bytes;
use cid::Cid;

let mut car = SimpleCar::new();
let cid = Cid::default();
car.add_block(cid, Bytes::from("data"));

// Stream export for efficient memory usage
let roots = vec![cid];
let mut stream = car.export_stream(&roots, None);

while let Some(chunk) = stream.next().await {
    let bytes = chunk?;
    // Process or send chunk (e.g., HTTP response, write to file)
    println!("Chunk size: {} bytes", bytes.len());
}

Example 4: Get CAR Roots Without Full Import

use helia_car::{SimpleCar, Car};
use tokio::fs::File;

let car = SimpleCar::new();
let file = File::open("data.car").await?;

// Quickly inspect CAR file roots without importing all blocks
let roots = car.get_roots(file).await?;
println!("CAR contains {} root CIDs", roots.len());
for root in roots {
    println!("Root: {}", root);
}

Performance Characteristics

Where n = number of blocks in the CAR file.

Memory Efficiency Tips:

• Use streaming export/import for large datasets
• Set max_blocks limit to control memory usage
• Process blocks incrementally rather than loading entire CAR

Error Handling

All CAR operations return Result<T, HeliaError>:

use helia_car::{SimpleCar, Car};
use helia_interface::HeliaError;
use tokio::fs::File;

let car = SimpleCar::new();

match File::open("data.car").await {
    Ok(file) => {
        match car.import(file, None).await {
            Ok(cids) => println!("Success! Imported {} CIDs", cids.len()),
            Err(e) => eprintln!("Import failed: {}", e),
        }
    }
    Err(e) => eprintln!("Failed to open file: {}", e),
}

Common error scenarios:

• Invalid CAR format: Malformed header or block data
• I/O errors: File system errors during read/write
• Verification failures: Block data doesn't match CID (when verify_blocks = true)
• Resource limits: max_blocks limit exceeded

Comparison with Other IPFS Storage Methods

CAR v1 Format Specification

A CAR v1 file consists of:

+------------------+
| Header           |  ← DAG-CBOR encoded CarHeader
| - version: 1     |
| - roots: [CID]   |
+------------------+
| Block 1          |
| - length (varint)|
| - CID            |
| - data           |
+------------------+
| Block 2          |
| ...              |
+------------------+
| Block N          |
+------------------+

Each block is length-prefixed using unsigned varints for efficient streaming.

Feature Flags

This crate is designed to work seamlessly with the Helia ecosystem:

• Integrates with helia-interface traits
• Compatible with all Helia blockstore implementations
• Works with helia-unixfs for filesystem operations
• Supports async/await throughout

See Also

• SimpleCar - In-memory CAR implementation
• [Car] trait - Core CAR operations interface
• CarReader - Low-level CAR file reading
• CarWriter - Low-level CAR file writing
• CAR Specification - Official CAR format spec