Crate helia_car 

CAR (Content Addressed aRchive) format support for Helia

This crate provides comprehensive functionality for working with CAR (Content Addressable aRchive) files, a format used to bundle IPFS blocks into a single archive for efficient transport, distribution, and storage.

What are CAR Files?

CAR files are container formats that package IPFS content-addressed data into a single archive. They include:

• A header with root CIDs and format version
• A sequence of blocks (CID + data pairs)
• Optional metadata about the contained content

CAR files are particularly useful for:

• Bulk data transfer: Moving large IPFS datasets between systems
• Content distribution: Sharing complete IPFS graphs via HTTP, S3, etc.
• Archival storage: Creating snapshots of IPFS data for backup
• Offline data exchange: Transporting content without network connectivity
• Dataset publishing: Distributing large datasets in academic/scientific contexts

When to Use This Crate

✅ Use CAR Files When You Need To:

• Export entire IPFS graphs: Bundle all blocks reachable from root CIDs
• Transfer data between systems: Move content between different IPFS nodes
• Create portable archives: Package content for distribution via CDN, HTTP, or file transfer
• Backup IPFS content: Create snapshots of important data graphs
• Seed content networks: Pre-load data into new IPFS nodes
• Publish static datasets: Distribute research data, media collections, or archives

❌ Don’t Use CAR Files When:

• Real-time streaming is needed → Use direct IPFS retrieval or streaming protocols
• Random access to individual blocks is required → Use native blockstore operations
• Live collaboration on mutable data → Use IPNS or other mutable references
• Small single-block operations → Use direct get()/put() operations

CAR Format Versions

This crate currently supports CAR v1 format:

• Simple, sequential format: header followed by blocks
• Suitable for streaming and sequential access
• Widely supported across IPFS ecosystem

Future support planned for CAR v2:

• Includes index for random access
• Better for large archives requiring frequent lookups

Usage Examples

Example 1: Export Blocks to CAR File

use helia_car::{SimpleCar, Car, CarBlock, ExportOptions};
use bytes::Bytes;
use cid::Cid;
use tokio::fs::File;

// Create a CAR instance and add some blocks
let mut car = SimpleCar::new();
let cid = Cid::default();
let data = Bytes::from("Hello, IPFS!");
car.add_block(cid, data);

// Export to a file
let file = File::create("output.car").await?;
let roots = vec![cid];
let options = ExportOptions {
    max_blocks: Some(1000),
    recursive: true,
};

car.export(file, &roots, Some(options)).await?;

Example 2: Import Blocks from CAR File

use helia_car::{SimpleCar, Car, ImportOptions};
use tokio::fs::File;

let car = SimpleCar::new();
let file = File::open("input.car").await?;

let options = ImportOptions {
    max_blocks: Some(5000),
    verify_blocks: true,  // Verify block integrity
};

// Import blocks and get list of imported CIDs
let imported_cids = car.import(file, Some(options)).await?;
println!("Imported {} blocks", imported_cids.len());

Example 3: Stream CAR Export

use helia_car::{SimpleCar, Car, ExportOptions};
use futures::stream::StreamExt;
use bytes::Bytes;
use cid::Cid;

let mut car = SimpleCar::new();
let cid = Cid::default();
car.add_block(cid, Bytes::from("data"));

// Stream export for efficient memory usage
let roots = vec![cid];
let mut stream = car.export_stream(&roots, None);

while let Some(chunk) = stream.next().await {
    let bytes = chunk?;
    // Process or send chunk (e.g., HTTP response, write to file)
    println!("Chunk size: {} bytes", bytes.len());
}

Example 4: Get CAR Roots Without Full Import

use helia_car::{SimpleCar, Car};
use tokio::fs::File;

let car = SimpleCar::new();
let file = File::open("data.car").await?;

// Quickly inspect CAR file roots without importing all blocks
let roots = car.get_roots(file).await?;
println!("CAR contains {} root CIDs", roots.len());
for root in roots {
    println!("Root: {}", root);
}

Performance Characteristics

Operation	Time Complexity	Memory Usage	Notes
Export	O(n)	O(block_size)	Streams blocks sequentially
Import	O(n)	O(block_size)	Processes blocks one at a time
Stream Export	O(n)	O(chunk_size)	Most memory-efficient option
Get Roots	O(1)	O(header_size)	Only reads header

Where n = number of blocks in the CAR file.

Memory Efficiency Tips:

• Use streaming export/import for large datasets
• Set max_blocks limit to control memory usage
• Process blocks incrementally rather than loading entire CAR

Error Handling

All CAR operations return Result<T, HeliaError>:

use helia_car::{SimpleCar, Car};
use helia_interface::HeliaError;
use tokio::fs::File;

let car = SimpleCar::new();

match File::open("data.car").await {
    Ok(file) => {
        match car.import(file, None).await {
            Ok(cids) => println!("Success! Imported {} CIDs", cids.len()),
            Err(e) => eprintln!("Import failed: {}", e),
        }
    }
    Err(e) => eprintln!("Failed to open file: {}", e),
}

Common error scenarios:

• Invalid CAR format: Malformed header or block data
• I/O errors: File system errors during read/write
• Verification failures: Block data doesn’t match CID (when verify_blocks = true)
• Resource limits: max_blocks limit exceeded

Comparison with Other IPFS Storage Methods

Feature	CAR Files	Direct Blockstore	IPFS Gateway
Portability	✅ Excellent	❌ Low	⚠️ Requires network
Bulk Transfer	✅ Optimized	❌ Inefficient	⚠️ Network-dependent
Random Access	❌ Sequential only	✅ Instant	⚠️ Network latency
Storage Efficiency	✅ Compact	✅ Native	N/A
Offline Use	✅ Full support	✅ Local only	❌ Requires network
Streaming	✅ Native support	⚠️ Manual	✅ HTTP streaming

CAR v1 Format Specification

A CAR v1 file consists of:

+------------------+
| Header           |  ← DAG-CBOR encoded CarHeader
| - version: 1     |
| - roots: [CID]   |
+------------------+
| Block 1          |
| - length (varint)|
| - CID            |
| - data           |
+------------------+
| Block 2          |
| ...              |
+------------------+
| Block N          |
+------------------+

Each block is length-prefixed using unsigned varints for efficient streaming.

Feature Flags

This crate is designed to work seamlessly with the Helia ecosystem:

• Integrates with helia-interface traits
• Compatible with all Helia blockstore implementations
• Works with helia-unixfs for filesystem operations
• Supports async/await throughout

See Also

• SimpleCar - In-memory CAR implementation
• Car trait - Core CAR operations interface
• CarReader - Low-level CAR file reading
• CarWriter - Low-level CAR file writing
• CAR Specification - Official CAR format spec

Structs

CarBlock

A CAR (Content Addressed aRchive) file block

CarHeader

CAR file header

CarReader

Reader for CAR (Content Addressed aRchive) v1 files

CarWriter

Writer for CAR (Content Addressed aRchive) v1 files

ExportOptions

Options for exporting CAR files

ImportOptions

Options for importing CAR files

SimpleCar

Simple in-memory implementation of CAR operations

Enums

CarVersion

CAR format version

Traits

Car

Trait for CAR file operations

Functions

create_car

Create a new CAR instance with the given blocks

create_car_from_blocks

Utility function to create a CAR file from blocks

Type Aliases

Result

Result type alias for this crate