seglog

A simple, high-performance segment log implementation for Rust.

seglog provides low-level read and write operations for fixed-size segment files with built-in CRC32C validation. It's designed for event sourcing systems, write-ahead logs, and other append-only storage use cases.

Features

• Single writer, multiple concurrent readers - Lock-free reads with atomic offset coordination
• CRC32C validation - Automatic data integrity checking on every read
• Optimized I/O - Reduces syscalls with optimistic reads (~40% faster for small records)
• Fixed-size segments - Pre-allocated files with configurable headers
• Corruption recovery - Detects and recovers from partial writes
• Zero-copy reads - Returns borrowed data when possible

Quick Example

use seglog::write::Writer;
use seglog::read::{Reader, ReadHint};

// Create a 1MB segment
let mut writer = Writer::create("segment.log", 1024 * 1024, 0)?;

// Append records
let (offset, _) = writer.append(b"event data")?;
writer.sync()?; // Flush to disk

// Read concurrently
let flushed = writer.flushed_offset();
let mut reader = Reader::open("segment.log", Some(flushed))?;
let data = reader.read_record(offset, ReadHint::Random)?;

assert_eq!(&*data, b"event data");

Record Format

Each record consists of an 8-byte header followed by variable-length data:

┌─────────────┬─────────────┬────────────────┐
│ Length (4B) │ CRC32C (4B) │ Data (N bytes) │
└─────────────┴─────────────┴────────────────┘

Performance Optimizations

Read Hints

• ReadHint::Sequential - Uses 64KB read-ahead buffer for streaming access
• ReadHint::Random - Optimistic reads (header + 2KB) to reduce syscalls

Optimistic Reads

For random access, the reader performs a single syscall to read the header plus 2KB of data. Since most events in event sourcing are small (< 2KB), this eliminates one syscall per read, improving performance by ~40%.

Header Support

Reserve space at the beginning of segments for application-specific headers:

const HEADER_SIZE: u64 = 64;
let mut writer = Writer::create("segment.log", 1024 * 1024, HEADER_SIZE)?;

// Write header data
writer.file().write_all_at(b"MAGIC", 0)?;

// Records automatically start after header
writer.append(b"data")?;

Concurrent Safety

The FlushedOffset provides atomic coordination between writers and readers:

• Writers update it after calling sync()
• Readers check it to avoid reading uncommitted data
• Shared via Arc for efficient cloning across threads

This ensures readers never see partial writes or corrupted data.

Use Cases

• Event sourcing - Store domain events in append-only segments
• Write-ahead logs - Durable transaction logging
• Message queues - Persistent queue segments
• Time-series data - Append-only metric storage

License

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.