Rusty Haystack

A high-performance Rust implementation of the Project Haystack specification for building automation and IoT data modeling.

Features

• Full Haystack 4 type system — all 15 scalar kinds (Marker, Number, Ref, DateTime, Coord, etc.) plus Dict, Grid, and List
• 6 codecs + RDF — Zinc, Trio, JSON (v4), JSON (v3), CSV, HBF (Haystack Binary Format) with content negotiation, plus RDF output (Turtle & JSON-LD)
• HBF binary codec — serde-based binary format with zstd compression, LEB128 varints, streaming encode support, and 23% faster federation queries vs Zinc text
• High-performance entity graph — in-memory graph with bitmap tag indexes, B-tree value indexes, CSR adjacency lists, LRU query cache, filter queries, ref traversal, reactive changelog, and concurrent read/write via SharedGraph
• Expression evaluator — parse and evaluate arithmetic expressions with variables, built-in functions (min, max, abs, sqrt, clamp), and unit-aware operations for computed points
• Unit conversion — Haystack unit database with quantity lookup, compatibility checking, and affine temperature conversions (~10.7M ops/sec)
• Graph traversal helpers — hierarchy tree building, entity classification, ref chain walking, parent/child queries, site resolution, and equipment point enumeration
• Haystack ontology — bundled ph, phScience, phIoT, phIct definitions with subtype checking and entity validation
• Xeto type system — spec parsing, structural fitting, slot resolution with inheritance, and library management (scan, load, validate)
• HLSS snapshots — graph serialization to Haystack Local Snapshot Store format with zstd compression and CRC32 integrity checks
• HTTP server — Actix Web 4 with 43 API endpoints, SCRAM SHA-256 authentication, WebSocket watches with deflate compression, streaming responses for large grids, and role-based access control
• Graph visualization API — 6 dedicated endpoints (graph/flow, graph/edges, graph/tree, graph/neighbors, graph/path, graph/stats) returning nodes + edges data optimized for React Flow and similar graph UI libraries
• Federation — hub-and-spoke entity aggregation from multiple remote Haystack servers with delta sync, adaptive sync intervals, write forwarding, history fan-out, watch federation, WebSocket-first transport, Arc-based zero-copy entity caching, and mTLS support
• HTTP/WS client — async client with SCRAM handshake, pluggable transport (HTTP + WebSocket), backpressure-aware WebSocket, HBF binary support, and mTLS
• CLI — import, export, serve, validate, info, libs, specs, client queries, user management, and federation config
• Python bindings — PyO3 0.28 module with full API parity: core types, codecs, graph, filter, ontology, client, server, federation, and auth (import rusty_haystack)
• Docker — multi-stage Alpine image (~15 MB) with a 5-container federation demo

Performance

See Benchmarks.md for full results on Apple M2.

Quick Start

Prerequisites

• Rust 1.93+ (edition 2024)
• cargo

Build

cargo build --workspace --exclude rusty-haystack

Run Tests

cargo test --workspace --exclude rusty-haystack
# ~1,320 tests across all crates

Start a Demo Server

cargo run -p rusty-haystack-cli -- serve --demo --port 8080

Then query it:

curl http://localhost:8080/api/about
curl -X POST http://localhost:8080/api/read \
  -H "Content-Type: text/zinc" \
  -d 'ver:"3.0"
filter
"site"'

Docker

docker build -t rusty-haystack .
docker run -p 8080:8080 rusty-haystack serve --demo --port 8080

Federation Demo

Run a 5-container federation cluster (1 lead + 4 building nodes):

cd demo
docker compose up --build

See demo/FederatedDemo.md for details.

Graph Visualization API

Six endpoints under /api/graph/* return entity relationship data structured for graph visualization UIs like React Flow:

The graph/flow and graph/neighbors endpoints return a nodes grid with an edges grid encoded in the metadata — one call gives a UI everything it needs to render. See Server API docs for full request/response specs.

Workspace Structure

Documentation

Security

• No unsafe code — entire codebase is safe Rust
• SCRAM SHA-256 authentication with PBKDF2 (100k iterations) credential storage
• Credential zeroization — zeroize crate clears salted passwords, client keys, and SCRAM state from memory on drop
• Username enumeration prevention — fake SCRAM challenges for unknown users
• Constant-time nonce comparison — prevents timing side-channels during auth
• Request limits — 2 MB body size, 100 concurrent watches, 10k IDs per watch, 1M history items
• WebSocket zip bomb protection — 10 MB decompressed message limit
• Filter recursion depth limit — max depth 100 to prevent stack overflow
• Expression evaluation limits — max depth 64, max nodes 1000 to prevent resource exhaustion
• Federation entity validation — max 1000 tags per entity, max 256-byte IDs, malformed entities rejected at sync
• HBF bounds checking — payload size limits and validated magic headers prevent binary format exploits
• Xeto loader safety — symlink traversal protection, file size limits, and directory depth guards
• Snapshot bomb guard — decompressed snapshot size limit prevents memory exhaustion
• Arithmetic overflow protection — checked operations throughout
• Sanitized error messages — internal details not leaked to clients
• Token rotation — configurable TTL with automatic expiration

Key Dependencies

What's New in v0.7.1

• HLSS v2 snapshot format — snapshot body codec switched from Zinc (text) to HBF (binary), same Zstd + CRC32 envelope
• 31% faster snapshot writes — HBF binary encode eliminates text formatting overhead (22.6ms → 16.3ms at 10K entities)
• 26% faster snapshot reads — HBF binary decode avoids text parsing (50.6ms → 37.3ms at 10K entities)
• Clean break — v1 (Zinc) snapshots are no longer readable; old snapshots must be recreated after upgrade

What's New in v0.7.0

• Roaring bitmap indexes — TagBitmapIndex migrated from hand-rolled Vec<u64> to RoaringBitmap with automatic compression and SIMD acceleration
• Delta indexing — update() re-indexes only changed tags, 3.4x faster entity updates (6.73µs → 1.96µs)
• Incremental CSR adjacency — patch buffer overlays base CSR, auto-compacts at 1,000 ops; avoids full rebuild on every mutation
• WL structural fingerprinting — Weisfeiler-Leman colour refinement produces per-entity structural fingerprints for topology-aware partitioning and anomaly detection
• ID freelist recycling — deleted entity IDs are reused in O(1), preventing unbounded ID growth
• Bulk load path — add_bulk() / finalize_bulk() skips per-entity changelog and indexing for 2.3x faster batch ingestion
• Configurable query cache — cache capacity auto-scales with entity count, clamped to [256, 1024]
• Changelog binary search — changes_since() uses partition_point() for O(log n) version lookup
• Entity ID safety — MAX_ENTITY_ID gate prevents overflow into roaring bitmap's u32 address space
• Structural DoS bounds — adaptive WL depth with hard entity caps (50K full WL, 200K tag-hash-only)

What's New in v0.6.0

• HBF (Haystack Binary Format) — serde-based binary codec with zstd compression, streaming encode, and 23% faster federation queries over HTTP
• Expression evaluator — parse and evaluate arithmetic expressions with variables and built-in functions for computed points
• Unit conversion engine — Haystack unit database with quantity-aware conversion and affine temperature transforms
• Graph traversal helpers — hierarchy trees, ref chain walking, entity classification, and site/equip/point navigation
• HLSS snapshots — graph persistence with zstd compression and CRC32 integrity
• Reactive changelog — 8,600x faster changes_since via version-indexed VecDeque
• Xeto library management — scan, load, and validate Xeto spec libraries with safety guards
• Security hardening — credential zeroization, entity validation, expression depth limits, symlink protection, snapshot bomb guards
• Streaming responses — batched row streaming for large grid responses (>25k rows)
• Arc entity caching — zero-copy federation entity sharing eliminates deep clones

License

MIT