tribev2-rs

TRIBE v2 — Multimodal fMRI Brain Encoding Model — Inference in Rust

Pure-Rust inference engine for TRIBE v2 (d'Ascoli et al., 2026), a deep multimodal brain encoding model that predicts fMRI brain responses to naturalistic stimuli (video, audio, text).

Same model, new runtime. tribev2-rs loads the exact same pretrained weights as facebook/tribev2 — no fine-tuning, no quantisation, no architectural changes. Every layer has been independently verified for numerical parity with the Python reference implementation.

Predicted cortical activity on the fsaverage5 surface (20,484 vertices), rendered from the pretrained TRIBE v2 model with multi-modal input.

Recent Additions

• End-to-end media pipeline (--video-path, --audio-path, --text-path) — raw media → brain predictions in one command
• GPU inference via --backend burn-gpu — 84× faster than pure-Rust CPU using wgpu Metal (3-backend parity: Pearson = 1.0)
• Volume-to-surface projection (--fmri-input) — load raw NIfTI fMRI and project to cortical surface
• NIfTI volume output (--nifti) — surface-to-volume projection with 6mm Gaussian smoothing
• HCP-MMP1 ROI analysis (--roi-summary, --roi-output) — per-region brain activation summaries
• Evaluation metrics (--ground-truth) — Pearson correlation, MSE, top-k retrieval vs ground-truth fMRI
• Per-modality contribution maps (--modality-maps) — ablation-based text/audio/video contribution
• Stimulus-aligned visualization (--stimulus-html) — HTML with video frames + brain activity + text aligned
• Text-to-speech — text input → TTS audio → whisperX → word events → predictions
• MP4 video output (--mp4) — animated brain activity over time via ffmpeg
• 8 numeric parity tests — verified identical to Python across all 3 backends

Workspace Structure

tribev2-rs/
├── crates/
│   ├── tribev2/              Core brain encoding model, CLI, features, plotting
│   ├── tribev2-audio/        Wav2Vec-BERT 2.0 audio feature extraction (burn)
│   └── tribev2-video/        V-JEPA2 ViT-G video feature extraction (burn)
├── scripts/
│   ├── extract_llama_features.py   True per-layer LLaMA extraction (HuggingFace)
│   ├── generate_parity_refs.py     Generate Python reference outputs for parity tests
│   └── generate_full_parity_refs.py  Extended references (metrics, ROI, correlation)
├── tests/
│   ├── full_parity.rs        8-test Python↔Rust numeric parity suite
│   └── burn_parity.rs        Cross-backend parity (CPU vs NdArray vs wgpu Metal)
└── data/
    ├── model.safetensors     Pretrained weights (from HuggingFace)
    ├── config.yaml           Model configuration
    ├── build_args.json       Feature dimensions, output shape
    ├── fsaverage5/           FreeSurfer cortical surface meshes
    └── parity_refs/          Python reference tensors for parity tests

Crate Overview

Features

• 100% inference parity with the Python implementation — every operation verified (8 parity tests)
• Two backends — pure-Rust (CPU) and burn (CPU/GPU via NdArray, wgpu Metal, Vulkan)
• Both backends load pretrained weights from safetensors
• Multi-modal inference — text, audio, and video features simultaneously
• Text feature extraction — LLaMA 3.2-3B via llama-cpp (Rust) or HuggingFace (Python script for true per-layer extraction)
• Audio feature extraction — Wav2Vec-BERT 2.0 in burn (16 kHz waveform → conformer hidden states)
• Video feature extraction — V-JEPA2 ViT-G in burn (frames → 3D patch embedding → ViT hidden states)
• Segment-based batching — long-form inference with configurable overlap
• Brain surface visualization — SVG rendering on fsaverage5 cortical mesh (6 views, 6 colormaps, colorbars, RGB overlays, MP4 time series)
• Events pipeline — whisperX transcription, ffmpeg audio extraction, sentence/context annotation
• HuggingFace Hub download support
• Rich output formats — binary f32, NIfTI (.nii.gz), SVG brain plots, MP4 video, JSON ROI summaries, per-modality contribution maps
• Evaluation metrics — Pearson correlation, MSE, top-k retrieval accuracy against ground-truth fMRI
• HCP-MMP1 ROI analysis — per-region summaries, top-k activated brain regions, wildcard ROI selection
• Subcortical structure analysis — Harvard-Oxford atlas labels for hippocampus, amygdala, thalamus, etc.
• Cross-resolution resampling — kd-tree interpolation between fsaverage3–6 meshes
• End-to-end media pipeline — video/audio/text → automatic feature extraction → predictions
• Text-to-speech — text input → TTS (gtts/macOS say/espeak) → transcription → features
• Volume-to-surface projection — load raw NIfTI fMRI and project to cortical surface (ball/line sampling)
• Stimulus-aligned HTML — video frames + brain activity + word annotations in scrollable timeline

Module Map (tribev2 crate)

Architecture

The model combines feature extractors — LLaMA 3.2 (text), V-JEPA2 (video), and Wav2Vec-BERT (audio) — into a unified x-transformers Encoder that maps multimodal representations onto the fsaverage5 cortical surface (~20,484 vertices).

Quick Start

1. Download weights

cargo run --bin tribev2-download --features hf-download -- \
  --repo eugenehp/tribev2 --output ./data

2. Run inference

# Text-only with LLaMA
cargo run --release --bin tribev2-infer -- \
  --config data/config.yaml \
  --weights data/model.safetensors \
  --llama-model path/to/llama-3.2-3b.gguf \
  --prompt "The quick brown fox jumps over the lazy dog"

# Multi-modal with pre-extracted features + brain plots
cargo run --release --bin tribev2-infer -- \
  --config data/config.yaml \
  --weights data/model.safetensors \
  --text-features text.bin \
  --audio-features audio.bin \
  --video-features video.bin \
  --n-timesteps 200 --segment \
  --plot-dir plots/ --view left --cmap coolwarm --colorbar

3. End-to-end media pipeline

# Video → extract audio → transcribe → extract features → brain predictions
cargo run --release --bin tribev2-infer -- \
  --config data/config.yaml --weights data/model.safetensors \
  --video-path clip.mp4 --llama-model llama-3.2-3b.gguf \
  --cache-dir ./cache --output predictions.bin \
  --stimulus-html brain_activity.html

# Audio-only (speech recording)
cargo run --release --bin tribev2-infer -- \
  --config data/config.yaml --weights data/model.safetensors \
  --audio-path speech.wav --llama-model llama-3.2-3b.gguf \
  --cache-dir ./cache --roi-summary 10

# Text-only (TTS → audio → transcribe → predict)
cargo run --release --bin tribev2-infer -- \
  --config data/config.yaml --weights data/model.safetensors \
  --text-path hamlet.txt --llama-model llama-3.2-3b.gguf \
  --cache-dir ./cache --plot-dir plots/

# Load raw fMRI NIfTI and project to surface
cargo run --release --bin tribev2-infer -- \
  --config data/config.yaml --weights data/model.safetensors \
  --fmri-input bold.nii.gz --subjects-dir data --vol-to-surf-radius 3.0

4. True per-layer LLaMA features (exact Python parity)

The llama-cpp backend extracts final-layer embeddings only. For true per-layer hidden states matching the Python pipeline:

# Extract with HuggingFace (requires: pip install transformers torch)
python scripts/extract_llama_features.py \
  --model meta-llama/Llama-3.2-3B \
  --input transcript.json \
  --output text_features.bin \
  --layers 0.5 0.75 1.0

# Use in Rust (auto-reads .json sidecar for shape metadata)
cargo run --release --bin tribev2-infer -- \
  --config data/config.yaml \
  --weights data/model.safetensors \
  --text-features text_features.bin

5. Library usage

use std::collections::BTreeMap;
use tribev2::model::tribe::TribeV2;
use tribev2::tensor::Tensor;

// Load pretrained model
let model = TribeV2::from_pretrained(
    "config.yaml", "model.safetensors", Some("build_args.json"),
)?;

// Build features: [1, n_layers*dim, timesteps]
let mut features = BTreeMap::new();
features.insert("text".into(),  Tensor::zeros(&[1, 9216, 100]));
features.insert("audio".into(), Tensor::zeros(&[1, 3072, 100]));
features.insert("video".into(), Tensor::zeros(&[1, 4224, 100]));

// Forward pass → [1, 20484, 100]
let output = model.forward(&features, None, true);

6. Burn backend (GPU inference)

use tribev2::config::{ModalityDims, TribeV2Config};
use tribev2::model_burn::tribe::TribeV2Burn;
use tribev2::model_burn::weights::{BurnWeightStore, load_burn_weights};

type B = burn::backend::NdArray;  // or burn::backend::Wgpu
let device = Default::default();

let config: TribeV2Config = serde_yaml::from_str(&std::fs::read_to_string("config.yaml")?)?;
let dims = ModalityDims::pretrained();

let mut model = TribeV2Burn::<B>::new(&dims, 20484, 100, &config.brain_model_config, &device);

// Load pretrained weights into burn model
let mut ws = BurnWeightStore::from_safetensors("model.safetensors")?;
load_burn_weights(&mut ws, &mut model, &device)?;

// Forward pass
let text  = burn::tensor::Tensor::<B, 3>::zeros([1, 9216, 100], &device);
let audio = burn::tensor::Tensor::<B, 3>::zeros([1, 3072, 100], &device);
let video = burn::tensor::Tensor::<B, 3>::zeros([1, 4224, 100], &device);

let output = model.forward(vec![("text", text), ("audio", audio), ("video", video)]);
// output: [1, 20484, 100]

Audio Feature Extraction (tribev2-audio)

use tribev2_audio::{Wav2VecBertConfig, Wav2VecBertWithConfig};
use tribev2_audio::audio_io::load_audio;
use tribev2_audio::weights::{WeightStore, load_wav2vec_bert_weights};

type B = burn::backend::NdArray;
let device = Default::default();
let config = Wav2VecBertConfig::default();  // facebook/w2v-bert-2.0

let mut model = Wav2VecBertWithConfig::<B>::new(&config, &device);

// Load HuggingFace weights
let mut ws = WeightStore::from_safetensors("w2v-bert-2.0/model.safetensors")?;
load_wav2vec_bert_weights(&mut ws, &mut model, &device)?;

// Extract features
let waveform = load_audio("audio.wav", 16000)?;
let features = model.extract_features(&waveform, 60.0, &device);
// features: [3, 1024, 120] at 2 Hz

Video Feature Extraction (tribev2-video)

use tribev2_video::{VJepa2Config, VJepa2WithConfig};
use tribev2_video::video_io;
use tribev2_video::weights::{WeightStore, load_vjepa2_weights};

type B = burn::backend::NdArray;
let device = Default::default();
let config = VJepa2Config::default();  // facebook/vjepa2-vitg-fpc64-256

let mut model = VJepa2WithConfig::<B>::new(&config, &device);

let mut ws = WeightStore::from_safetensors("vjepa2/model.safetensors")?;
load_vjepa2_weights(&mut ws, &mut model, &device)?;

// Extract frames and run model
// (see tribev2-video docs for full frame preprocessing pipeline)

Pretrained Model Details

Feature Flags

Benchmarks

Apple M4 Pro, 10 cores, 64 GB RAM. Full forward pass: 1152-d, 8-layer transformer, 20,484 outputs, T=20 input → 100 output timesteps, 3 modalities.

Forward Pass Only

Full Pipeline (forward + all output types)

*GPU forward is 773ms including CPU→GPU data transfer; 36ms warm with data on device.

Historical Benchmarks (T=100)

cargo run --release --example bench_burn
cargo run --release --example bench_burn --no-default-features --features wgpu-kernels-metal,llama-metal

Numeric Parity

Every output path is verified against the Python reference implementation using the real pretrained model (1152-d hidden, 8-layer transformer, 20,484 output vertices). Reference data is generated by scripts/generate_parity_refs.py and scripts/generate_full_parity_refs.py.

# Generate Python reference outputs (requires: pip install torch safetensors pyyaml numpy)
python3 scripts/generate_parity_refs.py
python3 scripts/generate_full_parity_refs.py

# Run all 8 parity tests
cargo test --release -p tribev2 --test full_parity -- --nocapture

All errors are within f32 accumulation noise through 8 transformer layers — functionally identical to Python.

Cross-Backend Parity

All three Rust backends produce identical results vs Python and vs each other. 0 out of 20,484 vertices fall below r=0.999 for any backend.

All output types (predictions, ROI summaries, correlation maps, metrics, top-k ranking) are identical across backends.

# Run cross-backend parity tests
cargo test --release -p tribev2 --test burn_parity -- --nocapture
cargo test --release -p tribev2 --test burn_parity \
  --no-default-features --features wgpu-metal,llama-metal -- --nocapture

Output Formats

Backend Selection

# Pure-Rust CPU (default — single-threaded, no dependencies)
cargo run --release --bin tribev2-infer -- --backend cpu ...

# Burn NdArray CPU (multi-threaded, ~10× faster)
cargo run --release --bin tribev2-infer -- --backend burn-cpu ...

# Burn wgpu Metal GPU (~84× faster forward pass, Apple Silicon)
cargo run --release --bin tribev2-infer --no-default-features \
  --features wgpu-metal,llama-metal -- --backend burn-gpu ...

# End-to-end: video → predictions (auto-extracts audio, transcribes, extracts features)
cargo run --release --bin tribev2-infer -- \
  --video-path clip.mp4 --llama-model llama.gguf --cache-dir ./cache \
  --backend burn-cpu --roi-summary 10 --stimulus-html brain.html ...

Example Outputs

All outputs below were generated from the pretrained model with 3-modality input (20 timesteps, 20,484 vertices). Full reproduction:

cargo run --release --bin tribev2-infer -- \
  --config data/config.yaml --weights data/model.safetensors --build-args data/build_args.json \
  --text-features examples/outputs/text_features.bin \
  --audio-features examples/outputs/audio_features.bin \
  --video-features examples/outputs/video_features.bin \
  --n-timesteps 20 --subjects-dir data \
  --output predictions.bin --roi-summary 20 --roi-output roi_summary.json \
  --plot-dir plots/ --cmap hot --colorbar \
  --modality-maps modality_maps/ \
  --ground-truth data/parity_refs/ground_truth.bin --correlation-map corr.bin

Brain Surface Plots

Predicted cortical activation rendered on the fsaverage5 mesh (left hemisphere, lateral view):

t=0	t=25	t=50	t=75	t=99

Multi-view overview (timestep 0):

Left	Right	Dorsal

Per-Modality Contribution Maps

Ablation-based contribution: for each modality, the difference in prediction when that modality is zeroed out.

Text	Audio	Video

Video contributes most strongly to occipital (visual) cortex, text to temporal/frontal language areas.

NIfTI Volume Output

Surface predictions projected into MNI152 volumetric space (96×96×96 voxels, 2mm isotropic):

Top row: axial, coronal, sagittal slices through the center of activity. Bottom row: axial slices at different z-levels. Coolwarm colormap shows predicted BOLD response (red = positive, blue = negative). The sparse pattern reflects the cortical surface vertices projected into volume space.

# Generate NIfTI output
cargo run --release --bin tribev2-infer -- \
  --config data/config.yaml --weights data/model.safetensors \
  --text-features text.bin --nifti predictions.nii.gz --nifti-dim 96

# View with any NIfTI viewer (FSLeyes, freeview, nibabel, etc.)
fsleyes predictions.nii.gz

Top-20 Activated Brain Regions (HCP-MMP1)

Rank   Region                      Activation
---------------------------------------------
1      a24                           0.075338
2      43                            0.071237
3      Pol1                          0.059974
4      LO2                           0.059593
5      FOP5                          0.057156
6      VIP                           0.056432
7      d32                           0.055918
8      IFSa                          0.053896
9      Ig                            0.051375
10     MIP                           0.051369
11     FOP4                          0.050351
12     IP2                           0.048767
13     9p                            0.048548
14     TE2a                          0.047815
15     PFm                           0.047319
16     23c                           0.045220
17     STSdp                         0.041179
18     IPS1                          0.041100
19     IFJa                          0.040690
20     2                             0.040539

Evaluation Metrics (vs synthetic ground truth)

Evaluation Metrics
=============================================
  Timesteps:          100
  Vertices:           20484
  Mean Pearson r:     0.926735
  Median Pearson r:   0.950039
  MSE:                0.000548
  Top-1 accuracy:    0.2000 (20.0%)

Output File Listing

examples/outputs/
├── roi_summary.json                  Per-ROI average activation (173 regions)
├── segments.json                     Segment metadata (100 timesteps)
├── nifti_slices.png                  NIfTI volume slice visualization
├── plots_selected/
│   ├── frame_0000.png – frame_0099.png  Per-timestep brain plots
│   └── overview_t0_{left,right,dorsal}.png  Multi-view overview
└── modality_maps/
    ├── text_contribution.png           Text modality contribution
    ├── audio_contribution.png          Audio modality contribution
    └── video_contribution.png          Video modality contribution

Citation

@article{dAscoli2026TribeV2,
  title={A foundation model of vision, audition, and language for in-silico neuroscience},
  author={d'Ascoli, St{\'e}phane and Rapin, J{\'e}r{\'e}my and Benchetrit, Yohann and
          Brookes, Teon and Begany, Katelyn and Raugel, Jos{\'e}phine and
          Banville, Hubert and King, Jean-R{\'e}mi},
  year={2026}
}

License