chore: gungraun benchmarks by turbocrime · Pull Request #390 · tachyon-zcash/ragu

turbocrime · 2026-01-26T21:50:13Z

Summary

Add gungraun (iai-callgrind) benchmarks for deterministic, instruction-count-based performance tracking. This enables reliable CI regression detection by measuring CPU instructions instead of wall-clock time.

Closes #47. Supersedes #375.

Why gungraun/iai-callgrind?

Deterministic: Measures CPU instructions, not wall-clock time — no statistical variance
CI-friendly: Single run produces reliable results (no need for many samples)
Fine-grained: Small functions can be meaningfully benchmarked
Cross-platform: Docker support allows macOS developers to run benchmarks locally

Changes

CI Workflow (`.github/workflows/bench.yml`)

Runs benchmarks on every PR and push to main
Path filtering skips benchmarks when no Rust files changed
Baseline caching per-branch for regression comparison
Uses github-action-benchmark for PR comments and summary reports

Note: External PRs cannot receive benchmark comments due to GitHub token permissions.

Local Development

just bench — auto-detects platform (native Linux, Docker on macOS)
Graceful Docker shutdown with proper signal forwarding (no orphan containers)

Feature Flag

Uses unstable-test-fixtures feature to expose test utilities needed by benchmarks. This follows the project convention for features with potentially unstable API surface.

Metrics

Transforms gungraun JSON output via .github/scripts/transform-gungraun.jq. Reports 6 key metrics:

Instructions (Ir)
L1 hits, LL hits, RAM hits
Total read+write accesses
Estimated cycles

Benchmark Coverage

Crate	Benchmarks
ragu_pcd	application_build, seed, fuse, verify_leaf, verify_node, rerandomize
ragu_primitives	element ops (mul, invert, fold_8, is_zero, multiadd_8), point ops (double, add_incomplete, double_and_add_incomplete, endo), boolean (multipack_256), sponge (absorb_squeeze), endoscalar ops
ragu_circuits	polynomial commits (structured, unstructured), polynomial ops (revdot, fold, eval, dilate), synthesis (into_object, rx, ky, square), registry ops
ragu_arithmetic	MSM (64–4096 elements), FFT (k=10,14,18), ell (k=10,14), polynomial (with_roots, eval, factor), field (dot, geosum)

Review Feedback Addressed

Feedback	Resolution
Graceful Docker shutdown	Added signal trapping, `--init` flag
`StepRng` deprecated	Switched to `StdRng::seed_from_u64()`
Simplify workflow	Removed baseline generation fallback and apt caching
Feature naming	Renamed to `unstable-test-fixtures`

Future Work

More granular fuse instrumentation (Benchmark fuse pipeline #393)
Chained seed/fuse tree-building benchmarks (Benchmark tree building #394)

Test Plan

CI workflow runs successfully
just bench works on Linux (native) and macOS (via Docker)
just bench -- --save-summary=json generates summary files
~~Benchmark comment appears on internal PRs~~ needs to merge first

turbocrime · 2026-01-27T06:58:07Z

unfortunately, it's confirmed that external PRs can't activate the 'comment' feature because the github token provided to external PRs does not have the necessary permissions.

alxiong

Great work! 👏 very solid starting point for us.

Left some suggestions (open to chat), I will likely push one commit improving on justfile (open to revert if you dislike)

TalDerei

first pass observations

TalDerei · 2026-01-27T22:42:19Z

+#[cfg(any(test, feature = "gg-callgrind"))]
+#[doc(hidden)]
+pub mod test_fixtures;


unstable-test-fixtures #375 (review)

I think we should call it unstable-test-fixtures (sorry for the verbosity!) because I want us to get in the habit of labeling features that might have breaking API changes.

#375 (review)

i guess i don't understand here - what's 'unstable' about this? no production API surface should actually change. i just didn't want to default expose things that weren't for consumers (nobody wants to use MySimpleCircuit or see it in their autocomplete)

changed feature name to unstable-test-fixtures

TalDerei · 2026-01-27T22:49:01Z

#375 (comment) wasn't addressed; deferring to @ebfull on benchmarking granularity in this first pass.

TalDerei · 2026-01-27T22:58:19Z

+      - name: Missing baseline. Run baseline benchmarks
+        id: baseline-bench
+        if: github.event_name == 'pull_request' && !steps.check-baseline.outputs.exists
+        continue-on-error: true


continue-on-error: true can slip into silent benchmark failures?

this is necessary because it's possible that there will be no benchmark results or broken benchmarks in the base. continue-on-error is present so it may proceed with measuring the present branch and compare it to nothing.

removed fallback baseline generation

TalDerei · 2026-01-27T23:01:44Z

+      - name: Cache apt packages
+        uses: actions/cache@v4
+        with:
+          path: /var/cache/apt


I’d be in favor of starting with a simpler script that just works, with minimal surface area, and then iteratively layering in this kind of caching complexity.

removed apt cache behavior

ebfull

Main concern is that ragu_pasta should not be a dependency of any of the crates in this workspace. (dev-dependency only)

ebfull · 2026-01-29T08:27:59Z

          components: clippy
      - name: Run clippy
-        run: cargo clippy --workspace --lib --tests --benches --locked -- -D warnings
+        run: cargo clippy --workspace --lib --tests --benches --locked --features unstable-test-fixtures -- -D warnings


I wonder if we want to --all-features this?

switched to --all-features

ebfull · 2026-01-29T08:29:49Z

+    gadgets::{GadgetKind, Kind},
+    maybe::Maybe,
+};
+use ragu_pasta::Fp;


We can avoid ragu_circuits depending on ragu_pasta (strict requirement!) by making the fixtures below be generic over the field type.

now parameterized.

dependency moved to dev dependency.

created issue suggesting cargo-deny to enforce this strict requirement #402

ebfull · 2026-01-29T08:30:02Z

+    pub times: usize,
+}
+
+impl Circuit<Fp> for SquareCircuit {


Suggested change

impl Circuit<Fp> for SquareCircuit {

impl<F: Field> Circuit<F> for SquareCircuit {

now parameterized

ebfull · 2026-01-29T08:30:27Z

 ff = { workspace = true }
 group = { workspace = true }
 ragu_core = { path = "../ragu_core", version = "0.0.0" }
+ragu_pasta = { path = "../ragu_pasta", version = "0.0.0", features = ["baked"] }


This breaks a dependency requirement of mine; we definitely don't want ragu_circuits to depend on ragu_pasta because then Ragu is not agnostic to the curve cycle.

dependency moved to dev dependency.

created issue suggesting cargo-deny to enforce this strict requirement #402

ebfull · 2026-01-29T08:32:46Z

+// ============================================================================
+// Test fixtures for registration_errors tests
+// ============================================================================


non-blocking nit: I have a general principle, which is that whenever I see "sectioned" comments in code like this I tend to think it means the file is too large and should be broken up into separate files/modules.

the original main version already had these (local) fixtures and the tests collected in a single file, the section header is really just diff noise.

some of this is now reverted - tests are back in their original location as integration tests rather than unit tests (this also helped address the dependency issue). the set of fixtures is now minimized and lives in a dedicated module.

ebfull · 2026-01-29T08:37:07Z

+
+[[package]]
+name = "serde_json"
+version = "1.0.148"


any reason we can't use serde_json 1.0.149?

now serde_json 1.0.149

ebfull · 2026-01-29T08:37:34Z

 [[package]]
 name = "zerocopy"
-version = "0.8.25"
+version = "0.8.31"


any reason we can't use zerocopy 0.8.33?

now zerocopy 0.8.33

ebfull · 2026-01-29T08:37:58Z

+
+[[package]]
+name = "zmij"
+version = "1.0.9"


any reason we can't use zmij 1.0.14?

now zmij 1.0.14

ebfull

Excellent, glad we're going to start having these in the codebase.

turbocrime requested a review from ebfull as a code owner January 26, 2026 21:50

turbocrime mentioned this pull request Jan 26, 2026

chore: initial benchmarks #375

Closed

turbocrime requested review from TalDerei and alxiong January 26, 2026 21:50

turbocrime force-pushed the iai-bench-ci branch 3 times, most recently from 3f9ddbb to c342f0d Compare January 27, 2026 02:23

alxiong requested changes Jan 27, 2026

View reviewed changes

turbocrime force-pushed the iai-bench-ci branch 2 times, most recently from b3cd23c to c2a3636 Compare January 27, 2026 11:32

TalDerei reviewed Jan 27, 2026

View reviewed changes

turbocrime force-pushed the iai-bench-ci branch 3 times, most recently from 1f62d07 to 11ec59d Compare January 28, 2026 00:35

turbocrime requested a review from alxiong January 28, 2026 01:06

turbocrime force-pushed the iai-bench-ci branch from 78546eb to 58a4ff3 Compare January 28, 2026 03:46

ebfull self-assigned this Jan 28, 2026

TalDerei unassigned ebfull Jan 28, 2026

TalDerei added the sean-unread Queued for Sean's review label Jan 28, 2026

alxiong approved these changes Jan 29, 2026

View reviewed changes

ebfull requested changes Jan 29, 2026

View reviewed changes

ebfull removed the sean-unread Queued for Sean's review label Jan 29, 2026

turbocrime and others added 3 commits January 29, 2026 11:43

initial iai benches

b34724e

address review comments

636ce04

review changes

3211cad

turbocrime force-pushed the iai-bench-ci branch from 94c802e to 3211cad Compare January 29, 2026 21:38

turbocrime mentioned this pull request Jan 29, 2026

Consider using proc-macro-error2 in ragu_macros for richer error diagnostics #401

Open

turbocrime requested a review from ebfull January 29, 2026 22:15

ebfull approved these changes Jan 30, 2026

View reviewed changes

ebfull merged commit 16c2af9 into tachyon-zcash:main Jan 30, 2026
11 checks passed

TalDerei mentioned this pull request Feb 5, 2026

Integrate performance regression tracking into CI pipeline #248

Closed

3 tasks

	impl Circuit<Fp> for SquareCircuit {
	impl<F: Field> Circuit<F> for SquareCircuit {

Conversation

turbocrime commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why gungraun/iai-callgrind?

Changes

CI Workflow (.github/workflows/bench.yml)

Local Development

Feature Flag

Metrics

Benchmark Coverage

Review Feedback Addressed

Future Work

Test Plan

Uh oh!

turbocrime commented Jan 27, 2026

Uh oh!

alxiong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TalDerei left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TalDerei Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TalDerei Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebfull left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

turbocrime Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

turbocrime commented Jan 26, 2026 •

edited

Loading

CI Workflow (`.github/workflows/bench.yml`)

TalDerei Jan 27, 2026 •

edited

Loading

TalDerei Jan 27, 2026 •

edited

Loading

turbocrime Jan 29, 2026 •

edited

Loading

turbocrime Jan 29, 2026 •

edited

Loading

turbocrime Jan 29, 2026 •

edited

Loading

turbocrime Jan 29, 2026 •

edited

Loading