Skip to content

SpanCache background flush errors are unhandled, crashing Eval() when the temp file cannot be written #2803

Open

Description

What goes wrong

Eval() enables SpanCache by default for experiment-backed evaluations. Every span write calls SpanCache.queueWrite(), which starts this.flushWriteBuffer() in the background and stores the promise on this.flushPromise without a .catch() handler (js/src/span-cache.ts). Nothing awaits or catches flushPromise during Eval().

When the temporary cache file cannot be opened or appended to (for example, TMPDIR pointing at a missing/stale directory, a read-only container filesystem without /tmp, or ENOSPC / EIO), flushWriteBuffer() rejects (ensureInitialized() and fileHandle.appendFile()). Under Node's default --unhandled-rejections=throw, the optional disk cache terminates the entire Eval() process with ENOENT / EIO.

In addition, ensureInitialized() caches the rejected this.initPromise, so every subsequent write rejects immediately as well.

Reproduction

Add js/src/span-cache-flush-failure.test.ts and run cd js && pnpm exec vp test run src/span-cache-flush-failure.test.ts:

import { afterEach, beforeAll, describe, expect, test } from "vite-plus/test";
import { SpanCache } from "./span-cache";
import { configureNode } from "./node/config";

describe("SpanCache background flush failures", () => {
  const unhandled: unknown[] = [];
  const onUnhandled = (reason: unknown) => unhandled.push(reason);
  const realTmpdir = process.env.TMPDIR;
  let cache: SpanCache | undefined;

  beforeAll(() => configureNode());

  afterEach(() => {
    process.off("unhandledRejection", onUnhandled);
    if (realTmpdir === undefined) delete process.env.TMPDIR;
    else process.env.TMPDIR = realTmpdir;
    cache?.stop();
    cache?.dispose();
    cache = undefined;
    unhandled.length = 0;
  });

  test("an unwritable temp dir does not cause an unhandled rejection", async () => {
    process.on("unhandledRejection", onUnhandled);
    process.env.TMPDIR = "/nonexistent-braintrust-tmp-dir";

    cache = new SpanCache();
    cache.start();
    cache.queueWrite("root-1", "span-1", { span_id: "span-1", input: "x" });
    await new Promise((resolve) => setTimeout(resolve, 100));

    expect(unhandled).toEqual([]);
  });
});

Running an experiment-backed Eval() / runEvaluator in a standalone Node process with TMPDIR=/nonexistent-bt-tmp exits 1 before the evaluation returns:

Error: ENOENT: no such file or directory, open '/nonexistent-bt-tmp/braintrust-span-cache-...jsonl'
    at ... (js/src/span-cache.ts:161:26)

(Note: open PR #2767 serializes concurrent writes in SpanCache, which solves a separate race, and still rethrows errors from flushWriteBuffer() via void this.flushWriteBuffer().)

Expected behavior

Disk cache initialization or append failures should log a warning via debugLogger.warn, reset initPromise on initialization failure so later writes can retry, and leave queued records readable from writeBuffer in memory without crashing the evaluation process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions