You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
SpanCache background flush errors are unhandled, crashing Eval() when the temp file cannot be written #2803
Eval() enables SpanCache by default for experiment-backed evaluations. Every span write calls SpanCache.queueWrite(), which starts this.flushWriteBuffer() in the background and stores the promise on this.flushPromise without a .catch() handler (js/src/span-cache.ts). Nothing awaits or catches flushPromise during Eval().
When the temporary cache file cannot be opened or appended to (for example, TMPDIR pointing at a missing/stale directory, a read-only container filesystem without /tmp, or ENOSPC / EIO), flushWriteBuffer() rejects (ensureInitialized() and fileHandle.appendFile()). Under Node's default --unhandled-rejections=throw, the optional disk cache terminates the entire Eval() process with ENOENT / EIO.
In addition, ensureInitialized() caches the rejected this.initPromise, so every subsequent write rejects immediately as well.
Reproduction
Add js/src/span-cache-flush-failure.test.ts and run cd js && pnpm exec vp test run src/span-cache-flush-failure.test.ts:
import{afterEach,beforeAll,describe,expect,test}from"vite-plus/test";import{SpanCache}from"./span-cache";import{configureNode}from"./node/config";describe("SpanCache background flush failures",()=>{constunhandled: unknown[]=[];constonUnhandled=(reason: unknown)=>unhandled.push(reason);constrealTmpdir=process.env.TMPDIR;letcache: SpanCache|undefined;beforeAll(()=>configureNode());afterEach(()=>{process.off("unhandledRejection",onUnhandled);if(realTmpdir===undefined)deleteprocess.env.TMPDIR;elseprocess.env.TMPDIR=realTmpdir;cache?.stop();cache?.dispose();cache=undefined;unhandled.length=0;});test("an unwritable temp dir does not cause an unhandled rejection",async()=>{process.on("unhandledRejection",onUnhandled);process.env.TMPDIR="/nonexistent-braintrust-tmp-dir";cache=newSpanCache();cache.start();cache.queueWrite("root-1","span-1",{span_id: "span-1",input: "x"});awaitnewPromise((resolve)=>setTimeout(resolve,100));expect(unhandled).toEqual([]);});});
Running an experiment-backed Eval() / runEvaluator in a standalone Node process with TMPDIR=/nonexistent-bt-tmp exits 1 before the evaluation returns:
Error: ENOENT: no such file or directory, open '/nonexistent-bt-tmp/braintrust-span-cache-...jsonl'
at ... (js/src/span-cache.ts:161:26)
(Note: open PR #2767 serializes concurrent writes in SpanCache, which solves a separate race, and still rethrows errors from flushWriteBuffer() via void this.flushWriteBuffer().)
Expected behavior
Disk cache initialization or append failures should log a warning via debugLogger.warn, reset initPromise on initialization failure so later writes can retry, and leave queued records readable from writeBuffer in memory without crashing the evaluation process.
Activity