Repository navigation
fix(core): stop keeping the whole file in memory for tail reads - #63030
Conversation
There was a problem hiding this comment.
Looks good. I ran the old and new reader on the same inputs and they matched: 40 random multi-chunk files (long lines, lines with no newline, CRLF, multi-byte text, a leading byte-order mark, stray NUL bytes) times 114 offset/limit combinations each, covering negative offsets around the 2000-line cap, offsets past the end and deep positive offsets. All 4,560 results were identical, including the binary-file errors. On a 284 MB log, offset: -5 went from about 4 s and +328 MB to about 0.4 s and +24 MB. Past-end and deep positive reads improved by about the same amount, and the more-than-2000-lines-back re-read took 0.76 s and +36 MB. tool-read-filesystem and tool-read tests pass (69), and typecheck:tsc passes.
Public API: unchanged. Only the internals of the read tool change, and its output stays the same.
No blocking concerns.
There was a problem hiding this comment.
The new commit looks correct. I ran the old and new reader again on 30 random multi-chunk files with long lines in 1–4 byte characters, CRLF, a leading byte-order mark, and NUL bytes placed both at random and inside the part of a long line that gets cut, with 114 offset/limit combinations each. All 3,420 results matched the base, including the binary-file errors. On a single 300 MB line, offset: -2 went from 6.3 s and +1.66 GB to 0.2 s and +25 MB, and offset: 1 from 444 s to 0.25 s. tool-read-filesystem and tool-read tests pass (70), and typecheck:tsc passes.
Public API: unchanged.
No new concerns.
23724ea to
42c72af
Compare
There was a problem hiding this comment.
This push only rebases onto dev. The one change in this PR's files is to the tests, which now match the emoji-safe line truncation from #62575, and that fix works fine with the 8 KB per-line cut. I reran the old-versus-new comparison on 3,420 reads (long lines in 1–4 byte characters including emoji near the 2000-character cut, CRLF, NUL bytes, negative, past-end and deep offsets) and every result matched the updated base. tool-read-filesystem and tool-read tests pass (70), and typecheck:tsc passes.
Public API: unchanged.
No new concerns.
42c72af to
399e2c7
Compare
There was a problem hiding this comment.
The rewrite looks correct. I ran the old and new reader on the same inputs again, against the current base: 7,980 reads over 70 random multi-chunk files, including long lines in 1–4 byte characters, CRLF, a leading byte-order mark and NUL bytes, with negative, past-end and deep offsets. Every result matched the base, including the binary-file errors. On a ~300 MB log, offset: -5 went from 5.1 s and +646 MB to 1.2 s and +119 MB, a past-end read from 4.3 s and +367 MB to 0.8 s and +85 MB, and offset: 2500000, limit: 5 from 3.4 s and +290 MB to 0.4 s and +34 MB. tool-read-filesystem and tool-read tests pass (69), and typecheck:tsc passes.
Public API: unchanged.
No new concerns.
There was a problem hiding this comment.
The new commit looks correct: a negative offset now reads forward only as far as the line count got, so it can't run past where the count stopped when the reported file size is too large. I ran the old and new reader on the same inputs again: 7,980 reads over 70 random multi-chunk files, all matching the base. I didn't reproduce the too-large-size case itself and checked that part by reading the code. tool-read-filesystem and tool-read tests pass (69), and typecheck:tsc passes.
Public API: unchanged.
No new concerns.
There was a problem hiding this comment.
The one-pass version looks correct. I ran the old and new reader on the same inputs again: 7,980 reads over 70 random multi-chunk files, with long lines in 1–4 byte characters, CRLF, a leading byte-order mark, NUL bytes, and negative, past-end and deep offsets. Every result matched the base. On a ~300 MB log, offset: -5 went from 14.8 s and +698 MB to 0.85 s and +117 MB, a past-end read from 4.2 s to 0.4 s, and offset: -5000, limit: 3 from 4.4 s and +318 MB to 0.4 s and +34 MB. tool-read-filesystem and tool-read tests pass (69), and typecheck:tsc passes.
Public API: unchanged.
No new concerns.
There was a problem hiding this comment.
The rewrite looks correct. I ran the old and new reader on the same inputs again, against the current base: 7,980 reads over 70 random multi-chunk files, including long lines in 1–4 byte characters, CRLF, a leading byte-order mark, NUL bytes inside long lines, and negative, past-end and deep offsets. Every result matched the base, including the binary-file errors. A single 108 MiB line went from 22.8 s and +1.1 GB to 245 ms and +29 MB at offset: 1, and from 4.6 s and +726 MB to 253 ms and +28 MB at offset: -1. On a ~300 MB log, tail, past-end and deep reads now take about 0.5–0.6 s and +30 MB instead of 3.4–4.7 s and +290–637 MB. tool-read-filesystem and tool-read tests pass (71), and typecheck:tsc passes.
Public API: unchanged.
No new concerns.
Issue for this PR
Follow-up to #62147.
Type of change
What does this PR do?
Reading a page from a large file could keep the whole file in memory. A 108 MiB file with one unterminated line took over 1 GiB of peak process memory to return a 2,034-byte preview.
The reader counts newlines in 256 KiB chunks. An ordinary read stops once it knows the page and whether another line follows. For a negative or past-end offset it counts through EOF, then uses the saved chunk positions to read the selected lines. Even if a line spans hundreds of chunks, only its first 8,004 bytes need decoding for the 2,000-character preview.
Local read times, median of three fresh Bun processes, compared with the previous version of this PR (
2c1e7b2a17):Median peak process memory for the single-line file fell from 1,061 MiB to 113 MiB. Tail reads still scan the full file to report absolute line numbers.
How did you verify your code works?
bun run checkand both read-tool test files passed (71 tests). I compared 195 file/page combinations with the previous version, including content, offsets, continuation and binary errors. All matched. I also checked the exact output for the large fixtures and a minified source map followed by three short lines.Visual
Checklist