Skip to content

fix(file-system): stop KVS pagination from skipping or repeating keys - #2431

Merged
vdusek merged 2 commits into
apify:masterfrom
peterolkhov:codex/fix-filesystem-kvs-cursor
Nov 10, 2026
Merged

vdusek merged 2 commits into
apify:masterfrom
peterolkhov:codex/fix-filesystem-kvs-cursor

Conversation

@peterolkhov

@peterolkhov peterolkhov commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Description

The filesystem KVS client sorted records by their encoded filenames, but it returned and compared the key stored in each record's metadata. The two orders differ, so cursor pagination could skip or repeat keys:

  • With item and item-1, item-1.__metadata__.json sorts first, so the cursor item-1 skips item.
  • Percent-encoding reorders keys such as éclair or a/b.json the same way.
  • The SDK input file INPUT.json has the metadata key INPUT, so it came back again after the cursor INPUT.

iterate_keys now reads all record metadata in one thread call. It then filters by exclusive_start_key, sorts, and applies limit, all by the metadata key. A record with invalid metadata is skipped. Before this change, such a record made iteration yield the previous record a second time.

Each call reads every metadata file, because the metadata key can differ from the filename. With 5,000 keys, full pagination at 1,000 keys per page takes 0.36 s (0.43 s on master), and one limit=10 page takes 0.06 s (0.03 s on master).

Public signatures, log messages, and stored files are unchanged.

AI assistance: I used Codex to investigate, implement, and test this change.

Issues

No linked issue.

Testing

  • New unit tests cover cursor pages (prefix, Unicode, path, and ASCII keys), the metadata-key alias, and invalid metadata. Five of them fail without the fix.
  • Real Apify SDK replay: bare INPUT and INPUT.json both end after cursor INPUT.

Checklist

  • CI passed

@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 9, 2026
@vdusek vdusek changed the title fix: paginate filesystem key-value stores by logical key fix(file-system): stop KVS pagination from skipping or repeating keys Nov 10, 2026

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vdusek
vdusek merged commit 7d271c0 into apify:master Nov 10, 2026
50 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants