You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
iterate_items terminates early whenever skip_empty, skip_hidden, or clean=True causes a server-side page to contain fewer than 1000 items — even when there are many more items left in the dataset. This results in silent data loss for any caller using the documented filter parameters.
This is only correct when every item in the scanned [offset, offset+limit) window is returned verbatim. The Apify API applies skip_empty, skip_hidden, and clean as post-filters, so a short response does not imply the dataset is exhausted.
Step-by-step proof
Dataset with 5000 items, ~half empty, caller invokes dataset.iterate_items(skip_empty=True):
First: list_items(offset=0, limit=1000, skip_empty=True) → server scans 0..999, filters out ~500 empties, returns ~500.
current_page_item_count = 500; read_items = 500.
500 < 1000 → should_finish = True.
Loop exits. ~2000 remaining non-empty items are silently dropped.
Secondary: offset drift
Line 285 advances the window using read_items (items returned) rather than items scanned: offset=offset + read_items. Under any filter, read_items < items scanned, so the next page re-reads items already scanned.
Fix
Track the scanned window explicitly, or terminate only on an empty response (current_page_item_count == 0).
3. iterate_items does not forward flatten and view
list_items() accepts flatten: list[str] | None and view: str | None and forwards them via _build_params. iterate_items() is documented as a paginating iterator on top of list_items(), and its internal call forwards every other filter/shape parameter (clean, desc, fields, omit, unwind, skip_empty, skip_hidden, signature, plus offset/limit/timeout). But its signature omits flatten and view, and the internal self.list_items(...) call does not pass them. The async twin has the identical gap.
Add flatten: list[str] | None = None and view: str | None = None to iterate_items signatures (sync and async) and forward both in the internal self.list_items(...) call at lines 284-296.
Activity
vdusek commentedon Apr 22, 2026
During the implementation, please make sure these points are considered, and either addressed or evaluated as false positives.
2.
iterate_itemsearly-terminates when filters shrink the pagesrc/apify_client/_resource_clients/dataset.py:295-304iterate_itemsterminates early wheneverskip_empty,skip_hidden, orclean=Truecauses a server-side page to contain fewer than 1000 items — even when there are many more items left in the dataset. This results in silent data loss for any caller using the documented filter parameters.This is only correct when every item in the scanned
[offset, offset+limit)window is returned verbatim. The Apify API appliesskip_empty,skip_hidden, andcleanas post-filters, so a short response does not imply the dataset is exhausted.Step-by-step proof
Dataset with 5000 items, ~half empty, caller invokes
dataset.iterate_items(skip_empty=True):list_items(offset=0, limit=1000, skip_empty=True)→ server scans 0..999, filters out ~500 empties, returns ~500.current_page_item_count = 500;read_items = 500.500 < 1000→should_finish = True.Secondary: offset drift
Line 285 advances the window using
read_items(items returned) rather than items scanned:offset=offset + read_items. Under any filter,read_items < items scanned, so the next page re-reads items already scanned.Fix
Track the scanned window explicitly, or terminate only on an empty response (
current_page_item_count == 0).3.
iterate_itemsdoes not forwardflattenandviewsrc/apify_client/_resource_clients/dataset.py:284-296list_items()acceptsflatten: list[str] | Noneandview: str | Noneand forwards them via_build_params.iterate_items()is documented as a paginating iterator on top oflist_items(), and its internal call forwards every other filter/shape parameter (clean,desc,fields,omit,unwind,skip_empty,skip_hidden,signature, plusoffset/limit/timeout). But its signature omitsflattenandview, and the internalself.list_items(...)call does not pass them. The async twin has the identical gap.Fix
Add
flatten: list[str] | None = Noneandview: str | None = Nonetoiterate_itemssignatures (sync and async) and forward both in the internalself.list_items(...)call at lines 284-296.