Skip to content

Add better support for pagination #539

Closed

Description

@Pijukatel

Activity

  1. self-assigned this
    on Nov 19, 2025
  2. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Nov 19, 2025
  3. added this to the 3.0 milestone on Feb 26, 2026
  4. vdusek commented on Apr 22, 2026

    @vdusek
    Contributor

    During the implementation, please make sure these points are considered, and either addressed or evaluated as false positives.


    2. iterate_items early-terminates when filters shrink the page

    • Severity: normal
    • File: src/apify_client/_resource_clients/dataset.py:295-304

    iterate_items terminates early whenever skip_empty, skip_hidden, or clean=True causes a server-side page to contain fewer than 1000 items — even when there are many more items left in the dataset. This results in silent data loss for any caller using the documented filter parameters.

    current_page_item_count = len(current_items_page.items)
    read_items += current_page_item_count
    
    if current_page_item_count < cache_size:
        should_finish = True

    This is only correct when every item in the scanned [offset, offset+limit) window is returned verbatim. The Apify API applies skip_empty, skip_hidden, and clean as post-filters, so a short response does not imply the dataset is exhausted.

    Step-by-step proof

    Dataset with 5000 items, ~half empty, caller invokes dataset.iterate_items(skip_empty=True):

    1. First: list_items(offset=0, limit=1000, skip_empty=True) → server scans 0..999, filters out ~500 empties, returns ~500.
    2. current_page_item_count = 500; read_items = 500.
    3. 500 < 1000 → should_finish = True.
    4. Loop exits. ~2000 remaining non-empty items are silently dropped.

    Secondary: offset drift

    Line 285 advances the window using read_items (items returned) rather than items scanned: offset=offset + read_items. Under any filter, read_items < items scanned, so the next page re-reads items already scanned.

    Fix

    Track the scanned window explicitly, or terminate only on an empty response (current_page_item_count == 0).


    3. iterate_items does not forward flatten and view

    • Severity: normal
    • File: src/apify_client/_resource_clients/dataset.py:284-296

    list_items() accepts flatten: list[str] | None and view: str | None and forwards them via _build_params. iterate_items() is documented as a paginating iterator on top of list_items(), and its internal call forwards every other filter/shape parameter (clean, desc, fields, omit, unwind, skip_empty, skip_hidden, signature, plus offset/limit/timeout). But its signature omits flatten and view, and the internal self.list_items(...) call does not pass them. The async twin has the identical gap.

    for item in client.dataset(ds_id).iterate_items(view='my_view'):
        ...
    # TypeError: iterate_items() got an unexpected keyword argument 'view'

    Fix

    Add flatten: list[str] | None = None and view: str | None = None to iterate_items signatures (sync and async) and forward both in the internal self.list_items(...) call at lines 284-296.

  5. self-assigned this
    on May 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions