Skip to content

Flaky test: test_snapshot_pruning_removes_outdated_records fails with assert 3 == 2 #1734

Closed

Description

@vdusek

Description

test_snapshot_pruning_removes_outdated_records in tests/unit/_autoscaling/test_snapshotter.py is consistently failing across multiple CI runs on Ubuntu (Python 3.13 and 3.14), even when running in isolation (run_alone marker with numprocesses=1).

Failure

FAILED tests/unit/_autoscaling/test_snapshotter.py::test_snapshot_pruning_removes_outdated_records - assert 3 == 2

The test expects that after emitting 4 events with timestamps [now-5h, now-3h, now-2h, now] and a _SNAPSHOT_HISTORY of 2 hours, only 2 snapshots should remain (the ones within the 2-hour window). However, 3 snapshots are retained instead of 2.

Root cause hypothesis

The snapshot pruning logic likely has an off-by-one or boundary condition issue when determining which records fall outside the history window. The now-2h timestamp sits exactly at the boundary of the 2-hour history window, and depending on timing/precision of datetime.now() between the test setup and the pruning execution, it may or may not be pruned.

Observed in

Steps to reproduce

Run the test repeatedly:

pytest tests/unit/_autoscaling/test_snapshotter.py::test_snapshot_pruning_removes_outdated_records -x --count=50

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Feb 11, 2026
  2. Pijukatel commented on Feb 11, 2026

    @Pijukatel
    Collaborator

    I think that the problem is probably in some race condition in sorting/pruning snapshots. Where both are modifying the list in place and most likely are running at the same time.

  3. added a commit that references this issue on Feb 16, 2026
    387c712
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions