Skip to content

Conversation

@smoreinis
Copy link
Collaborator

Summary

  • Adds REDIS_STREAM_MAXLEN environment variable (default: 10,000)
  • Configures xadd() with maxlen and approximate=True for O(1) trimming
  • Prevents Redis memory exhaustion from long-running task event streams

Problem

Redis streams were created without any size limit, causing unbounded memory growth over time. A long-running task could accumulate unlimited events in its stream.

Solution

Added maxlen parameter to xadd() calls with configurable limit via environment variable. Uses approximate=True for better performance (O(1) vs O(N) trimming).

Test Results

Metric Value
Before xadd 12,000 entries
After xadd with maxlen 10,001 entries
Entries trimmed ~2,000

How to Verify

# Check stream length before/after
redis-cli XLEN task:<task_id>

# Monitor memory usage
redis-cli INFO memory | grep used_memory_human

Files Changed

  • src/config/environment_variables.py - Added REDIS_STREAM_MAXLEN env var
  • src/adapters/streams/adapter_redis.py - Added maxlen to xadd()

- Add REDIS_STREAM_MAXLEN environment variable (default: 10000)
- Configure xadd() with maxlen and approximate=True for O(1) trimming
- Prevents Redis memory exhaustion from long-running task event streams
@smoreinis
Copy link
Collaborator Author

Thoughts on this? The existing comment already said "Add to Redis stream with a reasonable max length" so I feel like not having one here was an oversight but I don't have a good sense whether or not 10k here is reasonable or if something like 1k is a better default.

@smoreinis smoreinis marked this pull request as ready for review December 24, 2025 20:56
@smoreinis smoreinis requested a review from a team as a code owner December 24, 2025 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants