Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor(rust): Reduce Vec allocations in new-streaming parquet source #19493

Merged
merged 6 commits into from
Oct 28, 2024
Merged

refactor(rust): Reduce Vec allocations in new-streaming parquet source #19493

merged 6 commits into from
Oct 28, 2024

Conversation

nameexhaustion
Copy link
Collaborator

Replaces a Vec allocation we were doing for every row group with a single VecDeque that is extended with impl Iterator

@github-actions github-actions bot added internal An internal refactor or improvement rust Related to Rust Polars labels Oct 28, 2024
@nameexhaustion nameexhaustion marked this pull request as ready for review October 28, 2024 13:50
@ritchie46 ritchie46 merged commit 2879a2d into pola-rs:main Oct 28, 2024
18 of 20 checks passed
@nameexhaustion nameexhaustion deleted the parquet-source-opts branch October 29, 2024 06:32
@c-peters c-peters added the accepted Ready for implementation label Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ritchie46 ritchie46 ritchie46 approved these changes

@orlp orlp Awaiting requested review from orlp orlp is a code owner

@c-peters c-peters Awaiting requested review from c-peters c-peters is a code owner

Assignees

@nameexhaustion nameexhaustion

Labels
accepted Ready for implementation internal An internal refactor or improvement rust Related to Rust Polars
Projects
Backlog
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@nameexhaustion @ritchie46 @c-peters