Skip to content

perf: Reduce decode overhead during pruning keys in the memtable#3415

Merged
v0y4g3r merged 20 commits intoGreptimeTeam:mainfrom
evenyag:perf/memtable-prune
Mar 8, 2024
Merged

perf: Reduce decode overhead during pruning keys in the memtable#3415
v0y4g3r merged 20 commits intoGreptimeTeam:mainfrom
evenyag:perf/memtable-prune

Conversation

@evenyag
Copy link
Contributor

@evenyag evenyag commented Feb 29, 2024

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

This PR improves the pruning speed of the new memtable

  • Adds decode_value_at() method to decode a value at a specific position in the primary key
    • It will skip values before the position
    • Stores offsets in a buffer so we can reuse them while there are multiple filters to prune
  • Adds a PrimaryKeyFilter to filter primary keys via decode_value_at()
  • Pushes down filters to Shards and ShardBuilder to reduce DataBatch returned to the partition reader

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR does not require documentation updates.

Refer to a related PR or issue link (optional)

@github-actions github-actions bot added the docs-not-required This change does not impact docs. label Feb 29, 2024
@evenyag evenyag force-pushed the perf/memtable-prune branch from ebd00b0 to 69bdbf0 Compare March 1, 2024 06:39
@evenyag evenyag marked this pull request as ready for review March 1, 2024 06:56
@codecov
Copy link

codecov bot commented Mar 1, 2024

Codecov Report

Attention: Patch coverage is 91.91919% with 32 lines in your changes are missing coverage. Please review.

Project coverage is 84.72%. Comparing base (376409b) to head (aea9828).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3415      +/-   ##
==========================================
- Coverage   85.05%   84.72%   -0.33%     
==========================================
  Files         889      889              
  Lines      146427   146709     +282     
==========================================
- Hits       124547   124303     -244     
- Misses      21880    22406     +526     

@v0y4g3r v0y4g3r self-requested a review March 1, 2024 07:11
Copy link
Collaborator

@fengjiachun fengjiachun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@evenyag evenyag mentioned this pull request Mar 5, 2024
8 tasks
Copy link
Contributor

@v0y4g3r v0y4g3r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@v0y4g3r v0y4g3r added this pull request to the merge queue Mar 8, 2024
Merged via the queue into GreptimeTeam:main with commit 3ee5336 Mar 8, 2024
@v0y4g3r v0y4g3r deleted the perf/memtable-prune branch March 8, 2024 03:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-not-required This change does not impact docs.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants