Skip to content

perf: Half the size of Booleans in row encoding#19927

Merged
ritchie46 merged 1 commit intopola-rs:mainfrom
coastalwhite:perf/bool-row-encoding
Nov 23, 2024
Merged

perf: Half the size of Booleans in row encoding#19927
ritchie46 merged 1 commit intopola-rs:mainfrom
coastalwhite:perf/bool-row-encoding

Conversation

@coastalwhite
Copy link
Copy Markdown
Collaborator

This changes the encoding of pl.Boolean in the row encoding from needing 2 bytes (1 for validity, 1 for value) to 1 byte. Now, the encoding of Boolean values is as follows:

Value Encoding
None 0x00
False 0x20
True 0x30

The null is bitwise inverted for nulls_last=True and False / True are inverted for descending=True.

This is a continuation of #19874.

@github-actions github-actions bot added performance Performance issues or improvements python Related to Python Polars rust Related to Rust Polars labels Nov 22, 2024
@coastalwhite coastalwhite force-pushed the perf/bool-row-encoding branch from 4c3fd09 to d1a172a Compare November 22, 2024 16:00
This changes the encoding of `pl.Boolean` in the row encoding from needing 2
bytes (1 for validity, 1 for value) to 1 byte. Now, the encoding of Boolean
values is as follows:

| Value   | Encoding |
|---------|----------|
| `None`  | `0x00`   |
| `False` | `0x20`   |
| `True`  | `0x30`   |

The null is bitwise inverted for `nulls_last=True` and `False` / `True` are
inverted for `descending=True`.
@coastalwhite coastalwhite force-pushed the perf/bool-row-encoding branch from d1a172a to 7615e2c Compare November 22, 2024 16:02
@codecov
Copy link
Copy Markdown

codecov bot commented Nov 22, 2024

Codecov Report

Attention: Patch coverage is 58.53659% with 34 lines in your changes missing coverage. Please review.

Project coverage is 59.28%. Comparing base (414d883) to head (7615e2c).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
crates/polars-row/src/fixed.rs 40.38% 31 Missing ⚠️
crates/polars-row/src/row.rs 80.00% 2 Missing ⚠️
crates/polars-row/src/variable.rs 50.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #19927   +/-   ##
=======================================
  Coverage   59.28%   59.28%           
=======================================
  Files        1555     1555           
  Lines      216180   216213   +33     
  Branches     2456     2456           
=======================================
+ Hits       128155   128177   +22     
- Misses      87467    87478   +11     
  Partials      558      558           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:

@ritchie46 ritchie46 merged commit 5eeb369 into pola-rs:main Nov 23, 2024
@coastalwhite coastalwhite deleted the perf/bool-row-encoding branch November 23, 2024 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Performance issues or improvements python Related to Python Polars rust Related to Rust Polars

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants