Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add IPC source node for new streaming engine #19454

Merged
merged 15 commits into from
Nov 13, 2024
Merged

feat: Add IPC source node for new streaming engine #19454

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
b969701
feat: Add IPC source node for new streaming engine
coastalwhite Oct 25, 2024
c267b8c
Add predicate and row index handling
coastalwhite Oct 25, 2024
053cdbf
properly set up slice and decode tasks
coastalwhite Oct 27, 2024
e5d9445
fix unimplemented features
coastalwhite Oct 28, 2024
f435e39
move todo into lower_ir
coastalwhite Oct 31, 2024
cdf7f50
clean up ipc source
coastalwhite Oct 31, 2024
63c70a4
detect IdxSize overflows
coastalwhite Oct 31, 2024
6d441a1
fix invalid condition
coastalwhite Oct 31, 2024
6d772c4
use same source token between decoder tasks
coastalwhite Nov 4, 2024
b1cc5e8
add cancelation and chunking
coastalwhite Nov 4, 2024
9e2afaf
cargo fmt
coastalwhite Nov 5, 2024
0e5de60
fix deadlock and better distribute chunks if possible
coastalwhite Nov 6, 2024
49b1ee8
Revert "fix deadlock and better distribute chunks if possible"
coastalwhite Nov 6, 2024
a3deeb9
properly fix deadlock this time
coastalwhite Nov 6, 2024
8ff9766
Some review reformatting.
orlp Nov 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix invalid condition
  • Loading branch information
@coastalwhite
coastalwhite committed Nov 13, 2024
commit 6d441a1e13d676d6a80ea6d4591918397a3b8fed
2 changes: 1 addition & 1 deletion crates/polars-core/src/frame/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3338,7 +3338,7 @@ impl DataFrame {

pub fn append_record_batch(&mut self, rb: RecordBatchT<ArrayRef>) -> PolarsResult<()> {
polars_ensure!(
rb.arrays().len() != self.width(),
rb.arrays().len() == self.width(),
InvalidOperation: "attempt to extend dataframe of width {} with record batch of width {}",
self.width(),
rb.arrays().len(),
Expand Down