Out-of-core processing
AssessTechniques
A technique for handling datasets larger than RAM by spilling to disk.
Why it's here
Placed in Assess: 1 article(s) of evidence from 1 source(s), led by framework updates, with 1 in the last 30 days. Confidence 24%. Low accumulated evidence, so it defaults conservatively pending more signal.
Evidence (1)
- 8Hacker News·10/6/2026framework_updatePolars 2.0 adds streaming-by-default, spill-to-disk, and stronger SQL
Polars 2.0 makes the streaming engine the default for collect() on LazyFrame, adds out-of-core spill-to-disk support by default, and introduces a new Map dtype. The release also emphasizes major performance and SQL improvements, with the project claiming competitive benchmark results against DuckDB and DataFusion.