Apache Parquet
AssessTools
An open columnar storage file format commonly used for analytics workloads.
Why it's here
Placed in Assess: 3 article(s) of evidence from 2 source(s), led by research-stage coverage, with 2 in the last 30 days. Confidence 45%.
Evidence (3)
- 7InfoQ·8/12/2026researchSpotify Adds External Indexing for Low-Latency Parquet Queries
Spotify has introduced an external indexing architecture for Apache Parquet data lakes so point queries can be served with low latency without copying data into operational databases. The approach maps lookup keys to file and row locations, enabling targeted reads from cloud object storage for analytics, machine learning, AI applications, and online services.
- 7Hacker News·7/31/2026researchBillion-scale graph analytics run on 10GB RAM with Apache DataFusion
The author reports implementing graph algorithms such as PageRank and weakly connected components on billion-edge datasets using Apache DataFusion, relying on disk spill, bulk scans, joins, and aggregations instead of in-memory random access. The tests reportedly succeeded under strict memory limits, with PageRank on 1.05B edges using 5GB and WCC on 1.96B edges using 10GB, though some spill-pool deadlocks and optimization gaps remain.
- 5InfoQ·7/3/2026open_sourceHardwood 1.0 Brings Fast JVM Parquet Processing with No Required Dependencies
Hardwood, the Java project started by Gunnar Morling for working with Parquet files, has reached version 1. The library uses a multi-threaded design and requires no mandatory external dependencies, and it currently supports reading only, with writing planned for future releases.