A Spark file data source for reading and writing Vortex files. It
registers itself as vortex and supports the DataFrame API and Spark SQL without session
extensions or custom catalogs.
Choose the artifact matching both Spark and Scala:
| Artifact | Spark | Scala |
|---|---|---|
dev.vortex:vortex-spark-3.5_2.12 |
3.5.x | 2.12 |
dev.vortex:vortex-spark-3.5_2.13 |
3.5.x | 2.13 |
dev.vortex:vortex-spark-4.0_2.13 |
4.0.x to 4.3.x | 2.13 |
Use the all classifier JAR. It bundles the Vortex JNI bindings, native libraries, and
relocated Arrow, Guava, and Jackson dependencies. The unclassified thin JAR is not usable by
itself.
For example, with version VERSION:
spark-shell --jars https://repo1.maven.org/maven2/dev/vortex/vortex-spark-4.0_2.13/VERSION/vortex-spark-4.0_2.13-VERSION-all.jarOr configure it on the session builder, for example in PySpark:
spark = (
SparkSession.builder
.config("spark.jars", "/path/to/vortex-spark-4.0_2.13-VERSION-all.jar")
.getOrCreate()
)Or add the classified artifact to a JVM project:
implementation("dev.vortex:vortex-spark-4.0_2.13:VERSION:all")Spark's --packages flag cannot select the all classifier, so pass the classified JAR with
--jars instead.
df.write()
.format("vortex")
.mode(SaveMode.Overwrite)
.save("/path/to/output");
Dataset<Row> result = spark.read()
.format("vortex")
.load("/path/to/output");Spark discovers files, Hive-style partitions, and storage credentials through its Hadoop
configuration. This also enables standard file-source options such as pathGlobFilter and
recursiveFileLookup. All file content is read and written through Hadoop streams.
Only files that end with .vortex belong to a Vortex dataset. Writes produce that extension, and
reads skip every other file.
CREATE TABLE student (id INT, name STRING) USING vortex;
INSERT INTO student VALUES (1, 'Alice'), (2, 'Bob');
SELECT * FROM student;
SELECT * FROM vortex.`/path/to/output`;CREATE TABLE ... USING vortex and direct path queries work on Spark 3.5 and 4.x without
catalog configuration. Direct path queries are read-only, as with Spark's built-in formats.
Catalog tables read and write through the V1 file format;
spark.sql.sources.useV1SourceList=vortex puts every path on it, which is what the _metadata
column and dynamic partition pruning need.
Nearby storage reads can be merged to reduce requests. Set the maximum gap and merged span in bytes:
Dataset<Row> result = spark.read()
.format("vortex")
.option("vortex.read.coalesce.distance", 1048576)
.option("vortex.read.coalesce.maxSize", 16777216)
.load("s3a://bucket/path");The defaults are a 1 MiB gap and a 16 MiB maximum merged span. The gap must be non-negative (zero merges only adjacent or overlapping reads); the maximum span must be positive. Both accept integers up to 2147483647 bytes. The native reader caps maximums near that limit to leave room for alignment padding (up to 64 KiB minus one byte). Individual reads can exceed the maximum merged span, and merged reads may include leading alignment padding. A wider gap trades fewer storage requests for more bytes read, including segments already held in the file's segment cache.
Options are case-insensitive and work with both V1 and V2 reads. Session defaults use the same
names; cluster defaults can use spark.datasource.vortex.read.coalesce.distance and
spark.datasource.vortex.read.coalesce.maxSize.
Java callers supplying their own NativeReadable can override coalesceDistance() and
coalesceMaxSize(). Native code captures these limits once when registering each source.
Callers using VortexOutputWriter directly with their own NativeWritable can obtain an
immutable, serializable summary after a successful close, without reopening the file:
writer.close();
VortexOutputWriterSummary summary = writer.summary();
long bytes = summary.fileSize();
long rows = summary.rowCount();The summary also reports the number of batches submitted, the largest batch's Arrow buffer
capacities (including nested buffers), and the maximum native bufferedBytes() sampled after
each batch. These memory measurements exclude some retained allocations and can miss native
peaks between samples; they are not a bound on process memory. No summary is available before
close or after a failed write or close. Repeated closes retain the same summary.
JMH benchmarks live in common/src/jmh/java. They measure the per-file open cost of the
framework's file-splitting model and the footer reads behind statistics and COUNT(*) pushdown.
Run them for one variant and pass JMH arguments through -PjmhArgs:
cd java
./gradlew :vortex-spark-4.0_2.13:jmh -PjmhArgs="SparkScanBenchmark -p fileCount=1,128"
./gradlew :vortex-spark-4.0_2.13:jmh -PjmhArgs="FooterReadBenchmark -f 1 -wi 2 -i 5"See the Spark user guide for supported types,
options, partitioning, and remote storage details. The
Java quickstart covers the underlying
vortex-jni API.