The Spark-compatible engine without the JVM.

The public roadmap is on GitHub. Nothing in this table is a date.

Compatibility

Krishiv follows Spark's behaviour where you can observe it, keyed by Spark version, and checks it against Spark's own test results. Where it differs, the difference is listed in the compatibility page. "Spark-compatible" describes that goal and the API shape. Krishiv is not Apache Spark, and Apache Spark is a trademark of the Apache Software Foundation.

Quickstart

pip install "krishiv[pyarrow]"

import tempfile
from pathlib import Path

import pyarrow as pa
import pyarrow.parquet as pq

import krishiv
from krishiv.sql import functions as F

spark = krishiv.session()  # a PySpark-style SparkSession, in process

spark.sql("SELECT 1 AS answer").show()

path = str(Path(tempfile.mkdtemp()) / "orders.parquet")
pq.write_table(
    pa.table(
        {
            "region": ["east", "west", "east", "north", "west"],
            "amount": [10.0, 20.0, 5.5, 7.25, 30.0],
        }
    ),
    path,
)
orders = spark.read.parquet(path)
orders.createOrReplaceTempView("orders")

by_region = (
    orders.filter(F.col("amount") > 6)
    .groupBy("region")
    .agg(F.sum("amount").alias("revenue"))
    .orderBy("region")
)
by_region.show()
print(by_region.toArrow())  # a pyarrow.Table; also pa.table(by_region), polars.DataFrame(by_region)

same = spark.sql("SELECT region, sum(amount) AS revenue FROM orders WHERE amount > 6 GROUP BY region ORDER BY region")
assert same.collect() == by_region.collect()

Measured numbers

Method. Standard TPC-H, TPC-DS and ClickBench query sets over Parquet, read directly with no pre-loading step. Each query runs after warm-up iterations; we report the median of the timed iterations with its spread, the commit it was measured on, the machine, thread count and memory limit. Numbers are published only from a clean, committed tree, and the harness is in the repository under bench/ so you can run it yourself.

[Benchmark chart: TPC-H SF10, SF100, SF1000, with method; numbers at launch]