Krishiv

Spark-compatible compute. Written in Rust.

UNIFIED COMPUTE ENGINE

Spark-compatible compute. Written in Rust.

Run your Spark SQL and DataFrame code inside your Python or Rust process. No JVM, no cluster to start. Server and distributed modes are being built on the same engine.

Read the docsGitHub

pip install krishiv

Filter, group and sum a Parquet file

python
import krishiv
from krishiv.sql import functions as F

spark = krishiv.session()  # in process, no server
orders = spark.read.parquet("orders.parquet")

(orders
    .filter(F.col("amount") > 6)
    .groupBy("region")
    .agg(F.sum("amount").alias("revenue"))
    .orderBy("region")
    .show())

  • Spark semantics, checked

    SQL and DataFrame behaviour is tested against Spark's own test results, statement by statement, including its error classes. Differences are published.

  • Embedded first

    A library, not a cluster. Install a wheel or a crate and query Parquet from your own process, with results as Arrow.

  • One engine, every scale

    Built to run embedded, behind a Spark Connect server and across workers, with batch and streaming sharing it. Embedded runs today; the rest is in progress.

Start with the engine. The platform is next.

  1. Available

    Krishiv Engine

    The compute engine, open source under Apache-2.0. Spark SQL and the DataFrame API in Python and Rust; EXPLAIN ANALYZE per operator.

  2. Preview

    Krishiv Data Platform

    [Approved one-line description of the platform]

Measured, not claimed.

Every number comes with its method: dataset, scale, hardware, cold or warm, and the commit. [Benchmark chart: numbers at launch]

See the method

  • Operator

    Krishiv AI. Contact: info@krishiv.ai. Krishiv Engine is open source under Apache-2.0.

  • Trademarks

    Apache Spark and Spark are trademarks of the Apache Software Foundation. Krishiv is not Apache Spark and is not endorsed by or affiliated with the Apache Software Foundation.