Skip to content

Blog

engine

RSS
Using Krishiv from Rust: a Spark-style session in your crateAdd the krishiv crate, create a Session, read Parquet, filter and group with DataFrame calls or run SQL, and get Arrow record batches back. Errors carry Spark's error class.
UDFs in Krishiv: Python, pandas and Arrow, and RustA Krishiv Python UDF runs in your process: row by row with @udf, a whole pandas Series with @pandas_udf, a whole Arrow array with @arrow_udf. Measured on 200,000 rows, the pandas form is about 4 times and the Arrow form about 8 times faster than the row form, and the Arrow form is within about 15% of a built-in.
Spark in Rust: PySpark without a JVMHow Krishiv 0.2 runs PySpark-style code in your Python process with no JVM: an Arrow executor that spills, Spark semantics checked against real Spark, Python UDFs on a thread pool, and free-threaded 3.14t.
Spark compatibility in Krishiv 0.2: what matchesMeasured on Apache Spark 4.2.0's own SQL test files, Krishiv 0.2 parses 99.8% of the statements Spark parses (19,284 statements in the suite), gives Spark's error class for 89.6% of the statements Spark fails, and the same result for 89.0% of the queries that return rows.
Running Krishiv on your own serverKrishiv 0.2 is a library, so running it on your server means running your own service or job with Krishiv inside: install, CPU baseline, memory and spill settings, concurrency, metrics. A Spark Connect server is planned for 0.3.
Krishiv memory limits and spilling: a 64 MB cap, same answerSet krishiv.memory.limit and a grouped aggregation over 6 million rows keeps working at 64 MB, 16 MB and 8 MB by writing state to disk. The answer does not change; the time does.
Krishiv in Jupyter notebooks: SQL, DataFrames, profileKrishiv runs inside the notebook kernel: a PySpark-style session, DataFrames that render as tables, results as pandas, PyArrow or Polars, and an inline query profile.
Krishiv 0.2: 10x faster than Spark on a laptopOn the same laptop and the same Parquet files, Krishiv 0.2 runs 22 queries derived from TPC-H at SF100 10.4x faster than Spark 4.2.0 (geometric mean), and finishes all 22 at SF1000 in a 20 GB memory limit. A reference engine is still faster today.
Inside the Krishiv planner: from SQL to a physical planKrishiv turns SQL into a physical plan in four steps (parse, analyze, optimize, choose operators) and EXPLAIN VERBOSE prints every rule that changed the plan.
Incremental views explained: update the answer, not the queryIncrementally maintained views are planned, not in Krishiv 0.2. What the idea is, a hand-made example you can run on 0.2.0 (54 ms to recompute, 4 ms to fold in new rows), and why deletes make it hard.