Chart · milliseconds · public data
load ClickBench · 1M, warm.
neo 96.0 ms in the public-data run. Closest: Polars at 1.97×. Farthest: pandas at 61.2×.
Every engine's multiple of neo
- neo1×
- Polars1.97× · 189 vs 96.0
- Arrow3.89× · 373 vs 96.0
- DuckDB15.0× · 1,438 vs 96.0
- pandas61.2× · 5,876 vs 96.0
Bars to scale, from zero to the slowest engine. The gray pair is the engine's measurement and neo's in the same run, in ms.
What this measures
Read ClickBench · 1M from its path to a table in memory: 1,000,000 rows by 105 columns, 842 MB of web hits. Every engine in a fresh process each round, the file in the OS cache, each engine's start-up done before the clock; warm is the median of the loads after the first. neo's table is text, with no types, no nulls and no UTF-8 check, so the rows as text are the like-for-like ones.
Why neo is quick here
The same parser as the synthetic load, on a file this project did not make, checked column by column and row by row against DuckDB's reading of it before any time is taken.
The code that ran, this kernel only
Polars1.97× neo · neo vs Polars
def load_polars(path, mode, loads, opt=None):
import polars as pl
kw = {}
if opt and "infer" in opt: kw = {"infer_schema_length": opt["infer"]}
if opt and "overrides" in opt: kw = {"schema_overrides": {c: (pl.Float64 if t == "float" else pl.String) for c, t in opt["overrides"].items()}}
pl.read_csv(io.BytesIO(b"a,b\n1,x\n2,y\n")) # untimed: the reader's own start-up (its thread pool), on three lines that are not the file
out, rows, cols = [], 0, 0
for _ in range(loads):
t0 = time.perf_counter()
df = pl.read_csv(path, infer_schema=False) if mode == "text" else pl.read_csv(path, **kw)
out.append(ms_since(t0))
rows, cols = df.height, df.width
del df
return pl.__version__, out, rows, cols, pl.thread_pool_size()Arrow3.89× neo · neo vs Arrow
def load_arrow(path, mode, loads, opt=None):
import pyarrow as pa
import pyarrow.csv as pc
convert = pc.ConvertOptions(column_types={n: pa.string() for n in header_names(path)}) if mode == "text" else pc.ConvertOptions()
parse = pc.ParseOptions(newlines_in_values=True) if opt else pc.ParseOptions()
pc.read_csv(io.BytesIO(b"a,b\n1,x\n2,y\n")) # untimed: the reader's own start-up (its thread pools), on three lines that are not the file
out, rows, cols = [], 0, 0
for _ in range(loads):
t0 = time.perf_counter()
t = pc.read_csv(path, convert_options=convert, parse_options=parse)
out.append(ms_since(t0))
rows, cols = t.num_rows, t.num_columns
del t
return pa.__version__, out, rows, cols, pa.cpu_count()DuckDB15.0× neo · neo vs DuckDB
def load_duckdb(path, mode, loads, opt=None):
import duckdb
con = duckdb.connect()
con.execute("SELECT 42").fetchall() # untimed: what the binding sets up on its first query (it imports pandas) is not CSV loading
opts = ", all_varchar=true" if mode == "text" else ""
out, rows, cols = [], 0, 0
for _ in range(loads):
con.execute("DROP TABLE IF EXISTS t")
t0 = time.perf_counter()
con.execute(f"CREATE TABLE t AS SELECT * FROM read_csv(?, header=true{opts})", [path])
out.append(ms_since(t0))
rows = con.execute("SELECT count(*) FROM t").fetchone()[0]
cols = len(con.execute("SELECT * FROM t LIMIT 0").description)
threads = con.execute("SELECT current_setting('threads')").fetchone()[0]
return duckdb.__version__, out, rows, cols, int(threads)pandas61.2× neo · neo vs pandas
def load_pandas(path, mode, loads, opt=None):
import pandas as pd
pd.read_csv(io.BytesIO(b"a,b\n1,x\n2,y\n"), engine="pyarrow") # untimed: the reader's own start-up
out, rows, cols = [], 0, 0
for _ in range(loads):
t0 = time.perf_counter()
# engine="pyarrow": pandas' multithreaded reader, one keyword from the default (its C engine is single-threaded:
# 1.3-1.7 s on worldcitiespop where this takes tens of ms — bench/results/standards.jsonl, the records of 09:02-09:22)
eng = (opt or {}).get("engine", "pyarrow")
df = pd.read_csv(path, dtype=str, keep_default_na=False, engine=eng) if mode == "text" else pd.read_csv(path, engine=eng)
out.append(ms_since(t0))
rows, cols = df.shape
del df
import pyarrow as pa
return pd.__version__, out, rows, cols, (pa.cpu_count() if (opt or {}).get("engine", "pyarrow") == "pyarrow" else 1)neo
The kernel's body stays in the engine. Its declaration and the guarantees it is held to are below, with the harness line that calls it.
194 l.t = csv::parse(raw);
337 const double f5 = timeIt([&] { Bitmap b = compareDoubles(amt.data(), n, Cmp::Gt, 500, &pool); sink = b.words[0]; }, 7);
372 const double s4 = timeIt([&] { sortIndexByDouble(amt.data(), n, perm, &pool); sink = perm[0]; }, 3);
454 const double a5 = G <= 16 ? timeIt([&] { groupSumAnyOrder(amtP, idP, n, G, &pool, pairTotal); sink = static_cast<size_t>(pairTotal[0]); }, 9) : 0;Table parse(std::string_view raw); // parallel above 1 MB, serial below; identical resultsMeasured on Apple M5 Max, 18 cores, 26.4.1, 2026-10-11. Every number here is also on the comparison page of its engine, with the whole file it came from.