Chart · millijoules a run

sum by region, in energy.

neo 1.55 mJ in the energy run. Closest: Polars at 32.8×. Farthest: Spark at 1,518×.

—

neo crosses once

neo

Every engine's multiple of neo

  1. neo
    1×
  2. Polars
    32.8× · 50.9 vs 1.55
  3. DuckDB
    46.7× · 72.3 vs 1.55
  4. pandas
    58.5× · 90.6 vs 1.55
  5. Spark
    1,518× · 2,352 vs 1.55

Bars to scale, from zero to the slowest engine. The gray pair is the engine's measurement and neo's in the same run, in mJ.

What this measures

Total the amount column for each of five regions: a grouped sum, the kernel every dashboard runs. Energy is the same run in millijoules: what the machine billed each engine's whole process tree, with its draw at rest subtracted for the seconds of the window. neo's pool is told to rest after the kernel, as the editor tells it.

Why neo is quick here

The region column was dictionary-encoded at load, so its ids are the group numbers and no string is hashed. The sum is taken on the pool in any order only when the column is whole numbers small enough that no addition can round; otherwise it is added in row order on one thread, because a floating-point sum depends on its order, and the contract is the JavaScript loop's result to the bit.

The code that ran, this kernel only

Polars32.8× neo · neo vs Polars

Polars, this kernel · neon-cpp/bench/lang/python/yardstick.py:218
timed("agg", lambda: df.group_by("region").agg(pl.col("amount").sum()))

DuckDB46.7× neo · neo vs DuckDB

DuckDB, this kernel · neon-cpp/bench/lang/python/yardstick.py:237
timed("agg", lambda: con.execute("SELECT region, sum(amount) FROM t GROUP BY region").fetchall())

pandas58.5× neo · neo vs pandas

pandas, this kernel · neon-cpp/bench/lang/python/yardstick.py:197
timed("agg", lambda: df.groupby("region")["amount"].sum())

Spark1,518× neo · neo vs Spark

Spark, this kernel · neon-cpp/bench/lang/python/yardstick.py:255
timed("agg", lambda: df.groupBy("region").agg(F.sum("amount")).collect())

neo

The kernel's body stays in the engine. Its declaration and the guarantees it is held to are below, with the harness line that calls it.

the harness, calling the engine · neon-cpp/bench/bench_engine_options.cpp
 194  l.t = csv::parse(raw);
 337  const double f5 = timeIt([&] { Bitmap b = compareDoubles(amt.data(), n, Cmp::Gt, 500, &pool); sink = b.words[0]; }, 7);
 372  const double s4 = timeIt([&] { sortIndexByDouble(amt.data(), n, perm, &pool); sink = perm[0]; }, 3);
 454  const double a5 = G <= 16 ? timeIt([&] { groupSumAnyOrder(amtP, idP, n, G, &pool, pairTotal); sink = static_cast<size_t>(pairTotal[0]); }, 9) : 0;
the group sum's contract · neon-cpp/core/groupsum.h:1
// groupsum.h — a typed column's sum, count and extremes per group: what the aggregate node's measures need
// once the column is doubles (toDoubles) and every row has a group number.
//
// The results are the ones a loop over the rows in order gives — `if (isFinite(v)) { sum += v; n++;
// if (v < min) min = v; if (v > max) max = v; }` a group, which is what the JavaScript does — to the bit:
//   • counts and extremes are taken a part of the rows at a time on the pool and the parts joined in row
//     order (a strict comparison keeps the earlier of two equal values, as the loop does);
//   • a sum is added in row order on one thread, because a floating-point sum depends on its order —
//     unless the column is whole numbers small enough that no addition can round (wholeNumbers below),
//     where every order gives the same bits and the rows are summed on the pool, two a step.
// bench/bench_agg_node.cpp times each path against the node and checks it; PHD.md §30.

// The sum alone, any order, two rows a step; for tests and the benchmark (groupStats calls it when it may).
// False, and nothing written, when there are more groups than it is built for.
bool groupSumAnyOrder(const double* x, const uint32_t* id, size_t n, size_t groups, Pool* pool, double* sums);
how energy is read · neon-cpp/bench/energy.h:1
// energy.h — what a process has used, as the operating system's own ledger has it: the energy it was billed,
// its processor time, the wake-ups it caused. The one reader of that ledger, for two questions:
//   what a kernel costs — the Evals page's Energy view (bench_engine_options.cpp reads it around each kernel;
//     tools/energy/energy.cpp checks the counter and reads it for a command it runs;
//     bench/lang/python/yardstick.py reads the same field; PHD.md section 33)
//   what a process costs at rest — the idle studies (scripts/idle_cost.sh and scripts/idle_server.py take
//     `neon_energy --pid <pid>` twice and subtract; PHD.md section 31.1)
//
// macOS on Apple silicon: proc_pid_rusage(RUSAGE_INFO_V6). ri_energy_nj — nanojoules billed to the process's
// threads for the processor cores they ran on, since it started. Not the memory, the disk or the screen, and
// not the machine at rest: what running this code added. (Not ri_billed_energy: against half a second of a
// busy loop that field read 0.08 mJ where ri_energy_nj read 1,887.) Beside it, processor time in nanoseconds
// — `ps` has hundredths of a second, more than a resting window uses in a minute — and the wake-ups. No
// privilege is needed, and any process of the same user can be read, so a tree of processes can be summed.
// Linux: /proc/<pid>/stat and /proc/<pid>/status — clock ticks and context switches. No energy: RAPL is one
// figure for the whole machine and root's to read.
// Elsewhere: nothing is read (ok stays false).
//
// A reading is cumulative; a measurement is the difference of two. The counter moves in steps (a thread's
// share is settled when it leaves a core or enters the kernel), so a window should be a quarter of a second
// or more: `neon_energy --check` prints how far apart the steps are on the machine it runs on.

Measured on Apple M5 Max, 18 cores, 26.4.1, 2026-10-11. Every number here is also on the comparison page of its engine, with the whole file it came from.

All evalsEvery comparisonStart a conversation