Chart · millijoules a run
filter into a new table, in energy.
neo 21.6 mJ in the energy run. Closest: Polars at 1.02×. Farthest: Spark at 65.0×.
Every engine's multiple of neo
- neo1×
- Polars1.02× · 22.0 vs 21.6
- pandas4.04× · 87.0 vs 21.6
- DuckDB26.3× · 568 vs 21.6
- Spark65.0× · 1,402 vs 21.6
Bars to scale, from zero to the slowest engine. The gray pair is the engine's measurement and neo's in the same run, in mJ.
What this measures
Keep the rows whose amount is over 500 and build a new table of them, about half the file, every column copied. Energy is the same run in millijoules: what the machine billed each engine's whole process tree, with its draw at rest subtracted for the seconds of the window. neo's pool is told to rest after the kernel, as the editor tells it.
Why neo is quick here
The bitmap from the filter drives a parallel gather of every column, and the dictionary column copies its ids rather than its strings.
The code that ran, this kernel only
Polars1.02× neo · neo vs Polars
timed("filter_mat", lambda: df.filter(pred))pandas4.04× neo · neo vs pandas
timed("filter_mat", lambda: df[amount > 500])DuckDB26.3× neo · neo vs DuckDB
timed("filter_mat", lambda: con.execute("CREATE OR REPLACE TABLE f AS SELECT * FROM t WHERE amount > 500"))Spark65.0× neo · neo vs Spark
timed("filter_mat", lambda: sink(df.filter(pred)))neo
The kernel's body stays in the engine. Its declaration and the guarantees it is held to are below, with the harness line that calls it.
194 l.t = csv::parse(raw);
337 const double f5 = timeIt([&] { Bitmap b = compareDoubles(amt.data(), n, Cmp::Gt, 500, &pool); sink = b.words[0]; }, 7);
372 const double s4 = timeIt([&] { sortIndexByDouble(amt.data(), n, perm, &pool); sink = perm[0]; }, 3);
454 const double a5 = G <= 16 ? timeIt([&] { groupSumAnyOrder(amtP, idP, n, G, &pool, pairTotal); sink = static_cast<size_t>(pairTotal[0]); }, 9) : 0;// selection.h — the result of a filter: one bit per row. 1M rows = 125 KB,
// written at memory speed; indices are produced only when something needs them.
struct Bitmap {
std::vector<uint64_t> words;
size_t bits = 0;
bool test(size_t i) const { return (words[i >> 6] >> (i & 63)) & 1; }
size_t count() const;
std::vector<uint32_t> indices() const; // ascending row indices of the set bits
};
// JS measurefilter semantics: NaN never matches. Sequential below ~1M values; the pool otherwise.
Bitmap compareDoubles(const double* v, size_t n, Cmp op, double threshold, Pool* pool);// energy.h — what a process has used, as the operating system's own ledger has it: the energy it was billed,
// its processor time, the wake-ups it caused. The one reader of that ledger, for two questions:
// what a kernel costs — the Evals page's Energy view (bench_engine_options.cpp reads it around each kernel;
// tools/energy/energy.cpp checks the counter and reads it for a command it runs;
// bench/lang/python/yardstick.py reads the same field; PHD.md section 33)
// what a process costs at rest — the idle studies (scripts/idle_cost.sh and scripts/idle_server.py take
// `neon_energy --pid <pid>` twice and subtract; PHD.md section 31.1)
//
// macOS on Apple silicon: proc_pid_rusage(RUSAGE_INFO_V6). ri_energy_nj — nanojoules billed to the process's
// threads for the processor cores they ran on, since it started. Not the memory, the disk or the screen, and
// not the machine at rest: what running this code added. (Not ri_billed_energy: against half a second of a
// busy loop that field read 0.08 mJ where ri_energy_nj read 1,887.) Beside it, processor time in nanoseconds
// — `ps` has hundredths of a second, more than a resting window uses in a minute — and the wake-ups. No
// privilege is needed, and any process of the same user can be read, so a tree of processes can be summed.
// Linux: /proc/<pid>/stat and /proc/<pid>/status — clock ticks and context switches. No energy: RAPL is one
// figure for the whole machine and root's to read.
// Elsewhere: nothing is read (ok stays false).
//
// A reading is cumulative; a measurement is the difference of two. The counter moves in steps (a thread's
// share is settled when it leaves a core or enters the kernel), so a window should be a quarter of a second
// or more: `neon_energy --check` prints how far apart the steps are on the machine it runs on.Measured on Apple M5 Max, 18 cores, 26.4.1, 2026-10-11. Every number here is also on the comparison page of its engine, with the whole file it came from.