Parallel Evaluation in n0is: Throughput, Latency, and Replication

Abstract

Parallel workers can finish a test batch sooner even when individual tests take longer. We measured this tradeoff while evaluating small language models that drive interactive ETL workflows in n0is. Across 12 randomized blocks and 384 completed chains, four workers delivered 45.75 chains per minute against 29.16 for one worker, an estimated gain of 56.87%. Three and four workers had similar throughput within the uncertainty of the study. An independent experiment also found little evidence of a practical gain from a fifth worker. The measurements use local processes sharing one Mac GPU. The project also used four isolated Linux virtual machines for parallel test capacity.

How parallel trials feed a paired study

The diagram separates reported Linux virtual machine test capacity from the measured worker study on one Mac.

A · Linux VM test infrastructure

1. 1
   Prepare separate cases

   Keep each workflow chain intact.

2. 2
   Run on four Linux VMs

   Garrett reports using four isolated nodes for parallel tests.

   1234
3. 3
   Collect observations

   Record timing and correctness for each completed case.

B · Measured worker study on one Mac

1. 1
   Randomize each block

   Run the same eight cases at each worker count.

2. 2
   Vary worker count

   One to four local processes share the same GPU.

3. 3
   Compare within blocks

   Estimate throughput changes and simultaneous intervals.

The four VMs describe separate test infrastructure. The graphs use controller timings from the Mac worker trials.

1. Research question

The n0is test harness evaluates a chain of ETL requests. Each request uses the workflow state left by the previous one. A model proposes the action, the C++ evaluator executes the workflow, and the harness verifies the result. We asked how many such chains the controller could complete per minute at different worker counts, and what that meant for the latency of a single chain.

The engineering team used four isolated Linux VMs for separate test cases. The archived throughput studies described here used a separate instrumented setup with local processes on one Apple M5 Max sharing one GPU. The figures assign measured effects to that process setup. The Linux VM diagram documents the test infrastructure reported by Garrett.

2. Experimental design

We froze eight previously exercised tuning cases and kept each complete chain on one worker. Each block ran four rounds, one each at one, two, three, and four workers. The harness randomized the order of the four settings. Every round ran the same eight cases with static assignments. The study completed 48 rounds across 12 blocks, covering 384 chains. All 384 passed the recorded checks and reconciled with the verified result receipts.

The throughput clock covers controller time for the timed round, including native execution and independent grading. Throughput equals eight completed chains divided by elapsed round minutes. Individual latency uses the elapsed timer for a full chain and includes its model calls and native execution. These clocks answer different operational questions.

The analysis pairs rounds within the same block, giving 12 observations per worker-count contrast [1]. The reported throughput changes are ratios of geometric means. The analysis used paired sign-randomization tests with a Holm correction across six contrasts. It used Bonferroni-adjusted paired t intervals on log throughput ratios to give simultaneous 95% coverage for the six comparisons under the model assumptions. The figures show the three comparisons involving four workers.

3. Results

Parallel testing throughput on one Mac

The same workload ran with one through four local workers across 12 randomized blocks. Each block contributed one measurement at each worker count.

A · Chains completed per minute

• 1 worker
• 2 workers
• 3 workers
• 4 workers

B · Four workers versus each baseline

Dots mark paired estimates. Lines show simultaneous 95% intervals.

C · Five versus four workers in two experiments

Dots mark paired estimates. Lines show simultaneous 95% intervals.

View the measurements and confidence intervals

Completed chains per minute by block and worker count

Block	1 worker	2 workers	3 workers	4 workers
1	29.62	42.60	51.51	57.08
2	29.59	39.03	46.26	50.76
3	29.54	38.93	51.36	46.08
4	29.60	39.11	46.49	46.04
5	29.62	39.06	46.39	41.87
6	27.89	42.66	42.27	45.93
7	26.30	39.12	38.55	41.64
8	29.60	36.17	38.73	41.88
9	29.61	42.57	46.54	41.95
10	29.63	42.59	42.07	46.08
11	29.60	42.65	46.39	45.82
12	29.60	42.64	38.73	46.06

Throughput change with four workers, estimates and simultaneous 95% intervals

Comparison	Estimate	Lower	Upper
4 vs 1	+56.87%	+44.60%	+70.18%
4 vs 2	+12.87%	+3.94%	+22.56%
4 vs 3	+3.06%	-5.60%	+12.51%

Throughput change in discovery and confirmation, estimates and simultaneous 95% intervals

Comparison	Estimate	Lower	Upper
Discovery	+11.11%	-3.53%	+27.98%
Confirmation	+1.56%	-2.31%	+5.58%

Local workers	Chains passed	Geometric mean chains/min	Mean chain latency (s)
1	96/96	29.16	1.84
2	96/96	40.53	2.44
3	96/96	44.39	3.13
4	96/96	45.75	3.76

Table 1. Throughput summarizes the timed rounds. Individual latency is the arithmetic mean of the chain clocks.

Four workers increased throughput by 56.87% over one (simultaneous 95% interval 44.60% to 70.18%) and by 12.87% over two (3.94% to 22.56%). The comparison with three gave 3.06%, with an interval from −5.60% to 12.51%. Mean individual chain time rose from 1.84 seconds at one worker to 3.76 seconds at four as the chains overlapped. Higher batch throughput made repeat testing quicker. The block design and paired estimates defined the uncertainty.

4. Independent confirmation

A separate discovery experiment compared four to eight workers across 12 blocks, 60 rounds, and 480 chains. Five had the highest observed throughput, with an estimated gain of 11.11% over four and a simultaneous 95% interval from −3.53% to 27.98%. A new 12-block experiment compared four, five, and six workers in 36 rounds and 288 chains. Five’s estimated gain over four fell to 1.56%, with an interval from −2.31% to 5.58%. The team retained four as the working setting for short batches.

5. Scope and replication

The eight cases were fixed tuning workloads, so the 384 passing chains describe execution integrity for those cases. The paired blocks are the analysis units. Each study measures a particular model, machine, and controller boundary. A replication on the four Linux VMs would preserve the case assignments, record per-VM resources and timings, and repeat the randomized blocks under pinned configurations. Report controller and per-chain timing separately so an operator can choose between batch throughput and interactive latency [2].

Further reading

• Comparing Swift with the n0is C++ engine
• First and warmed CSV loads in Zig and n0is C++
• Scheduling a Local LLM Under a Live Microphone from Dev3lop

References

1. Kalibera, T., and Jones, R. Quantifying Performance Changes with Effect Size Confidence Intervals. arXiv, 2020.
2. Raasveldt, M., Holanda, P., Gubner, T., and Mühleisen, H. Fair Benchmarking Considered Difficult: Common Pitfalls in Database Performance Testing. DBTest, 2018.