← Research papers

Systems methods · Working paper 0.2

Ports as Instruments: CSV Load Timing in Zig and n0is C++

Tyler Garrett

Download PDF Open PDF builder

Abstract

The same CSV input and output contract can yield different performance rankings depending on how many loads a process has already run. We compare a Zig port with the n0is C++ CSV loader on one Apple M5 Max in two sessions, with 20 and 30 paired blocks per condition. In the later session, Zig was faster in 23 of 30 first-load pairs at one million rows, and n0is C++ was faster in all 30 warmed pairs. At ten million rows, n0is C++ again won all warmed pairs, while first-load wins split 15 each and the adjusted interval spanned equal time. The second session reproduced the one-million-row crossover and the warmed advantage at both sizes. The first-load lead at ten million rows depended on the session. Descriptive splits by execution order and a leave-one-pair-out check show where the comparisons were sensitive.

1. Research question

The companion Swift study used a port to expose differences in work and timer placement. Here the question is whether the ordering of two implementations reverses between first and warmed loads when both follow the same timed output contract. A workflow may load a table once in a newly launched worker or repeatedly in a long-lived process. A load-only measurement gives a view of each case, while user-visible latency may also include launch and file acquisition.

A November 8 comparison of five builds first showed a Zig lead on the initial load and a C++ lead after repeated loads at both tested sizes. Two subsequent sessions ran only the C++ and Zig executables in pairs. They used the same input bytes and executables as the five-build comparison. The pairwise sessions ran the two observations consecutively within each block and balanced which executable ran first. The five-build schedule used different preceding workloads and a different family of intervals. We analyze each pairwise session on its own.

2. Workload and clocks

2.1 Input and output contract

The fixtures contain one million and ten million rows with the same six-column schema and occupy about 46.3 MB and 472.6 MB. Each timed load parses all six columns into a materialized text table, converts amount to float64, and builds one dictionary for region. The harness disables the C++ path’s two optional dictionaries so both sides perform the same requested materialization. Parallel delimiter parsing is already well studied [2]. This comparison measures a larger unit of work that includes typed conversion and dictionary construction.

Separate preflight and postflight checks match both builds’ parsed-cell fingerprints against the archived reference for each input. The reference covers table dimensions and the text of all parsed cells in row order. This can catch a change in row counts, fields, or ordering that would invalidate the comparison. Zig uses its serial path for the fingerprint check. The check leaves derived numeric values, dictionary IDs, and the output of each timed parallel load unverified. Every timing record separately reports the expected row count and the requested participant count.

2.2 Clock boundaries

  • First load. The first timed load in a fresh process. Both builds read the input into memory before timing. Worker setup also starts before the load clock, with build-specific readiness. The OS file cache is uncontrolled.
  • Warmed load. The median of three timed loads after three warmup loads in the same process. A prior output remains live during the next load. The harness uses the individual load clocks to compute one median per process.

The first-load label refers to a process state. Since input acquisition precedes the clock, repeated access to the same fixture can benefit from the operating system file cache without that access being timed. In warmed mode the earlier in-process loads can also change allocator, worker, and memory state before the measured work begins. The design compares those bundled conditions. It does not identify the contribution of each state change.

The harness also records command wall time from subprocess invocation through exit and output capture. In the second session, the 10M first-load command medians were 112.692 ms for C++ and 99.663 ms for Zig, compared with internal load medians of 46.267 ms and 45.998 ms. The command clock combines launch, acquisition, runtime setup, the load, and teardown. Its gap shows how the selected measurement boundary can change the picture. It cannot assign the extra time to any one stage. The warmed commands include all six loads and therefore have a different unit from the internal per-load statistic.

3. Paired design and analysis

3.1 Collection

The November 8 pairwise session used 20 pairs per condition, for 80 pairs and 160 processes. The November 9 session used 30 pairs per condition, for 120 pairs and 240 processes. Both ran on the same Apple M5 Max with macOS 26.4.1, on battery. The harness requested 16 participants for both builds and set a common utility scheduling priority ceiling. Pool readiness, scheduling, allocation, and chunking remained implementation choices. The frozen plan recorded binary and fixture digests, which the harness rechecked after collection.

The C++ binary was built for arm64 with Apple clang 17.0.0, C++20, and -O3. The archived Zig build procedure uses ReleaseFast and -mcpu=native. The two sessions fixed the resulting executables so rebuilds and changes in optimization settings would be outside the comparison.

Each round visited all four conditions in a seeded randomized order. A pair ran the two executables consecutively in separate processes, with one process finishing before the next began. Each build occupied the first position 10 times per condition on November 8 and 15 times on November 9. Balancing order gives equal weight to the two execution sequences. The second process still runs on the host state left by the first. The fixed schedule required valid clocks and intact inputs. We retained all timed processes.

The harness recorded power state and background activity just before and after each pair. It flagged a process using at least half a CPU core at the boundaries of two pairs on November 8 and four on November 9. These samples describe the host at the boundaries. They do not continuously measure competing activity during the load.

3.2 Estimation

Each process contributes one value. In warmed mode that value is the median of the three timed loads, so three loads in the same process do not count as three independent observations. We divide the C++ time by the Zig time within each pair and exponentiate the mean of the log ratios [1]. A ratio above 1 favors Zig. A ratio below 1 favors n0is C++. The two arm medians provide times in milliseconds. Their quotient and the geometric mean of paired ratios summarize the data differently.

For each condition, 50,000 seeded resamples draw complete pairs with replacement and recompute the geometric mean. The percentile endpoints use 0.625% in each tail, a Bonferroni adjustment across four contrasts within a session. This targets approximate 95% simultaneous coverage if pairs are exchangeable within each condition. The prespecified directional criterion calls for both first-load intervals above 1 and both warmed intervals below 1. An interval that contains 1 leaves the direction unresolved for that session and does not establish equivalence.

4. Results

4.1 November 9 load clocks

Three of four conditions met the directional criterion. At 1M rows, the paired estimates put Zig about 15% lower on first-load time and n0is C++ about 37% lower on warmed time. At 10M rows, n0is C++ was about 23% lower on warmed time. First-load wins split evenly and the interval spanned 1.

CSV load C++ (ms) Zig (ms) Paired ratio (interval) C++ wins Zig wins
1M, first load 5.714 5.122 1.174 (1.081–1.276) 7 23
1M, warmed 2.864 4.702 0.629 (0.606–0.661) 30 0
10M, first load 46.267 45.998 0.993 (0.952–1.016) 15 15
10M, warmed 32.337 41.617 0.775 (0.765–0.787) 30 0

Table 1. Times are medians across 30 processes per build and condition. The ratio and adjusted interval use within-pair times. Win counts describe the observed pairs. There were no ties.

At 10M the first-load interquartile ranges nearly coincide, while in warmed mode the slowest C++ observation, 35.340 ms, is still below the fastest Zig observation, 40.457 ms. The 1M warmed observations also have disjoint ranges. Figure 1 shows the corresponding spread within pairs.

CSV load C++ middle 50% (ms) Zig middle 50% (ms) C++ min to max (ms) Zig min to max (ms)
1M, first load 5.387–6.444 4.286–5.615 5.267–7.131 4.018–6.811
1M, warmed 2.816–2.919 4.570–4.790 2.756–3.948 4.095–5.004
10M, first load 45.602–46.632 45.515–46.623 44.617–49.419 44.107–65.982
10M, warmed 31.736–32.796 41.262–41.966 30.666–35.340 40.457–43.688

Table 2. The middle 50% spans the 25th through 75th percentiles. Ranges show the observed extrema.

Paired CSV load times across runs

Each dot represents one block with both implementations.

Paired CSV load times across runsEach row shows all paired observations and an adjusted interval. Ratios below 1 favor n0is C++, and ratios above 1 favor Zig. The values are in the table below.n0is C++ fasterZig faster0.40.60.811.21.41.61.81M · first load1M · first load, block 1. C++ 6.166 ms, Zig 4.98 ms. Ratio 1.2382.1M · first load, block 2. C++ 5.635 ms, Zig 5.875 ms. Ratio 0.9591.1M · first load, block 3. C++ 6.842 ms, Zig 4.85 ms. Ratio 1.4107.1M · first load, block 4. C++ 5.821 ms, Zig 5.738 ms. Ratio 1.0145.1M · first load, block 5. C++ 5.311 ms, Zig 4.469 ms. Ratio 1.1884.1M · first load, block 6. C++ 5.374 ms, Zig 5.09 ms. Ratio 1.0558.1M · first load, block 7. C++ 5.383 ms, Zig 6.811 ms. Ratio 0.7903.1M · first load, block 8. C++ 5.294 ms, Zig 4.2 ms. Ratio 1.2605.1M · first load, block 9. C++ 5.374 ms, Zig 5.275 ms. Ratio 1.0188.1M · first load, block 10. C++ 5.437 ms, Zig 5.136 ms. Ratio 1.0586.1M · first load, block 11. C++ 5.368 ms, Zig 5.238 ms. Ratio 1.0248.1M · first load, block 12. C++ 5.267 ms, Zig 4.136 ms. Ratio 1.2735.1M · first load, block 13. C++ 6.536 ms, Zig 4.225 ms. Ratio 1.5470.1M · first load, block 14. C++ 6.593 ms, Zig 4.126 ms. Ratio 1.5979.1M · first load, block 15. C++ 5.27 ms, Zig 4.05 ms. Ratio 1.3012.1M · first load, block 16. C++ 5.779 ms, Zig 6.333 ms. Ratio 0.9125.1M · first load, block 17. C++ 7.131 ms, Zig 5.131 ms. Ratio 1.3898.1M · first load, block 18. C++ 6.893 ms, Zig 5.313 ms. Ratio 1.2974.1M · first load, block 19. C++ 6.981 ms, Zig 4.092 ms. Ratio 1.7060.1M · first load, block 20. C++ 5.729 ms, Zig 5.086 ms. Ratio 1.1264.1M · first load, block 21. C++ 5.697 ms, Zig 5.759 ms. Ratio 0.9892.1M · first load, block 22. C++ 5.757 ms, Zig 6.086 ms. Ratio 0.9459.1M · first load, block 23. C++ 5.449 ms, Zig 5.112 ms. Ratio 1.0659.1M · first load, block 24. C++ 5.399 ms, Zig 4.039 ms. Ratio 1.3367.1M · first load, block 25. C++ 7.096 ms, Zig 5.142 ms. Ratio 1.3800.1M · first load, block 26. C++ 5.732 ms, Zig 5.893 ms. Ratio 0.9727.1M · first load, block 27. C++ 5.698 ms, Zig 5.715 ms. Ratio 0.9970.1M · first load, block 28. C++ 6.878 ms, Zig 5.167 ms. Ratio 1.3311.1M · first load, block 29. C++ 5.5 ms, Zig 4.756 ms. Ratio 1.1564.1M · first load, block 30. C++ 5.9 ms, Zig 4.018 ms. Ratio 1.4684.1M · first load, Nov 9, 1.174 (1.081 to 1.276)1M · warmed1M · warmed, block 1. C++ 2.837 ms, Zig 4.095 ms. Ratio 0.6928.1M · warmed, block 2. C++ 2.845 ms, Zig 4.688 ms. Ratio 0.6069.1M · warmed, block 3. C++ 2.992 ms, Zig 4.839 ms. Ratio 0.6183.1M · warmed, block 4. C++ 2.964 ms, Zig 4.872 ms. Ratio 0.6084.1M · warmed, block 5. C++ 2.815 ms, Zig 4.809 ms. Ratio 0.5854.1M · warmed, block 6. C++ 2.834 ms, Zig 4.688 ms. Ratio 0.6045.1M · warmed, block 7. C++ 2.917 ms, Zig 4.668 ms. Ratio 0.6249.1M · warmed, block 8. C++ 2.889 ms, Zig 4.482 ms. Ratio 0.6446.1M · warmed, block 9. C++ 3.948 ms, Zig 4.76 ms. Ratio 0.8294.1M · warmed, block 10. C++ 2.808 ms, Zig 4.69 ms. Ratio 0.5987.1M · warmed, block 11. C++ 2.814 ms, Zig 4.879 ms. Ratio 0.5768.1M · warmed, block 12. C++ 3.029 ms, Zig 4.311 ms. Ratio 0.7026.1M · warmed, block 13. C++ 3.582 ms, Zig 4.175 ms. Ratio 0.8580.1M · warmed, block 14. C++ 2.874 ms, Zig 4.565 ms. Ratio 0.6296.1M · warmed, block 15. C++ 2.867 ms, Zig 4.671 ms. Ratio 0.6138.1M · warmed, block 16. C++ 3.079 ms, Zig 4.773 ms. Ratio 0.6451.1M · warmed, block 17. C++ 2.797 ms, Zig 4.776 ms. Ratio 0.5856.1M · warmed, block 18. C++ 2.861 ms, Zig 4.257 ms. Ratio 0.6721.1M · warmed, block 19. C++ 2.802 ms, Zig 4.726 ms. Ratio 0.5929.1M · warmed, block 20. C++ 3.046 ms, Zig 4.713 ms. Ratio 0.6463.1M · warmed, block 21. C++ 2.775 ms, Zig 5.004 ms. Ratio 0.5546.1M · warmed, block 22. C++ 2.92 ms, Zig 4.716 ms. Ratio 0.6192.1M · warmed, block 23. C++ 2.833 ms, Zig 4.4 ms. Ratio 0.6439.1M · warmed, block 24. C++ 2.756 ms, Zig 4.585 ms. Ratio 0.6011.1M · warmed, block 25. C++ 2.809 ms, Zig 4.794 ms. Ratio 0.5859.1M · warmed, block 26. C++ 2.889 ms, Zig 4.666 ms. Ratio 0.6192.1M · warmed, block 27. C++ 2.881 ms, Zig 4.724 ms. Ratio 0.6099.1M · warmed, block 28. C++ 2.826 ms, Zig 4.544 ms. Ratio 0.6219.1M · warmed, block 29. C++ 2.818 ms, Zig 4.883 ms. Ratio 0.5771.1M · warmed, block 30. C++ 2.881 ms, Zig 4.842 ms. Ratio 0.5950.1M · warmed, Nov 9, 0.629 (0.606 to 0.661)10M · first load10M · first load, block 1. C++ 46.851 ms, Zig 45.515 ms. Ratio 1.0294.10M · first load, block 2. C++ 47.003 ms, Zig 44.107 ms. Ratio 1.0657.10M · first load, block 3. C++ 47.039 ms, Zig 46.537 ms. Ratio 1.0108.10M · first load, block 4. C++ 46.333 ms, Zig 49.14 ms. Ratio 0.9429.10M · first load, block 5. C++ 46.251 ms, Zig 47.185 ms. Ratio 0.9802.10M · first load, block 6. C++ 45.569 ms, Zig 46.545 ms. Ratio 0.9790.10M · first load, block 7. C++ 45.702 ms, Zig 45.516 ms. Ratio 1.0041.10M · first load, block 8. C++ 46.652 ms, Zig 46.824 ms. Ratio 0.9963.10M · first load, block 9. C++ 45.926 ms, Zig 46.122 ms. Ratio 0.9958.10M · first load, block 10. C++ 46.305 ms, Zig 45.423 ms. Ratio 1.0194.10M · first load, block 11. C++ 45.758 ms, Zig 45.898 ms. Ratio 0.9969.10M · first load, block 12. C++ 46.817 ms, Zig 45.359 ms. Ratio 1.0321.10M · first load, block 13. C++ 46.572 ms, Zig 45.633 ms. Ratio 1.0206.10M · first load, block 14. C++ 47.264 ms, Zig 45.842 ms. Ratio 1.0310.10M · first load, block 15. C++ 47.035 ms, Zig 44.644 ms. Ratio 1.0536.10M · first load, block 16. C++ 45.557 ms, Zig 46.334 ms. Ratio 0.9832.10M · first load, block 17. C++ 46.515 ms, Zig 44.802 ms. Ratio 1.0382.10M · first load, block 18. C++ 45.537 ms, Zig 45.151 ms. Ratio 1.0085.10M · first load, block 19. C++ 49.419 ms, Zig 46.808 ms. Ratio 1.0558.10M · first load, block 20. C++ 44.954 ms, Zig 45.531 ms. Ratio 0.9873.10M · first load, block 21. C++ 45.828 ms, Zig 45.894 ms. Ratio 0.9986.10M · first load, block 22. C++ 46.429 ms, Zig 46.277 ms. Ratio 1.0033.10M · first load, block 23. C++ 46.283 ms, Zig 46.806 ms. Ratio 0.9888.10M · first load, block 24. C++ 45.895 ms, Zig 44.861 ms. Ratio 1.0230.10M · first load, block 25. C++ 45.484 ms, Zig 46.097 ms. Ratio 0.9867.10M · first load, block 26. C++ 46.088 ms, Zig 46.919 ms. Ratio 0.9823.10M · first load, block 27. C++ 45.49 ms, Zig 46.305 ms. Ratio 0.9824.10M · first load, block 28. C++ 46.377 ms, Zig 45.726 ms. Ratio 1.0142.10M · first load, block 29. C++ 45.158 ms, Zig 65.982 ms. Ratio 0.6844.10M · first load, block 30. C++ 44.617 ms, Zig 46.649 ms. Ratio 0.9564.10M · first load, Nov 9, 0.993 (0.952 to 1.016)10M · warmed10M · warmed, block 1. C++ 32.021 ms, Zig 41.736 ms. Ratio 0.7672.10M · warmed, block 2. C++ 32.845 ms, Zig 42.956 ms. Ratio 0.7646.10M · warmed, block 3. C++ 31.725 ms, Zig 41.506 ms. Ratio 0.7643.10M · warmed, block 4. C++ 32.97 ms, Zig 41.184 ms. Ratio 0.8006.10M · warmed, block 5. C++ 32.059 ms, Zig 41.616 ms. Ratio 0.7704.10M · warmed, block 6. C++ 32.361 ms, Zig 43.359 ms. Ratio 0.7464.10M · warmed, block 7. C++ 32.528 ms, Zig 41.641 ms. Ratio 0.7812.10M · warmed, block 8. C++ 33.019 ms, Zig 41.394 ms. Ratio 0.7977.10M · warmed, block 9. C++ 32.541 ms, Zig 41.861 ms. Ratio 0.7774.10M · warmed, block 10. C++ 31.697 ms, Zig 41.759 ms. Ratio 0.7590.10M · warmed, block 11. C++ 30.949 ms, Zig 42.001 ms. Ratio 0.7369.10M · warmed, block 12. C++ 33.361 ms, Zig 43.688 ms. Ratio 0.7636.10M · warmed, block 13. C++ 33.191 ms, Zig 42.128 ms. Ratio 0.7879.10M · warmed, block 14. C++ 32.649 ms, Zig 42.354 ms. Ratio 0.7709.10M · warmed, block 15. C++ 30.849 ms, Zig 41.503 ms. Ratio 0.7433.10M · warmed, block 16. C++ 30.666 ms, Zig 41.135 ms. Ratio 0.7455.10M · warmed, block 17. C++ 31.216 ms, Zig 41.635 ms. Ratio 0.7498.10M · warmed, block 18. C++ 31.77 ms, Zig 42.015 ms. Ratio 0.7562.10M · warmed, block 19. C++ 32.36 ms, Zig 41.127 ms. Ratio 0.7868.10M · warmed, block 20. C++ 32.849 ms, Zig 40.902 ms. Ratio 0.8031.10M · warmed, block 21. C++ 34.42 ms, Zig 40.924 ms. Ratio 0.8411.10M · warmed, block 22. C++ 31.869 ms, Zig 41.054 ms. Ratio 0.7763.10M · warmed, block 23. C++ 32.611 ms, Zig 43.003 ms. Ratio 0.7583.10M · warmed, block 24. C++ 35.34 ms, Zig 41.642 ms. Ratio 0.8487.10M · warmed, block 25. C++ 31.987 ms, Zig 41.242 ms. Ratio 0.7756.10M · warmed, block 26. C++ 31.41 ms, Zig 41.321 ms. Ratio 0.7601.10M · warmed, block 27. C++ 32.293 ms, Zig 41.553 ms. Ratio 0.7772.10M · warmed, block 28. C++ 31.559 ms, Zig 40.457 ms. Ratio 0.7801.10M · warmed, block 29. C++ 32.583 ms, Zig 41.43 ms. Ratio 0.7865.10M · warmed, block 30. C++ 32.313 ms, Zig 41.618 ms. Ratio 0.7764.10M · warmed, Nov 9, 0.775 (0.765 to 0.787)Elapsed-time ratio (n0is C++ / Zig)Geometric mean and adjusted 95% interval
View individual paired timings
Per-block process observations used in the chart
ConditionBlockC++ (ms)Zig (ms)Ratio
1M · first load16.1664.981.2382
1M · first load25.6355.8750.9591
1M · first load36.8424.851.4107
1M · first load45.8215.7381.0145
1M · first load55.3114.4691.1884
1M · first load65.3745.091.0558
1M · first load75.3836.8110.7903
1M · first load85.2944.21.2605
1M · first load95.3745.2751.0188
1M · first load105.4375.1361.0586
1M · first load115.3685.2381.0248
1M · first load125.2674.1361.2735
1M · first load136.5364.2251.5470
1M · first load146.5934.1261.5979
1M · first load155.274.051.3012
1M · first load165.7796.3330.9125
1M · first load177.1315.1311.3898
1M · first load186.8935.3131.2974
1M · first load196.9814.0921.7060
1M · first load205.7295.0861.1264
1M · first load215.6975.7590.9892
1M · first load225.7576.0860.9459
1M · first load235.4495.1121.0659
1M · first load245.3994.0391.3367
1M · first load257.0965.1421.3800
1M · first load265.7325.8930.9727
1M · first load275.6985.7150.9970
1M · first load286.8785.1671.3311
1M · first load295.54.7561.1564
1M · first load305.94.0181.4684
1M · warmed12.8374.0950.6928
1M · warmed22.8454.6880.6069
1M · warmed32.9924.8390.6183
1M · warmed42.9644.8720.6084
1M · warmed52.8154.8090.5854
1M · warmed62.8344.6880.6045
1M · warmed72.9174.6680.6249
1M · warmed82.8894.4820.6446
1M · warmed93.9484.760.8294
1M · warmed102.8084.690.5987
1M · warmed112.8144.8790.5768
1M · warmed123.0294.3110.7026
1M · warmed133.5824.1750.8580
1M · warmed142.8744.5650.6296
1M · warmed152.8674.6710.6138
1M · warmed163.0794.7730.6451
1M · warmed172.7974.7760.5856
1M · warmed182.8614.2570.6721
1M · warmed192.8024.7260.5929
1M · warmed203.0464.7130.6463
1M · warmed212.7755.0040.5546
1M · warmed222.924.7160.6192
1M · warmed232.8334.40.6439
1M · warmed242.7564.5850.6011
1M · warmed252.8094.7940.5859
1M · warmed262.8894.6660.6192
1M · warmed272.8814.7240.6099
1M · warmed282.8264.5440.6219
1M · warmed292.8184.8830.5771
1M · warmed302.8814.8420.5950
10M · first load146.85145.5151.0294
10M · first load247.00344.1071.0657
10M · first load347.03946.5371.0108
10M · first load446.33349.140.9429
10M · first load546.25147.1850.9802
10M · first load645.56946.5450.9790
10M · first load745.70245.5161.0041
10M · first load846.65246.8240.9963
10M · first load945.92646.1220.9958
10M · first load1046.30545.4231.0194
10M · first load1145.75845.8980.9969
10M · first load1246.81745.3591.0321
10M · first load1346.57245.6331.0206
10M · first load1447.26445.8421.0310
10M · first load1547.03544.6441.0536
10M · first load1645.55746.3340.9832
10M · first load1746.51544.8021.0382
10M · first load1845.53745.1511.0085
10M · first load1949.41946.8081.0558
10M · first load2044.95445.5310.9873
10M · first load2145.82845.8940.9986
10M · first load2246.42946.2771.0033
10M · first load2346.28346.8060.9888
10M · first load2445.89544.8611.0230
10M · first load2545.48446.0970.9867
10M · first load2646.08846.9190.9823
10M · first load2745.4946.3050.9824
10M · first load2846.37745.7261.0142
10M · first load2945.15865.9820.6844
10M · first load3044.61746.6490.9564
10M · warmed132.02141.7360.7672
10M · warmed232.84542.9560.7646
10M · warmed331.72541.5060.7643
10M · warmed432.9741.1840.8006
10M · warmed532.05941.6160.7704
10M · warmed632.36143.3590.7464
10M · warmed732.52841.6410.7812
10M · warmed833.01941.3940.7977
10M · warmed932.54141.8610.7774
10M · warmed1031.69741.7590.7590
10M · warmed1130.94942.0010.7369
10M · warmed1233.36143.6880.7636
10M · warmed1333.19142.1280.7879
10M · warmed1432.64942.3540.7709
10M · warmed1530.84941.5030.7433
10M · warmed1630.66641.1350.7455
10M · warmed1731.21641.6350.7498
10M · warmed1831.7742.0150.7562
10M · warmed1932.3641.1270.7868
10M · warmed2032.84940.9020.8031
10M · warmed2134.4240.9240.8411
10M · warmed2231.86941.0540.7763
10M · warmed2332.61143.0030.7583
10M · warmed2435.3441.6420.8487
10M · warmed2531.98741.2420.7756
10M · warmed2631.4141.3210.7601
10M · warmed2732.29341.5530.7772
10M · warmed2831.55940.4570.7801
10M · warmed2932.58341.430.7865
10M · warmed3032.31341.6180.7764
Figure 1. Each dot is one pair’s C++ time divided by Zig time. Vertical offsets separate nearby observations. Whiskers show adjusted intervals around the paired geometric means. The line at 1 marks equal elapsed time.

4.2 Repeatability across sessions

The November 8 pairwise session met the directional criterion in all four conditions. Zig won all 20 pairs on the first 10M load. On November 9 the estimate for that condition moved from 1.072 to 0.993 and its adjusted interval included 1. The 1M first-load estimates, by comparison, remained near 1.18 in both sessions. The warmed estimates stayed below 1 at both sizes even as their magnitudes changed. C++ won every warmed pair in each session. These observations support three directional findings across the two sessions and leave the first 10M load sensitive to the collection.

Paired estimates across two sessions

Intervals were adjusted within each session and analyzed separately.

Paired estimates across two sessionsThe earlier estimate sits above the later estimate in each row. Ratios below 1 favor n0is C++, and ratios above 1 favor Zig. The later 10M first-load interval crosses 1.n0is C++ fasterZig faster0.40.60.811.21.41M · first load1M · first load, Nov 8, 1.180 (1.068 to 1.309)1M · first load, Nov 9, 1.174 (1.081 to 1.276)1M · warmed1M · warmed, Nov 8, 0.694 (0.679 to 0.708)1M · warmed, Nov 9, 0.629 (0.606 to 0.661)10M · first load10M · first load, Nov 8, 1.072 (1.058 to 1.085)10M · first load, Nov 9, 0.993 (0.952 to 1.016)10M · warmed10M · warmed, Nov 8, 0.788 (0.778 to 0.798)10M · warmed, Nov 9, 0.775 (0.765 to 0.787)Elapsed-time ratio (n0is C++ / Zig)Nov 8 · 20 pairs per conditionNov 9 · 30 pairs per condition
Figure 2. Paired geometric means and adjusted intervals for the November 8 and November 9 sessions. Each session uses its own four-contrast family for approximate 95% simultaneous coverage. The two sessions are analyzed separately.

4.3 Execution order and influential pairs

The 1M first-load comparison varied substantially with execution order. In both sessions Zig had a larger lead when it ran second. The estimate was close to equal time when Zig ran first. Each descriptive split uses half the pairs from its session. The primary estimate describes a schedule with equal counts of both sequences. The split does not isolate a host mechanism or establish that running second has the same effect on both programs.

Session First process in pair Pairs Paired C++/Zig ratio Zig wins
November 8 C++ 10 1.384 10
November 8 Zig 10 1.006 7
November 9 C++ 15 1.364 15
November 9 Zig 15 1.010 8

Table 3. Exploratory order splits for the 1M first load. The ratios use the same paired geometric mean as the primary analysis.

For the 10M first load on November 9, Zig’s longest observation was 65.982 ms against 45.158 ms for C++ in the same pair. Both activity snapshots were below the logging threshold. Recomputing the geometric mean after leaving out each of the 30 pairs in turn gives estimates from 0.990 to 1.005. The upper endpoint omits that longest Zig observation. The sign can therefore change by omitting a single pair, while all of these descriptive point estimates remain near 1. The reported interval and win counts retain all 30 pairs.

The first-load timing records also identify where to look for differences. For the 10M fixture, Zig’s median reported parse phase was 38.635 ms on November 8 and 40.256 ms on November 9. The C++ medians were 37.140 ms and 36.596 ms. Zig’s longest observation recorded 55.060 ms in its parse phase. These phase fields locate variation within the measured load, while separate phase medians cannot be summed to recover a median total. The records do not explain what caused the phase timing to move.

5. Interpretation and replication

Benchmarking work recommends specifying the work and checking outputs before comparing systems [3]. In this study, keeping those contracts fixed and separating the two load clocks exposes a reliable warmed ordering alongside a less stable first-load comparison. For an interactive workflow that reuses a running process, the warmed results give a stronger reason to investigate the C++ path. For a workflow that launches a process per request, the internal first-load clock answers a narrower question than launch-to-result latency. The 10M results demonstrate why both clocks should be reported if the application cares about both.

5.1 Scope and follow-up

The synthetic fixtures share one schema, and the sessions reuse the same builds on one host. The study therefore cannot distinguish effects of implementation, compiler, or scheduling policy. A follow-up could record launch, input acquisition, pool readiness, parsing, and materialization boundaries under the same workload, then change one condition at a time. In particular, an explicit readiness checkpoint would test whether the first-load results change when both pools are ready before timing. A replication should also compare canonical typed and dictionary outputs from the timed paths, then extend to a second host and CSVs with different row widths and quoting patterns.

The research archive retains every per-pair timing receipt, the frozen schedules and artifact digests, the correctness receipts, and the seeded analysis. The public plot lists the November 9 pairs so readers can inspect the spread and reconstruct the point estimates. Repeating acquisition requires the private n0is executable and the matching port and fixtures.

Further reading

References

  1. Kalibera, T., and Jones, R. Quantifying Performance Changes with Effect Size Confidence Intervals. arXiv, 2020.
  2. Stehle, E., and Jacobsen, H.-A. ParPaRaw: Massively Parallel Parsing of Delimiter-Separated Raw Data. PVLDB, 2020.
  3. Raasveldt, M., Holanda, P., Gubner, T., and Mühleisen, H. Fair Benchmarking Considered Difficult: Common Pitfalls in Database Performance Testing. DBTest, 2018.