Performance Benchmark

Methodology

Python

3.12.5

Platform

Windows 11 Pro

Runs per scenario

100 — median of 100 consecutive runs reported

Test body

time.sleep(0.001) — 1 ms sleep, isolates framework overhead

Test counts

1 · 100 · 1,000

Worker counts

1 · 2 · 5 · 10 threads

Invocation commands

Each framework was invoked via subprocess.Popen from the benchmark harness. The commands below are what the harness executed for each scenario. N = test count, W = worker count.

Packages installed

Each framework was installed with only what it needs to support the tested feature set: parallel execution and parametrized tests. No extras.

unittest parallel execution is handled via ThreadPoolExecutor with unittest.TestLoader — no package provides a clean test-level parallel runner for unittest, so threading was wired manually in the benchmark harness consistent with how it would be used in practice.

Package footprint

Total installed size including all required plugins to reach feature parity with test_junkie (parallel + parametrization).

Wall clock — sequential

Single-threaded execution, one test at a time. Wall clock is the total real-world time from process start to exit, measured externally by the benchmark harness. Overhead = wall clock − (N × 1 ms) — it isolates framework startup, collection, and per-test bookkeeping from the test body itself. Negative overhead means the OS timer resolution made the measured sleep total slightly shorter than N × 1 ms; treat it as zero.

Each framework is invoked once per scenario with all N tests in a single run — the subprocess approach is consistent across all frameworks and matches real-world CLI usage. pytest and Robot Framework are genuinely slower at scale because their per-test internal machinery (hook dispatch, fixture bookkeeping, result recording) carries higher overhead per test than unittest or test_junkie. At N=1,000 that overhead compounds to nearly 2 seconds for pytest. unittest and test_junkie keep overhead near zero regardless of test count.

Wall clock — parametrized

Single-threaded; N parametrized test variants generated via each framework's native parametrization mechanism.

Wall clock — parallel parametrized

N parametrized variants distributed across W worker threads/processes. No N=1 row — parallelism only measured at 100 and 1,000 tests.

pytest-xdist spawns new interpreter processes per worker — process startup overhead dominates at all test counts, which explains its high parallel wall clock even at N=1,000. Robot Framework's pabot uses subprocesses similarly but has far less per-process overhead, producing consistent ~320 ms regardless of worker count. test_junkie uses threads, so startup cost is near-zero. unittest's ThreadPoolExecutor wins at small N + high W because thread creation is cheap and its per-test overhead is minimal.

Per-phase breakdown — sequential

Collection = time from session start until tests begin executing. Call = average per-test execution time. Not all phases are measurable for all frameworks via the introspection API used.

test_junkie's Listener is an outcome-notification API — it fires events like on_success and on_failure when a test completes, not phase-wrapping hooks that bracket setup, call, and teardown independently. There is no hook that fires exactly at "setup ends / call begins" or "call ends / teardown begins", so the three phases cannot be timed in isolation without modifying the test methods themselves. pytest's hookwrapper mechanism wraps each phase as a separate generator, which is why its numbers are clean. The benchmark attempted to use on_before_test / on_success as approximate call-phase markers, but on_before_test is not a valid Listener event in test_junkie 0.9a0, so the timing was never started and the data came out empty.

Parallel speedup vs sequential

Speedup = sequential wall clock ÷ parallel wall clock. Values >1 mean parallel was faster; values <1 mean parallel was slower (startup overhead exceeded gains).

Robot Framework achieves the highest speedup ratios because its sequential baseline is slow (~3.3 s for 1,000 parametrized tests) — even modest parallel scaling produces impressive-looking multipliers. test_junkie's sequential baseline is already fast enough that parallelism yields near-neutral results for a 1 ms test body; threading overhead roughly cancels the concurrency gain on this hardware. pytest-xdist is net-negative at all N=100 scenarios — process-spawn cost exceeds 3 s per run; at N=1,000 it barely breaks even at W=2 (×1.07) before becoming net-negative again at W=10 (×0.73).

Key takeaways