How long Test Junkie takes to run 1, 100 and 1,000 tests, one at a time and on 2, 5 and 10 threads, and how much time the framework itself adds per test. Measured with test_junkie 0.9a5 on Python 3.12.5 on 2026-10-06; every number is the median of 30 runs. How it was measured.
parallelized_parameters=True.
Wall clock is the real time from process start to exit, measured outside the process by the benchmark harness. Lower is better.
Overhead is wall clock minus the time spent in test bodies (N tests of 1 ms each): what Test Junkie adds for start-up, collection and per-test bookkeeping such as result tracking and listener events.
The sequential run has N separate test functions; the parametrized run has one test with N parameters (@test(parameters=...)). At 1,000 tests the overhead works out to about 1 ms per test.
| Tests (N) | Sequential | Overhead | Parametrized |
|---|---|---|---|
| 1 | 95.3 ms | +94 ms | 93.2 ms |
| 100 | 296.2 ms | +196 ms | 290.2 ms |
| 1,000 | 1,972.1 ms | +972 ms | 1,917.4 ms |
One parametrized test with N parameters, run with parallelized_parameters=True and test_multithreading_limit set to the thread count. Tests run on threads inside one process. Speedup = the 1-thread wall clock ÷ the wall clock at that thread count; higher is better.
| Threads | N = 100 | Speedup | N = 1,000 | Speedup |
|---|---|---|---|---|
| 1 | 290.2 ms | baseline | 1,917.4 ms | baseline |
| 2 | 173.2 ms | 1.68 | 867.3 ms | 2.21 |
| 5 | 127.4 ms | 2.28 | 415.1 ms | 4.62 |
| 10 | 116.7 ms | 2.49 | 308.3 ms | 6.22 |
At 100 tests the run is short enough that start-up (about 87 ms, see below) is a large share of it, so extra threads help less. At 1,000 tests the speedup keeps growing up to 10 threads.
Collection is importing the suite and building the Runner, up to the point where tests start. It stays near 87–89 ms from 1 to 1,000 tests, and it's most of the 95 ms a one-test run takes.
Call is the average time per test from the on_in_progress listener event to on_success, so it includes the 1 ms test body and listener dispatch. Setup and teardown aren't separate events in the Listener API (@beforeTest / @afterTest run your own code), so they aren't timed separately.
| Tests (N) | Collection | Avg call / test |
|---|---|---|
| 1 | 87.4 ms | 1.351 ms |
| 100 | 86.7 ms | 1.800 ms |
| 1,000 | 89.1 ms | 1.571 ms |
test-junkie 0.9a5, no other packagestime.sleep(0.001), so the framework's own time is what's leftRunner([BenchSuite]).run(); parallel files look like the one below, with N parameters and test_multithreading_limit=W.python bench_1000_10.py) and time it from outside, start to exit, with subprocess.Popen.Runner([BenchSuite]) without calling .run().import time from test_junkie.decorators import Suite, test from test_junkie.runner import Runner @Suite() class BenchSuite: @test(parameters=list(range(1000)), parallelized_parameters=True) def bench(self, parameter): time.sleep(0.001) # the whole test body: 1 ms Runner([BenchSuite]).run(test_multithreading_limit=10)