Back to home
Performance test_junkie 0.9a5 Measured 2026-10-06 · v0.9a5

Test Junkie performance

How long Test Junkie takes to run 1, 100 and 1,000 tests, one at a time and on 2, 5 and 10 threads, and how much time the framework itself adds per test. Measured with test_junkie 0.9a5 on Python 3.12.5 on 2026-10-06; every number is the median of 30 runs. How it was measured.

1,000 tests · 1 thread 1,972 ms Process start to exit, sequential run, 1 ms test body.
1,000 tests · 10 threads 308 ms Parametrized test with parallelized_parameters=True.
Speedup · 10 threads 6.22 1,000 parametrized tests: 1,917 ms on 1 thread, 308 ms on 10.
Overhead per test ~1 ms +972 ms on top of the 1,000 test bodies, sequential run.

One thread: wall clock and overhead

Wall clock is the real time from process start to exit, measured outside the process by the benchmark harness. Lower is better.

Overhead is wall clock minus the time spent in test bodies (N tests of 1 ms each): what Test Junkie adds for start-up, collection and per-test bookkeeping such as result tracking and listener events.

The sequential run has N separate test functions; the parametrized run has one test with N parameters (@test(parameters=...)). At 1,000 tests the overhead works out to about 1 ms per test.

Wall clock and overhead on one thread, test_junkie 0.9a5, median of 30 runs
Tests (N) Sequential Overhead Parametrized
195.3 ms+94 ms93.2 ms
100296.2 ms+196 ms290.2 ms
1,0001,972.1 ms+972 ms1,917.4 ms

Scaling with threads

One parametrized test with N parameters, run with parallelized_parameters=True and test_multithreading_limit set to the thread count. Tests run on threads inside one process. Speedup = the 1-thread wall clock ÷ the wall clock at that thread count; higher is better.

1,000 tests · wall clock by thread count
1 thread1,917.4 ms
2 threads867.3 ms
5 threads415.1 ms
10 threads308.3 ms

Bar length is wall clock; shorter is better. The 1-thread bar is the parametrized run from the table above.

Wall clock and speedup by thread count, test_junkie 0.9a5, median of 30 runs
Threads N = 100 Speedup N = 1,000 Speedup
1290.2 msbaseline1,917.4 msbaseline
2173.2 ms1.68867.3 ms2.21
5127.4 ms2.28415.1 ms4.62
10116.7 ms2.49308.3 ms6.22

At 100 tests the run is short enough that start-up (about 87 ms, see below) is a large share of it, so extra threads help less. At 1,000 tests the speedup keeps growing up to 10 threads.

Where the time goes

Collection is importing the suite and building the Runner, up to the point where tests start. It stays near 87–89 ms from 1 to 1,000 tests, and it's most of the 95 ms a one-test run takes.

Call is the average time per test from the on_in_progress listener event to on_success, so it includes the 1 ms test body and listener dispatch. Setup and teardown aren't separate events in the Listener API (@beforeTest / @afterTest run your own code), so they aren't timed separately.

Collection time and average call time per test, sequential run, test_junkie 0.9a5, median of 30 runs
Tests (N) Collection Avg call / test
187.4 ms1.351 ms
10086.7 ms1.800 ms
1,00089.1 ms1.571 ms

How it was measured

Version
test-junkie 0.9a5, no other packages
Measured
2026-10-06
Python
CPython 3.12.5
OS
Windows 11 Pro
CPU
Intel Core i9-10980HK, 8 cores / 16 threads, 2.40 GHz base
RAM
32 GB
Test body
time.sleep(0.001), so the framework's own time is what's left
Sizes
1, 100 and 1,000 tests; 1, 2, 5 and 10 threads
Runs
30 consecutive runs per scenario, warm filesystem; the median is reported
Check
A run only counts if Test Junkie's own summary reports all N tests passed; otherwise the benchmark stops

Reproduce it

  1. Generate one Python file per scenario. Sequential and parametrized files end with Runner([BenchSuite]).run(); parallel files look like the one below, with N parameters and test_multithreading_limit=W.
  2. Run each file as its own process (python bench_1000_10.py) and time it from outside, start to exit, with subprocess.Popen.
  3. Do 30 runs per scenario and take the median. For collection, time a file that builds Runner([BenchSuite]) without calling .run().
bench_1000_10.pypython
import time

from test_junkie.decorators import Suite, test
from test_junkie.runner import Runner


@Suite()
class BenchSuite:

    @test(parameters=list(range(1000)), parallelized_parameters=True)
    def bench(self, parameter):
        time.sleep(0.001)  # the whole test body: 1 ms


Runner([BenchSuite]).run(test_multithreading_limit=10)
Install size What pip installs for Test Junkie: the package, its four dependencies and the size of each.
← Back to Test Junkie home View on GitHub →