Random order and --seed

Tests that only pass in a certain order hide real bugs: shared state, leftover data, a cache that one test warms for the next. Shuffling the order finds them; the seed lets you repeat the order that failed.

An order-dependent test

TestOrder.RANDOM shuffles the tests of a suite on every run:

cart_suite.pypython

from test_junkie.constants import TestOrder
from test_junkie.decorators import Suite, beforeClass, test


@Suite(order=TestOrder.RANDOM)
class CartSuite:

    @beforeClass()
    def setup(self):
        self.cart = Cart()

    @test()
    def add_item(self):
        self.cart.add("book")
        assert len(self.cart) == 1

    @test()
    def checkout_total(self):
        # passes only when add_item ran first - the hidden dependency
        assert self.cart.total() == 12.99

In declaration order this suite always passes. Shuffled, checkout_total fails whenever it runs first.

The header prints the seed

Every run with a shuffled suite prints the seed that produced its order:

tj runoutput

Test Junkie 0.9a6 · Python 3.12.5 · Linux
  tests    tests · 1 suite, 2 tests · found in 0.01s
  mode     sequential
  order    random · seed 583207  repeat this order with --seed 583207

Repeat the order that failed

Pass the seed back and the tests run in exactly the same order, so the failure happens again while you fix it:

terminalbash

tj run --seed 583207

From Python, give the runner the seed:

run.pypython

from test_junkie.runner import Runner

Runner([CartSuite], seed=583207).run()

Same seed, same order

The seed fixes the order of every shuffled suite in the run. Filtering with -t or tags keeps the remaining tests in the same relative order, so you can narrow a failure down to the two tests involved.

Fix the dependency, not the order

Once you can repeat it, give each test what it needs instead of relying on another test:

cart_suite.pypython

@Suite(order=TestOrder.RANDOM)
class CartSuite:

    @beforeTest()
    def fresh_cart(self):
        self.cart = Cart()  # every test starts from an empty cart

    @test()
    def add_item(self):
        self.cart.add("book")
        assert len(self.cart) == 1

    @test()
    def checkout_total(self):
        self.cart.add("book")
        assert self.cart.total() == 12.99

Shuffle in CI, not just locally

A suite that runs in the same order every time can't tell you it has hidden dependencies. Keeping TestOrder.RANDOM on suites in CI finds them early, and the seed in the log makes any failure repeatable.