Documentation

Command Line Interface since 0.8a.8

Reading the output since 0.9a6

All three commands print in the same style. Colors carry the same meaning everywhere: green passed or saved, yellow failed, red an error, blue skipped or restored, orange ignored or missing, gray cancelled. Colors are on in a terminal and in GitHub Actions; set NO_COLOR to turn them off, or FORCE_COLOR to keep them when output goes to a file. Errors print a red ERROR line with what to type instead, and exit with code 120.

tj run new since 0.8a.8

Executes tests without requiring any additional code. Accepts the same options as Runner.run(), all settable from the terminal:

# Run everything under --sources with default settings
tj run

# 2 suite threads, 10 test threads, scoped to 2FA and OAuth components
tj run -c 2FA OAuth -T 10 -S 2

# Only problems and the result line, resource monitoring, HTML report
tj run -q -m --html-report "/path/to/report.html"

# One line per test, and test output shown live (for breakpoint() / pdb)
tj run -p --no-capture

# Load from a config file (explicit CLI args still take precedence)
tj run --config "/path/to/your_config.cfg"

These examples assume --sources is already saved in your config (see tj config below). Full reference: tj run -h.

$ tj run -s tests
Test Junkie 0.9a6 · Python 3.12.5 · Windows 11
  tests    tests · 5 suites, 44 tests · found in 0.14s
  mode     sequential

LoginSuite     tests/ui/login.py     [||||||||||||||||||||||||]  10/10  4.12s
CheckoutSuite  tests/ui/checkout.py  [||||||||||||||||||||||||]  11/11  3.05s
PaymentsSuite  tests/api/payments.py [||||||||||||||||||||||||]    6/6  0.03s
SearchSuite    tests/api/search.py   [||||||||||||||||||||||||]    9/9  0.84s
ReportsSuite   tests/api/reports.py  [||||||||||||||||||||||||]    8/8  1.20s

Total                                [||||||||||||||||||||||||]  44/44  100%  9.24s

Problems 4 ─────────────────────────────────────────────────────────────────────

 FAIL  LoginSuite.rejects_bad_password  [user='admin']
       tests/ui/login.py:42 · failed all 3 runs · 1.83s

  Runs
    #1  ERROR  0.61s  TimeoutException: login form not loaded after 10s
    #2  FAIL   0.60s  AssertionError: expected 'Invalid password', got ''
    #3  FAIL   0.62s  AssertionError: expected 'Invalid password', got ''

  Run #1  traceback
  │ tests/ui/login.py:38 in rejects_bad_password
  │     page = LoginPage.open(self.driver)
  │ tests/ui/pages.py:17 in open
  │     WebDriverWait(driver, 10).until(form_visible)
  │ selenium.common.exceptions.TimeoutException: login form not loaded after 10s

  Runs #2–#3  same traceback both times
  │ tests/ui/login.py:42 in rejects_bad_password
  │     assert page.error_text() == "Invalid password"
  │ AssertionError: expected 'Invalid password', got ''

  Output  run #3
  │ opening https://staging.example.com/login
  │ submitting form as admin

  Log  run #3
  │ WARNING  selenium.webdriver.remote  element #error not visible after 5.0s


 ERROR  ReportsSuite.export_csv
        tests/api/reports.py:88 · 1 run · 0.12s

  Traceback
  │ tests/api/reports.py:88 in export_csv
  │     rows = client.get("/reports/export").json()
  │ tests/api/client.py:31 in get
  │     return self.session.get(self.base + path, timeout=5)
  │ requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080):
  │ Max retries exceeded with url: /reports/export

 IGNORED  PaymentsSuite  whole suite
          tests/api/payments.py:14 · @beforeClass failed · 6 tests not run

  Traceback
  │ tests/api/payments.py:14 in before_class
  │     self.gateway = Gateway.connect(sandbox=True)
  │ tests/api/gateway.py:22 in connect
  │     raise GatewayUnavailable(self.url)
  │ GatewayUnavailable: https://sandbox.pay.example.com is not responding


 IGNORED  SearchSuite.filters_by_date
          tests/api/search.py:57 · bad parameters · not run

  │ BadParameters: "parameters" in @test() must be a list, but found: dict
  │ https://www.test-junkie.com/documentation/#test-decorator

Summary ────────────────────────────────────────────────────────────────────────

  Suite             Pass  Fail  Error  Ignore  Skip  Cancel     Time
  LoginSuite           9     1      ·       ·     ·       ·    4.12s
  CheckoutSuite       10     ·      ·       ·     1       ·    3.05s
  PaymentsSuite        ·     ·      ·       6     ·       ·    0.03s
  SearchSuite          8     ·      ·       1     ·       ·    0.84s
  ReportsSuite         6     ·      1       ·     1       ·    1.20s
  ──────────────────────────────────────────────────────────────────
  Total               33     1      1       7     2       ·    9.24s
  44 tests           75%    2%     2%     16%    5%

────────────────────────────────────────────────────────────────────────────────

 FAILED   exit code 1

$ tj run -s tests -q
Problems 4 ─────────────────────────────────────────────────────────────────────

 FAIL  LoginSuite.rejects_bad_password  [user='admin']
       tests/ui/login.py:42 · failed all 3 runs · 1.83s
  … runs, tracebacks, output and log as in the default view …

 ERROR  ReportsSuite.export_csv
        tests/api/reports.py:88 · 1 run · 0.12s
  … traceback …

 IGNORED  PaymentsSuite  whole suite
          tests/api/payments.py:14 · @beforeClass failed · 6 tests not run
  … traceback …

 IGNORED  SearchSuite.filters_by_date
          tests/api/search.py:57 · bad parameters · not run
  … error …

 FAILED   1 failed, 1 error, 7 ignored · 33 passed, 2 skipped · 9.21s · exit code 1

$ tj run -s tests/ui/checkout.py -q
 PASSED   10 passed, 1 skipped · 3.05s

$ tj run -s tests -p
Test Junkie 0.9a6 · Python 3.12.5 · Windows 11
  tests    tests · 5 suites, 44 tests · found in 0.14s
  mode     sequential · one line per test

tests/ui/login.py::LoginSuite
  PASSED   accepts_valid_user                                    0.41s  [  2%]
  PASSED   accepts_valid_user[user='qa_basic']                   0.38s  [  5%]
  PASSED   remembers_session                                     0.52s  [  7%]
  …
  FAILED   rejects_bad_password[user='admin']        run 3/3     0.62s  [ 24%]
  PASSED   logout_clears_cookie                                  0.29s  [ 26%]

tests/ui/checkout.py::CheckoutSuite
  PASSED   adds_item_to_cart                                     0.33s  [ 29%]
  …
  SKIPPED  pays_with_paypal                          skip=True          [ 45%]
  PASSED   applies_coupon                            passed on run 2    0.71s  [ 53%]
  …

Test Junkie 0.9a6 · Python 3.12.5 · Windows 11
  tests    tests · 5 suites, 44 tests · found in 0.14s
  mode     parallel · 4 suite threads · 10 test threads

LoginSuite     tests/ui/login.py     [||||||||||||||··········]   6/10  cancelling
CheckoutSuite  tests/ui/checkout.py  [|||||||||||·············]   5/11  cancelling
PaymentsSuite  tests/api/payments.py [||||||||||||||||||||||||]    6/6  0.03s
SearchSuite    tests/api/search.py   [||||||||||||||||||||||||]    9/9  0.84s
ReportsSuite   tests/api/reports.py  [||||||||||||||||||||||||]    8/8  1.20s

Total                                [|||||||||||||||||||·····]  34/44   77%  1.9s

 CANCELLING   Ctrl+C received. No new tests will start and nothing is retried.
               Waiting for 3 tests in progress to finish, then cleanup (@afterTest, @afterClass, rules).
                 › LoginSuite.remembers_session  running 1.2s
                 › CheckoutSuite.applies_coupon  running 0.4s
                 › CheckoutSuite.pays_with_card  running 0.1s
               Press Ctrl+C again to stop immediately. Cleanup and reports will be skipped.

 CANCELLING   Ctrl+C received. No new tests will start and nothing is retried.
               Waiting for 1 test in progress to finish, then cleanup (@afterTest, @afterClass, rules).
                 › LoginSuite.remembers_session  running 31.4s
               Press Ctrl+C again to stop immediately. Cleanup and reports will be skipped.
^C
 STOPPED   Second Ctrl+C: stopped immediately.
            1 test was still running: LoginSuite.remembers_session (31.4s).
            Cleanup and reports were skipped. Browsers or other resources it opened may still be open.
            exit code 130

Picking tests since 0.9a6

-t takes test names, Suite.test names and patterns. -x skips suites by class name or pattern. Tag options read the same in every command: -k --tags-any / -l --tags-all for any or all of the tags, -g --skip-tags-any / -j --skip-tags-all to skip them.

# every test whose name starts with login_
tj run -t "login_*"

# one test in one suite
tj run -t CheckoutSuite.applies_coupon

# everything except the legacy suites
tj run -x "Legacy*"

# tests tagged smoke and fast
tj run -l smoke fast

Options use dashes now (--test-multithreading-limit). The old spellings, like --test_multithreading_limit, still work but aren't listed in -h.

Random order and --seed since 0.9a6

When a suite uses @Suite(order=TestOrder.RANDOM), the header prints the seed of the order. Pass it back with --seed to run the tests in exactly the same order, e.g. to reproduce a failure that depends on order.

Test Junkie 0.9a6 · Python 3.12.5 · Windows 11
  tests    tests · 1 suite, 8 tests · found in 0.00s
  mode     sequential
  order    random · seed 4242  repeat this order with --seed 4242

Retries for one run since 0.9a6

These options change retries for one run, without touching the code:

• --retry N lets every failing test run up to N times, whatever its @test(retry=) says.
• --no-retry runs every test and suite once, e.g. to see the real failure rate.
• --retry-policy module:Class gives every test without a retry of its own a retry policy.

The header shows which one is on.

Flaky tests in one run new since 0.9a8

A test that failed and then passed on a retry is flaky. It counts as passed unless you ask otherwise:

• No flag: with -p its line reads PASSED with passed on run 2, and the run passes.
• --flag-flaky: with -p it shows as FLAKY, and it is listed after the summary with the run it passed on and the first error. The run still passes.
• --fail-on-flaky: the same, plus a Problems entry for each flaky test (every run and the traceback of each failed run), and the run fails with exit code 1.

$ tj run -s tests -p
...
CheckoutSuite  tests\checkout.py
  PASSED     adds_item_to_cart                                                  0.00s  [ 33%]
  PASSED     pays_with_card                                passed on run 2      0.00s  [ 66%]
  PASSED     shows_receipt                                                      0.00s  [100%]
...
 PASSED   exit code 0

$ tj run -s tests -p --flag-flaky
...
CheckoutSuite  tests\checkout.py
  PASSED     adds_item_to_cart                                                  0.00s  [ 33%]
  FLAKY      pays_with_card                                passed on run 2      0.00s  [ 66%]
  PASSED     shows_receipt                                                      0.00s  [100%]
...
Flaky 1 ────────────────────────────────────────────────────────────────────────

  CheckoutSuite.pays_with_card   passed on run 2
      TimeoutError: payment page not loaded after 10s

 PASSED   exit code 0

$ tj run -s tests -p --fail-on-flaky
...
  FLAKY      pays_with_card                                passed on run 2      0.00s  [ 66%]
...
Problems 1 ─────────────────────────────────────────────────────────────────────

 FLAKY  CheckoutSuite.pays_with_card
        tests\checkout.py:17 · passed on run 2 (--fail-on-flaky) · 0.00s

  Runs
    #1  ERROR   0.00s  TimeoutError: payment page not loaded after 10s
    #2  PASS    0.00s

  Run #1  traceback
  │ tests\checkout.py:17 in pays_with_card
  │     raise TimeoutError("payment page not loaded after 10s")
  │ TimeoutError: payment page not loaded after 10s
...
Flaky 1 ────────────────────────────────────────────────────────────────────────

  CheckoutSuite.pays_with_card   passed on run 2
      TimeoutError: payment page not loaded after 10s

 FAILED   1 flaky test (--fail-on-flaky)  exit code 1

Throttling, ramp-up and truncation new since 0.9a8

Pace a run so it doesn't flood the system under test, and decide how much of each failure is kept.

Save any of them with tj config update so every run uses them. More in Throttling & Pools and Truncation.

JSON report, and report folders since 0.9a6

--json-report writes every result as JSON: each test with its status, whether it was flaky and every run with its runtime and error, plus totals and the seed. All three report options take a file or a folder.

# all three reports into one folder
tj run --html-report out/ --xml-report out/ --json-report out/

{
  "test_junkie": "0.9a8", "runtime": 0.42, "seed": 4242,
  "totals": {"total": 3, "success": 2, "fail": 1, …},
  "suites": [
    {
      "name": "FlakySuite", "module": "flaky", "status": "fail", "runtime": 0.42,
      "tests": [
        {
          "name": "always_fails", "parameter": null, "suite_parameter": null,
          "status": "fail", "flaky": false, "meta": {},
          "runs": [
            {"status": "fail", "runtime": 0.0, "error": "AssertionError: nope", "retry": null, "meta_set": {}}
          ]
        },
        …
      ]
    }
  ]
}

Run again what didn't pass since 0.9a7

--rerun FILE runs only the tests that failed, errored or were ignored in a --json-report, down to the parameter. The header shows how many. See Rerun.

tj run --json-report reports/
tj run --rerun reports/report.json

GitHub Actions annotations since 0.9a6

On GitHub Actions, tj run also prints an ::error line for every failed, errored or ignored test, so GitHub marks the failing line in the pull request. Nothing to set up: it's on when GITHUB_ACTIONS is set, which GitHub does.

::error file=tests/checkout.py,line=42,title=CheckoutSuite.applies_coupon::FAIL: AssertionError: expected 90, got 100

When a saved config is used since 0.9a6

If any setting comes from your saved config (see tj config), the header says so: where the config is and which settings it applied. Anything you pass on the command line wins and isn't listed.

Test Junkie 0.9a6 · Python 3.12.5 · Windows 11
  tests    tests · 1 suite, 6 tests · found in 0.00s
  mode     parallel · 1 suite thread · 4 test threads
  filters  tags any smoke
  config   C:\Users\you\AppData\Local\Test-Junkie\Test-Junkie\.tj.cfg
           run_on_match_any smoke · sources tests · test_multithreading_limit 4

Resource chart (-m) since 0.9a6

With -m / --monitor-resources, Test Junkie samples CPU and memory four times a second while the tests run, and charts them after the Summary. CPU is the filled area and memory the line, both on one 0–100% axis. Every test is a dot at the moment it finished, in its result color, and the row under the dots shows when each suite ran, so you can see what was running when the machine got busy. The key line says which suites were running at the CPU peak.

  44 tests           75%    2%     2%     16%    5%

────────────────────────────────────────────────────────────────────────────────

Resources CPU and memory, whole machine ────────────────────────────────────────

  █ CPU     avg 52% · peak 81% at 3.2s · during LoginSuite
  ━ Memory  avg 58% · peak 64% at 7.0s

   100% ┤                                                            
        ┤  ▂▂                ▄▄▁▁                                    
    75% ┤  █████▂▂     ▄▆▆  ▅████▆▂▂      ━━━━━━━━━━━━━━━━━          
        ┤  ████━━━━━━━━━━━━━━━━━━━━━━━━━━━  ▁▁             ━━━━━━━━━━
    50% ┤━━━━━━█████▆▁▁███████████████████████▂               ▁▅▅████
        ┤  ████████████████████████████████████▆▆▄▄▄██▇▆▆▆▆▅▆▆███████
    25% ┤  ██████████████████████████████████████████████████████████
        ┤▆▅██████████████████████████████████████████████████████████
     0% ┼────────────────────────────────────────────────────────────
   tests   ● ●  ●  ● ●  ●  ●  ● ●  ● ● ●● ● ● ● ●● ● ● ●●●●●●●●●●●●●●
                                                       ●● ●●   ●     
                                                       +             
  suites ├Login────────────────────├Checkout───────────├Sear…├Reports
         0s          2s           4s           6s           8s   9.2s

  ● passed  ● failed  ● error  ● skipped  ● ignored  ● cancelled

When suites or tests run in threads, every suite thread gets its own lane, so overlapping suites stay readable, and a running row shows how many tests were in progress at each moment, out of your test threads:

Resources CPU and memory, whole machine ────────────────────────────────────────

  █ CPU     avg 72% · peak 83% at 0.05s · during Login, Checkout, Search, Reports
  ━ Memory  avg 70% · peak 74% at 2.30s
  ▆ tests running at once, out of 10 test threads

   100% ┤                                                            
        ┤ ▅▅▄▃    ▃▁▄▄▁▄▄  ▄▁▄▁  ▂▃   ▄▃     ▃ ▂▁ ▃ ▂ ▁▁             
    75% ┤██████━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
        ┤━━━━━━████████████████████████████████████████████▇▄█ ▁     
    50% ┤███████████████████████████████████████████████████████▄    
        ┤████████████████████████████████████████████████████████▃   
    25% ┤█████████████████████████████████████████████████████████▇▇▅
        ┤████████████████████████████████████████████████████████████
     0% ┼────────────────────────────────────────────────────────────
  running█████████████████████████████████████████████████▇▆▆▆▅▅▅▄▂▂▂  peak 10 of 10
   tests ●●              ●    ●●●  ● ●●    ●● ● ●● ●●●● ●●●  ●  ●● ●●
         ●                    ●●●           ●      ● ●   ●●     ●●   
         ●                    ●                                      
         +                                                           
  suites ├Login───────────────────────                               
         ├Checkout─────────────────────────────────────────          
         P├Reports───────────────────────────────────────────────────
         ├Search───────────────────────────────────────────          
         0s        0.5s       1s         1.5s       2s          2.74s

  ● passed  ● failed  ● error  ● skipped  ● ignored  ● cancelled

• The numbers are for the whole machine, not just Test Junkie. They're the same samples the HTML report charts.
• The chart is as wide as your terminal. In CI logs and files it prints at 60 columns, without colors. -q leaves it out.
• A run shorter than a quarter of a second is too short to sample, and says so.

tj audit since 0.8a.8

Scans your codebase and aggregates test data based on the query you give it, without running anything. Useful for coverage reads, Git hooks, or finding gaps in metadata. Pick a view, and every block in it is one owner, suite, feature, component or tag:

# All tests broken down by owner
tj audit owners

# Tests missing a tag
tj audit tags --no-tags

# Owners for specific components
tj audit owners --components 2FA OAuth

# Each component split by feature
tj audit components --by-features

# Suites missing an owner
tj audit suites --no-owners

Views: suites, features, components, tags, owners. They all take the same options. tj audit conflicts lists every pair of tests that must not run together. Full reference: tj audit suites -h.

What an audit prints since 0.9a6

One block per entry, biggest first, with its share of all tests and what it's made of. Missing metadata reads none, and the Gaps section at the end counts the tests that lack each kind, with the command that lists them.

Test Junkie 0.9a6 · Python 3.12.5 · Windows 11
  tests    tests · 5 suites, 13 tests · found in 0.00s
  audit    suites

LoginSuite                          4 tests ·  31%  [|||||···········]
  feature      Accounts
  owners       qa-team 4
  components   auth 4
  tags         regression 2 · smoke 2

CheckoutSuite                       3 tests ·  23%  [||||············]
  feature      Billing
  owners       checkout-team 3
  components   payment 3
  tags         regression 2 · smoke 1

PaymentsSuite                       2 tests ·  15%  [||··············]
  feature      Billing
  owners       checkout-team 2
  components   payment 1 · refunds 1
  tags         regression 2

ReportsSuite                        2 tests ·  15%  [||··············]
  feature      Analytics
  owners       none 2
  components   reports 1 · none 1
  tags         none 2

SearchSuite                         2 tests ·  15%  [||··············]
  feature      Catalog
  owners       search-team 2
  components   search 2
  tags         smoke 1 · none 1

Gaps 3 ─────────────────────────────────────────────────────────────────────────

  2 tests  have no owner         tj audit suites --no-owners
  1 test   has  no component     tj audit suites --no-components
  3 tests  have no tags          tj audit suites --no-tags

────────────────────────────────────────────────────────────────────────────────

13 tests · 5 suites · 4 features · 3 owners · 5 components · 2 tags

• With a filter (--no-owners, -o, -c, -f, -l …) each block also lists its tests by name, up to 10, so you can go straight to them.
• --by-features and --by-components split every block by feature or component, with the owners and tags of each part.
• When nothing matches, it says how many tests were scanned and which filters ruled them out.

Test Junkie 0.9a6 · Python 3.12.5 · Windows 11
  tests    tests · 1 suite, 2 tests · found in 0.00s
  audit    owners
  filters  no owner

no owner                            2 tests · 100%  [||||||||||||||||]
  suites       ReportsSuite 2
  features     Analytics 2
  components   reports 1 · none 1
  tags         none 2
  tests        ReportsSuite.lists_reports · ReportsSuite.export_csv

Gaps 2 ─────────────────────────────────────────────────────────────────────────

  1 test   has  no component     tj audit owners --no-components
  2 tests  have no tags          tj audit owners --no-tags

────────────────────────────────────────────────────────────────────────────────

2 tests · 1 suite · 1 feature · 0 owners · 1 component · 0 tags

Test Junkie 0.9a6 · Python 3.12.5 · Windows 11
  tests    tests · 5 suites, 13 tests · found in 0.00s
  audit    components · by features

auth                                4 tests ·  31%  [|||||···········]
  suites       LoginSuite 4
  owners       qa-team 4
  tags         regression 2 · smoke 2
  feature      Accounts 4
               owners qa-team 4   tags regression 2 · smoke 2

payment                             4 tests ·  31%  [|||||···········]
  suites       CheckoutSuite 3 · PaymentsSuite 1
  owners       checkout-team 4
  tags         regression 3 · smoke 1
  feature      Billing 4
…

In CI: --json and --fail-on-gaps since 0.9a6

--json prints the view as JSON for scripts. --fail-on-gaps exits with code 1 when tests are missing metadata: owners, features, components or tags, or the kinds you list. Tag filters are the ones tj run uses: -k for any of the tags, -l for all of them.

# fail the build when a test has no owner or no tags
tj audit suites --fail-on-gaps owners,tags

# owners as JSON
tj audit owners --json

…
13 tests · 5 suites · 4 features · 3 owners · 5 components · 2 tags
 FAILED   2 tests without owner, 3 tests without tags  exit code 1

tj config since 0.8a.8

Saves settings for tj run and tj audit, so you don't type them every time. Three sub-commands: show, update and restore. Every option of tj run can be saved.

# Every setting, grouped, and where the config file is
tj config show --all

# Save sources and test threads once instead of repeating them on every run
tj config update -s /path/to/tests -T 4

# Clear individual settings, or all of them
tj config restore -s -T
tj config restore --all

Saved values can always be overridden at runtime — e.g. tj run --sources /path/to/suite.py takes precedence over whatever is in config.

What it prints since 0.9a6

update shows each setting's old and new value, and the exact command that undoes the change:

 SAVED   3 settings · C:\Users\you\AppData\Local\Test-Junkie\Test-Junkie\.tj.cfg
  test_multithreading_limit     ·  →  4
  run_on_match_any              ·  →  smoke
  sources                       ·  →  tests

  Undo: tj config restore -T -k -s

show --all groups the settings by what they control. Saved values are bold; unset ones are a dot, with the default when there is one. show -s -T prints just those settings.

Config  C:\Users\you\AppData\Local\Test-Junkie\Test-Junkie\.tj.cfg

Discovery
  sources                       tests
  guess_root                    ·  default off

Parallel and retries
  test_multithreading_limit     4
  suite_multithreading_limit    ·  default 1
  retry                         ·
  no_retry                      ·  default off
  retry_policy                  ·

Limits
  suite_throttling              ·  default 0
  test_throttling               ·  default 0
  ramp_up                       ·  default 0
  traceback_limit               ·  default 3000
  message_limit                 ·  default 3000
  truncate                      ·  default middle

Filters
  tests                         ·
  features                      ·
  components                    ·
  owners                        ·
  run_on_match_any              smoke
  run_on_match_all              ·
  skip_on_match_any             ·
  skip_on_match_all             ·

Reports and output
  html_report                   ·
  xml_report                    ·
  json_report                   ·
  monitor_resources             ·  default off
  code_cov                      ·  default off
  cov_rcfile                    ·
  quiet                         ·  default off
  per_test                      ·  default off
  no_capture                    ·  default off
  flag_flaky                    ·  default off
  fail_on_flaky                 ·  default off

────────────────────────────────────────────────────────────────────────────────

3 of 32 settings saved · tj config update -h to change one, tj config restore -h to clear one

 RESTORED   1 setting to its default
  test_multithreading_limit     4  →  default 1

A wrong value is rejected before anything is saved:

 ERROR   -T / --test-multithreading-limit needs a whole number, got "four".
         Nothing was saved. tj config update -h lists every option.

Project config since 0.9a6

Keep a project's settings with the project: save them to a tj.cfg in its root, and commit it. tj run, tj audit and tj config look for one from the current folder up and use it instead of your user config. Without -s and tj run picks up the saved sources, and if a suite's imports fail it retries with the project folder on the import path.

# create the project config
tj config update -s tests -T 4 --config ./tj.cfg

# from anywhere in the project
tj run

# which config is in use
tj version

A [tool.test_junkie] table in pyproject.toml works too (Python 3.11+, or with tomli installed). Test Junkie only reads it: change it in the file.

[tool.test_junkie]
sources = ["tests"]
test-multithreading-limit = 4

Where config is stored since 0.9a3

By default Test Junkie keeps its config (and resource-monitoring temp files) in your per-user app data directory — tj config show --all prints the exact path. Set the TEST_JUNKIE_HOME environment variable to use a different directory instead — useful in CI, containers, or anywhere you want runs isolated from a shared config.

# Use a throwaway config dir for this shell
export TEST_JUNKIE_HOME=/tmp/tj_home
tj config show --all

@Suite since 0.8a.8

from test_junkie.decorators import Suite

@Suite()
class ExampleTestSuite:
    ...

@beforeClass since 0.8a.8

from test_junkie.decorators import beforeClass

...
@beforeClass()
def before_class(self):
    ...

This decorator doesn't accept any properties of its own. In a parameterized suite, add a suite_parameter argument to receive the value for the current pass. Leaving it out doesn't stop the function from running on every pass; it just isn't told which one. See which functions run per suite parameter.

@beforeTest since 0.8a.8

from test_junkie.decorators import beforeTest

...
@beforeTest()
def before_test(self):
    ...

This decorator doesn't accept any properties of its own. The function it decorates can take two optional arguments, in any order: suite_parameter, the current pass's value in a parameterized suite (None around a test that runs once), and test, described below.

The test argument new since 0.9a8

@Suite()
class CheckoutSuite:

    @beforeTest()
    def start_trace(self, test):
        # one trace per test, named after it
        self.trace = tracing.start(
            name=test.get_function_name())

    @afterTest()
    def save_trace(self, test):
        name = test.get_function_name()
        self.trace.save("traces/" + name + ".zip")

    @test()
    def pays_with_card(self):
        ...

It's read-only, by design

Test Junkie keeps using the test after your hook returns: its tags, parameters, retry state and meta decide what runs next, what's retried and what the reports say. So test is a read-only view, not the live object. Nothing a hook does through it can change the run, even when many tests run in parallel.

isinstance(test, TestObject) is True. To record something about a run (an account id, a link), call Meta.update() from the test itself; the hook's view is for reading. Rules before_test()/after_test() get the same view in kwargs["test"].

An individual test can opt out of running its suite's @beforeTest() — see skip special functions on the @test() page.

@test since 0.8a.8

from test_junkie.decorators import test

...
@test()
def example_test(self):
    ...

This decorator accepts many properties that let you control how each test executes.

@afterTest since 0.8a.8

from test_junkie.decorators import afterTest

...
@afterTest()
def after_test(self):
    ...

The test argument new

This decorator doesn't accept any properties of its own. The function it decorates can take two optional arguments, the same as @beforeTest(): suite_parameter (None after a test that runs once) and test, a read-only view of the test that just ran. See the test argument.

@afterTest()
def save_trace(self, test):
    # saved under the name of the test that just ran
    self.trace.save(
        "traces/" + test.get_function_name() + ".zip")

An individual test can opt out of running its suite's @afterTest() — see skip special functions on the @test() page.

@afterClass since 0.8a.8

from test_junkie.decorators import afterClass

...
@afterClass()
def after_class(self):
    ...

This decorator doesn't accept any properties of its own. Add a suite_parameter argument to receive the value for the pass that just finished. It runs on every pass either way. See which functions run per suite parameter.

@GroupRules since 0.8a.8

This decorator defines rules that span a set of test suites, and it must decorate a method on a subclass of Rules.

Inside a method decorated with @GroupRules(), import the suites you care about and wrap nested functions with @beforeGroup() and @afterGroup() to build the actual rules. Define as many as you like — see the Rules section for a fuller example.

@beforeGroup

from test_junkie.decorators import GroupRules, beforeGroup
from test_junkie.rules import Rules

class ExampleRules(Rules):

    @GroupRules()
    def group_rules(self):

        from ... import ExampleSuite1, ExampleSuite2

        @beforeGroup([ExampleSuite1, ExampleSuite2])
        def before_group_a():
            # your code
            pass

@afterGroup

@afterGroup([ExampleSuite1, ExampleSuite2])
def after_group_a():
    # your code
    pass

Skip since 0.8a.8

from test_junkie.decorators import Suite, test

def custom_skip_function(meta):
    # meta is optional
    # do your processing
    return True  # or False, must return a boolean

@Suite(skip=True)  # or skip=custom_skip_function
class ExampleSuite:

    @test(skip=True)  # or skip=custom_skip_function
    def example_test(self):
        pass

runtime skip since 0.9a2

from test_junkie.decorators import Suite, test
from test_junkie.shortcuts import skip

@Suite()
class CheckoutSuite:

    @test()
    def test_checkout_flow(self):
        if not account_has_payment_method():
            skip("payment method not configured")
        # ... rest of the test

skip special functions

...
@test(skip_before_test=True,
      skip_after_test=True,
      skip_before_test_rule=True,
      skip_after_test_rule=True)
def example_test(self):
    pass

Retry since 0.8a.8

from test_junkie.decorators import Suite, test

@Suite(retry=2)  # default is 1
class ExampleSuite:

    @test(retry=2)  # default is 1
    def example_test(self):
        # this test can run up to 4 times total:
        # suite_retry x test_retry = max total test retries
        assert True is False

...
@test(retry=2, retry_on=[AssertionError])
def example_test(self):
    # will be retried, since it failed with AssertionError
    assert True is False
...

...
@test(retry=2, no_retry_on=[AssertionError])
def example_test(self):
    # won't be retried, this exception is explicitly excluded
    assert True is False
...

Both retry_on and no_retry_on take a list of exception classes — useful when you want to retry on a flaky, known-transient error without masking every other kind of failure. If you're using retries on a parameterized test, only the parameters that didn't pass get retried. To run only what didn't pass again in a later run, see Rerun.

Subclasses count: retry_on=[requests.exceptions.Timeout] also retries a ReadTimeout. since 0.9a6 A test can use both lists; no_retry_on is checked first.

Flaky tests new since 0.9a8

runner.run()
for suite in runner.get_executed_suites():
    for test in suite.get_test_objects():
        for flaky in test.get_flaky():
            print(test.get_function_name(),
                  flaky["suite_parameter"],
                  flaky["passed_on_run"],
                  flaky["errors"])

# or ask about one combination
test.is_flaky(suite_parameter="admin")

$ tj run -s tests --flag-flaky
...
Flaky 1 ────────────────────────────────────────────────────────────────────────

  CheckoutSuite.pays_with_card [suite: admin]  passed on run 2
      TimeoutError: payment page not loaded after 10s

 PASSED   exit code 0

$ tj run -s tests --fail-on-flaky
...  a Problems entry per flaky test and the same Flaky block, then:
 FAILED   1 flaky test (--fail-on-flaky)  exit code 1

Retry Policies new since 0.9a8

from test_junkie.retry import RetryPolicy, When

class Flaky(RetryPolicy):
    no_retry_on = [PermissionError]  # never retried
    when = [
        When(TimeoutError, attempts=3, delay=1),
        When(message="503", attempts=2,
             delay=20, backoff=2),
    ]

@Suite(retry_policy=Flaky)  # tests without retry=
class CheckoutSuite:

    @test()
    def pays_with_card(self):
        ...

    @test(retry=Flaky(attempts=2))  # at most 2 runs
    def refunds(self):
        ...

    @test(retry=1)  # opt out
    def shows_receipt(self):
        ...

Here is Flaky on a real run. The timeout and the 503 each pass on their second run, and the PermissionError is never retried, so it fails after one run:

$ tj run -s tests -p
...
CheckoutSuite  tests\checkout.py
  PASSED     pays_with_card   passed on run 2 (Flaky, when TimeoutError: waited 1s)   0.00s  [ 33%]
  PASSED     refunds          passed on run 2 (Flaky, when "503": waited 20s)         0.00s  [ 66%]
  ERROR      shows_receipt                                                             0.00s  [100%]
...
[ERROR] CheckoutSuite.shows_receipt
        tests\checkout.py:36 · 1 run · 0.00s
...
  │ PermissionError: receipt needs the billing role

Mistakes surface when the class is defined, not halfway through a run: a negative delay or a When without attempts, in a policy without attempts, raises BadParameters naming the field. A policy and retry_on/no_retry_on on the same @test() is an error too; put them on the policy.

Building a When

import re
from test_junkie.retry import RetryPolicy, When


class Checkout(RetryPolicy):
    delay = 1  # every When that sets no delay
    when = [
        # either class, up to 3 runs
        When(TimeoutError, ConnectionError,
             attempts=3),

        # ApiError AND one of the texts
        When(ApiError,
             message=["503",
                      re.compile("rate limit", re.I)],
             attempts=2, delay=10, backoff=2,
             name="server busy"),

        # anything else: one more try
        When(attempts=2),
    ]

An argument a When leaves out comes from the policy, so set shared values like delay once on the class. chain and no_retry_on are policy-wide and apply to every When; no_retry_on is checked first and always wins.

Hooks and a fresh setup

class Session(RetryPolicy):
    when = [When(message="401", attempts=2),
            When(message="stale session",
                 attempts=2, reset="class")]
    max_time = 120

    def should_retry(self, error, decision, test):
        # never retry tests that move money
        tags = test.get_tags()
        return (decision.retry
                and "payments" not in tags)

    def before_retry(self, error, decision, test):
        tokens.refresh()  # runs right before the retry

For a whole run

A policy can cover a whole run: Runner.run(retry_policy=Flaky) or tj run --retry-policy my_project.retries:Flaky. It applies to every test that sets no retry of its own and whose suite sets no retry_policy, for that run only.

Rerun since 0.9a7

# run everything and keep the results
tj run -s tests --json-report reports/

# run again only what didn't pass
tj run -s tests --rerun reports/report.json

Only what runs again is reported, so a rerun's report and summary show the tests you're looking at, not the ones that already passed. Keep the first report if you need the full picture, or write the rerun's report somewhere else. Retries still apply inside a rerun, and only to the parameters it runs. If nothing failed, nothing runs and the run passes.

From code

Runner.run() takes the path to a report, or a Rerun. Build one from a report, choosing which results run again, or add the tests yourself:

from test_junkie.constants import TestCategory
from test_junkie.rerun import Rerun
from test_junkie.runner import Runner

runner = Runner([LoginSuite, CheckoutSuite])
runner.run(rerun="reports/report.json")

# only the failures, not the errors and ignored tests
runner.run(rerun=Rerun.from_report("reports/report.json", statuses=[TestCategory.FAIL]))

# or say what runs again: suite, test, parameter, suite parameter
runner.run(rerun=Rerun().add("LoginSuite", "login", parameter="admin", suite_parameter="staging"))

Your own matching

If your results live somewhere else, like a database, subclass Rerun: add() the tests that run again and override includes() to decide which of their parameters do. It gets the real parameter values, not their text.

class FailedInDatabase(Rerun):

    def __init__(self, failed):
        super().__init__()
        self.failed = failed  # {(suite, test): {(suite parameter, parameter), ...}}
        for suite, test in failed:
            self.add(suite, test)

    def includes(self, suite, test, parameter=None, suite_parameter=None):
        return (suite_parameter, parameter) in self.failed.get((suite, test), ())

runner.run(rerun=FailedInDatabase(load_failures()))

Parametrization since 0.8a.8

Test Junkie has one of the most flexible parametrization engines of any Python test runner — parametrization is supported at both the suite and test level.

@Suite(parameters=[1, 2])
class ExampleSuite:

    @beforeClass()
    def before_class(self, suite_parameter):
        print(suite_parameter)

    @beforeTest()
    def before_test(self, suite_parameter):
        print(suite_parameter)

@Suite(parameters=[1, 2])
class ExampleSuite:

    @test(parameters=[10, 20])
    def example_test(self, parameter, suite_parameter):
        print(parameter)
        print(suite_parameter)

Which functions run per suite parameter

So you opt a test in or out of suite parameters with its signature; there's no flag for it. Which way to go depends on whether the test's outcome can change with the suite parameter:

@Suite(parameters=["admin", "viewer"])
class AccountSuite:

    @beforeClass()
    def log_in(self, suite_parameter):
        self.session = login(suite_parameter)

    # opted in: runs for admin and for viewer
    @test()
    def sees_dashboard(self, suite_parameter):
        assert self.session.can_open("/dashboard")

    # opted out: runs once, the role doesn't matter
    @test()
    def api_is_up(self):
        assert ping("/health") == 200

Providers

def active_accounts():
    # called when the suite runs
    return db.accounts(active=True)

@Suite(parameters=active_accounts)
class AccountSuite:
    ...

class Account:
    def __str__(self):
        return "Account"  # same for every account

@test(parameters=[alice, bob], ids=["alice", "bob"])
def logs_in(self, parameter):
    ...

# or work the names out
@test(parameters=load_accounts,
       ids=lambda account: account.login)

When a parameters function raises new since 0.9a8

If it raises, that suite or test is ignored with the traceback, and the rest of the run carries on; the run still fails at the end. A function that raised isn't called again in the same run, so suites sharing a provider that times out don't each wait for it.

Parallel Execution since 0.8a.8

A test suite that takes 40 minutes to run sequentially stops being something people actually run before pushing. Test Junkie supports multi-threading out of the box, with nothing extra to install, and you allocate N threads as a pool for your suites and/or tests. Thread allocation happens through the run() method of the Runner instance, independently for suites (suite_multithreading_limit) and tests (test_multithreading_limit) — the same limits the CLI's -S and -T flags set. See priority for how execution order interacts with parallel scheduling.

Parallelized parameters

@test(parallelized_parameters=True, parameters=["a", "b", "c"])
def example_test(self, parameter):
    ...

Opting out of parallel mode

@Suite(parallelized=False)
class ExampleSuite:

    @test(parallelized=False)
    def example_test(self):
        ...

Conflicts new since 0.9a8

# every test of both suites
@Suite(conflicts_with=[CheckoutSuite])
class InventorySuite:

    @test(conflicts_with=[
        "PricingSuite.updates_prices"])
    def recounts_stock(self):
        ...

    # a test can name a whole suite
    @test(conflicts_with=[ReportsSuite])
    def rebuilds_index(self):
        ...

Many tests, one thing they all touch

from test_junkie.objects import Limiter

Limiter.pool("inventory_db", max_concurrent=1)

@test(uses="inventory_db")
def recounts_stock(self): ...

@test(uses="inventory_db")
def imports_catalog(self): ...

Seeing it work

{
  "name": "updates_prices",
  "status": "success",
  "conflict_waits": [
    {
      "seconds": 2.001,
      "with": ["InventorySuite.recounts_stock"]
    }
  ],
  ...
}

@Suite()
class InventorySuite:

    @test()
    def lists_warehouses(self):
        time.sleep(0.3)

    @test(conflicts_with=[
        "PricingSuite.updates_prices"])
    def recounts_stock(self):
        time.sleep(2)


@Suite()
class PricingSuite:

    @test()
    def shows_discounts(self):
        time.sleep(0.3)

    @test()
    def updates_prices(self):
        time.sleep(0.5)

$ tj run -S 2 -T 2 -p

InventorySuite  shop_suites.py
  PASSED     lists_warehouses                                         0.30s  [ 25%]
  PASSED     recounts_stock                                           2.00s  [ 50%]

PricingSuite  shop_suites.py
  PASSED     shows_discounts                                          0.30s  [ 75%]
  PASSED     updates_prices    waited 2s (conflicts with InventorySuite.recounts_stock)   0.50s  [100%]

tj audit conflicts lists every pair of tests kept apart, without running anything. --json prints it for scripts.

$ tj audit conflicts

  InventorySuite.recounts_stock  x  PricingSuite.updates_prices

$ tj audit conflicts --json
{
 "conflicts": [
  [
   "InventorySuite.recounts_stock",
   "PricingSuite.updates_prices"
  ]
 ]
}

Throttling & Resource Pools new since 0.9a8

Parallel execution is fast until it's too fast for something your tests share: a Selenium Grid with five browser slots, a sandbox API that allows two calls a second, a staging database that falls over at twenty connections. When that happens, tests fail with session timeouts and HTTP 429s that have nothing to do with your code. Test Junkie has five settings for this, from broad to targeted:

Throttling

from test_junkie.objects import Limiter

# the grid queues 12 sessions that arrive at once
# one new test per second, run-wide
Limiter.TEST_THROTTLING = 1
# one new suite (and its login) every 5 s
Limiter.SUITE_THROTTLING = 5

runner.run(suite_multithreading_limit=4,
           test_multithreading_limit=12)

Limiter.TEST_THROTTLING = 1

# pure API checks: no spacing
@Suite(throttling=0)
class PricingApiSuite: ...

# legacy billing service: 3 s apart
@Suite(throttling=3)
class LegacyBillingSuite: ...

Resource pools

from test_junkie.objects import Limiter

# 5 browser slots
Limiter.pool("grid", max_concurrent=5)
# sandbox API: 2 calls at once, 2 a second
Limiter.pool("payments", max_concurrent=2,
             min_interval=0.5)

# every test in the suite takes a grid slot
@Suite(uses="grid")
class CheckoutUiSuite: ...

@Suite()
class OrdersSuite:

    # a slot in both pools
    @test(uses=["grid", "payments"])
    def pay_by_card(self): ...

    # no pool: runs whenever a thread is free
    @test()
    def order_history_api(self): ...

runner.run(test_multithreading_limit=20)

• A test takes its slots right before @beforeTest() and gives them back after @afterTest(), even if it fails. Each retry attempt takes them again. @beforeClass() and @afterClass() don't take slots.
• A test waiting for a slot keeps its thread, so set -T higher than the pool. Then the other tests keep running while some wait.
• Tests that use several pools take them in name order, so two tests can never each hold the pool the other is waiting for.
• A uses= that names a pool you never defined fails the run before anything starts, with the name and the fix.
• Pools work in single-threaded runs too: min_interval still spaces the tests out.
• Cancelling the run (Ctrl+C or runner.cancel()) cancels the tests still waiting for a slot.

Limiter.get_pools() shows each pool's settings and how many tests hold it right now. Limiter.remove_pools() removes them. Pools and conflicts solve different problems: conflicts_with= keeps two named tests from running together, and a pool caps how many tests run together.

Ramp-up

# 1 thread, ~6 after 15 s, all 12 after 30 s
Limiter.RAMP_UP = 30
runner.run(test_multithreading_limit=12)

For one run, from the CLI, or saved

# CI against the shared grid
tj run -S 4 -T 12 --test-throttling 1 --ramp-up 30

# or save it for every tj run
tj config update --test-throttling 1 --ramp-up 30

runner.run(test_multithreading_limit=12,
           test_throttling=1, ramp_up=30)

Rules since 0.8a.8

Suites often share setup/teardown logic that doesn't belong hardcoded into every @beforeClass() — clearing a shared cache, resetting a feature flag, logging timing for every test regardless of which suite it's in. Rules gives you a reusable class for exactly that: write the logic once, then attach it to as many suites as you like via @Suite(), instead of copy-pasting the same lifecycle code into every suite that needs it. For logic that spans a group of suites rather than living inside one, see @GroupRules below.

To create rules, subclass Rules and override whichever lifecycle methods you need:

A suite's own @beforeClass()/@beforeTest()/etc. and its Rules class both run — Rules doesn't replace them, it adds a second, reusable layer alongside.

from test_junkie.decorators import GroupRules, afterGroup, beforeGroup
from test_junkie.rules import Rules


class TestRules(Rules):

    def __init__(self, **kwargs):
        Rules.__init__(self, **kwargs)

    def before_class(self):
        # your code
        pass

    def before_test(self, **kwargs):
        # your code
        pass

    def after_test(self, **kwargs):
        # your code
        pass

    def after_class(self):
        # your code
        pass

    @GroupRules()
    def group_rules(self):

        from my.suites.examples import ExampleSuite1, ExampleSuite2

        @beforeGroup([ExampleSuite1, ExampleSuite2])
        def before_group_a():
            # your code
            pass

        @afterGroup([ExampleSuite1, ExampleSuite2])
        def after_group_a():
            # your code
            pass


@Suite(rules=TestRules)
class ExampleSuite:
    ...

The test they get new since 0.9a8

class LoggingRules(Rules):

    def before_test(self, **kwargs):
        test = kwargs["test"]
        log.info("start %s.%s (owner: %s)",
                 test.suite.get_class_name(),
                 test.get_function_name(),
                 test.get_owner())

    def after_test(self, **kwargs):
        test = kwargs["test"]
        log.info("done %s", test.get_function_name())


# one rule, every suite that attaches it
@Suite(rules=LoggingRules, owner="Payments team")
class CheckoutSuite:
    ...

Priority since 0.8a.8

When one test has to run before another — a login has to succeed before you can test account settings, say — priority is how you say so. It influences execution order for suites and tests. Influences, not guarantees: priority can't be honored exactly in threaded mode once some suites have parallelized set to False, or parallel restrictions configured. Test Junkie always looks for a suite or test it can run right now; if the next one in priority order is currently restricted and can't execute, it moves on to something that can, rather than stalling the whole run waiting for one slot to clear. That's a deliberate throughput trade-off, not a bug — if you need strict ordering, pair priority with parallel restrictions or turn off parallel execution for the suites/tests involved.

Priority is integer-based, starting at 1 (the highest priority) and increasing from there. Set it via the priority property of @Suite() or @test():

from test_junkie.decorators import Suite, test

@Suite()
class AccountSuite:

    @test(priority=1)
    def test_login(self):
        # runs before test_update_profile below
        ...

    @test(priority=2)
    def test_update_profile(self):
        ...

Order since 0.9a2

The order argument on @Suite() controls how tests within that suite are sequenced. It accepts a TestOrder constant and overrides the default priority-ascending sort for that suite entirely — including any numeric priority values set on individual tests.

from test_junkie.decorators import Suite, test
from test_junkie.constants import TestOrder

@Suite(order=TestOrder.ALPHABETICAL)
class LoginSuite:

    @test()
    def test_login_valid_credentials(self): ...

    @test()
    def test_login_expired_password(self): ...

    @test()
    def test_login_invalid_credentials(self): ...

Listeners since 0.8a.8

Test Junkie fires 22 named events as a run progresses - a test starting, passing, retrying, a suite's @beforeClass() blowing up, and so on. Defining a listener is optional, but it's the seam you'd use for anything that isn't "run the tests": pushing metrics somewhere, posting to Slack on failure, collecting artifacts, or driving your own dashboard off live run state.

from test_junkie.listener import Listener

class MyTestListener(Listener):

    def __init__(self, **kwargs):
        Listener.__init__(self, **kwargs)

    def on_failure(self, **kwargs):
        properties = kwargs["properties"]
        test = properties["jm"]["jto"]
        exc = kwargs.get("exception")
        notify_slack(f"{test.get_function_name()} failed: {exc}")

If a listener raises new since 0.9a8

A bug in a listener doesn't change any test results, and it doesn't stop the run. The exception is recorded, and the suite carries on with its remaining tests. Each failed call is listed under Problems as a LISTENER entry, with the event, the test's parameters and the traceback.

The run still fails: once every suite is done, Runner.run() raises TestListenerError and the console verdict reads, for example, FAILED 2 listener calls failed.

Test events

Suite & group events

Meta since 0.8a.8

Meta has no effect on how Test Junkie runs tests and can hold any data type. It's supported by both the @Suite() and @test() decorators.

Why is this useful? You can use meta to attach:

• Test name, suite name, description, expected results, or anything else useful in reports
• Test case IDs - if you have a test management system, link test scripts directly to test cases and build further integrations from there
• Bug ticket IDs - if you have a bug tracker (like Jira), link known failures to real tickets and process them differently (e.g. as a known failure rather than a fresh regression)

Test Junkie can do a lot for reporting out of the box, but no off-the-shelf tool beats a custom report built in-house by someone who knows the product. Meta exists to feed useful data into whatever reporting you build.

from test_junkie.decorators import Suite, test
from test_junkie.meta import meta


@Suite(listener=MyTestListener, meta=meta(name="some value", id=123))
class ExampleSuite:

    @test(meta=meta(name="some value", expected="some value",
                    known_bugs=[111, 222, 333], id=321))
    def example_test(self):
        pass

Setting meta while a test runs new since 0.9a8

from test_junkie.decorators import (
    Suite, test, beforeClass, afterTest)
from test_junkie.meta import Meta, meta


def place_order(account):
    order = api.post("/orders", account=account)
    # a helper: no self, no parameters
    Meta.update(order_id=order.id)
    return order


@Suite(parameters=["admin"], listener=Recorder,
       meta=meta(team="payments"))
class CheckoutApiSuite:

    @beforeClass()
    def login(self, suite_parameter):
        session = api.login(suite_parameter)
        Meta.suite.update(session=session)

    @afterTest()
    def evidence(self):
        Meta.update(log=api.save_log())

    @test(retry=2, meta=meta(
        name="Checkout", expected="order is created"))
    def checkout(self, suite_parameter):
        order = place_order(suite_parameter)
        Meta.update(status=order.status)
        assert Meta.get("status") == 201

Threads you start new since 0.9a8

import threading
from test_junkie.meta import Meta


@test()
def bulk_import(self):
    def worker(batch):
        Meta.update(**{batch: "imported"})

    thread = threading.Thread(target=Meta.bind(worker),
                              args=("batch_a",))
    thread.start()
    thread.join()
    assert Meta.get("batch_a") == "imported"

Suite meta from class hooks new since 0.9a8

Meta.suite.update(**values) sets suite-level values, once per suite parameter: a session id from @beforeClass(), a cleanup count from @afterClass(). Listeners get them merged into properties["suite_meta"], after the declared @Suite(meta=...), which stays unchanged too. Meta.suite.get(key) reads them back. It works in per-test hooks and tests as well, and always writes to the current suite parameter.

Lists, links and attachments new since 0.9a8

from test_junkie.decorators import Suite, test
from test_junkie.meta import Meta

JIRA = "https://jira.example.com/browse/"


@Suite()
class CheckoutSuite:

    @test(retry=2)
    def pays_with_card(self):
        Meta.append("steps", "opened the cart")
        Meta.link("Jira", JIRA + "SHOP-482")
        Meta.attach("cart.png",
                    driver.get_screenshot_as_png())
        Meta.attach("app.log", "logs/app.log")
        checkout.pay_with_card()
        Meta.append("steps", "paid with card")

# run 1 failed after both steps, run 2 passed
test.get_meta()["steps"]
["opened the cart", "paid with card",
 "opened the cart", "paid with card"]
test.get_meta()["links"]
[{"label": "Jira",
  "url": "https://jira.example.com/browse/SHOP-482"}]
[a["name"] for a in
 test.get_meta()["attachments"]]
["cart.png", "app.log", "cart.png", "app.log"]
[a["attempt"] for a in
 test.get_meta_attempts(None, None)]
[1, 2]  # each: 2 steps, 1 link, 2 files

Retries: the merged view and each attempt new since 0.9a8

When a test is retried, its values carry forward: a key set in the first run is still there after the second, unless the second sets it again. test.get_meta(parameter, suite_parameter) gives that merged view. test.get_meta_attempts(parameter, suite_parameter) shows what each run set, oldest first, so you can tell which order belongs to the failed run. Each run keeps the latest value per key, not every write, so a test that updates one key a thousand times stores one value. Runs that set nothing aren't listed.

From the example above, in a real run: the first run timed out after creating order A-1001, and the retry created A-1002.

test.get_meta(None, "admin")
{"name": "Checkout", "expected": "order is created", "order_id": "A-1002",
 "log": "logs/checkout.txt", "status": 201}

test.get_meta_attempts(None, "admin")
[{"attempt": 1, "meta_set": {"order_id": "A-1001", "log": "logs/checkout.txt"}},
 {"attempt": 2, "meta_set": {"order_id": "A-1002", "status": 201, "log": "logs/checkout.txt"}}]

Reading meta in listeners new since 0.9a8

from test_junkie.listener import Listener


class Recorder(Listener):

    def on_complete(self, **kwargs):
        properties = kwargs["properties"]
        # merged values + "parameter"
        test_meta = properties["test_meta"]
        # what each run set
        attempts = properties["meta_attempts"]
        results_db.save(test_meta["name"],
                        test_meta.get("order_id"),
                        attempts)

properties["test_meta"]
{"name": "Checkout", "expected": "order is created", "order_id": "A-1002",
 "log": "logs/checkout.txt", "status": 201, "parameter": None}

properties["suite_meta"]
{"team": "payments", "session": "sess-42", "parameter": "admin"}

properties["meta_attempts"]
[{"attempt": 1, "meta_set": {"order_id": "A-1001", "log": "logs/checkout.txt"}},
 {"attempt": 2, "meta_set": {"order_id": "A-1002", "status": 201, "log": "logs/checkout.txt"}}]

Which values an event sees depends on when it fires:

A listener that saves evidence from @afterTest (logs, screenshots) should do it in on_complete, which runs once at the end with every value.

Meta in reports new since 0.9a8

• JSON report: meta on every test (the merged values) and meta_set on every run.
• HTML report: a Metadata panel for each test variant, with keys set while running marked with the attempt that set them, and each attempt's own values under it.
• XML report: JUnit <properties> on each test case, whose name includes its parameters.

Reports need text, so values are converted: bytes are decoded (bad bytes replaced), other objects become their str(), and values longer than 2,000 characters (500 in the HTML report) are cut with a note. Listeners still get the original objects. --rerun ignores the meta fields.

{
 "name": "checkout",
 "suite_parameter": "admin",
 "meta": {"name": "Checkout", "expected": "order is created", "order_id": "A-1002",
          "log": "logs/checkout.txt", "status": 201},
 "runs": [
  {"status": "error", "meta_set": {"order_id": "A-1001", "log": "logs/checkout.txt"}, ...},
  {"status": "success", "meta_set": {"order_id": "A-1002", "status": 201, "log": "logs/checkout.txt"}, ...}
 ]
}

<testcase name="checkout[suite: admin]" classname="CheckoutApiSuite" status="success" time="0.000">
  <properties>
    <property name="name" value="Checkout"/>
    <property name="expected" value="order is created"/>
    <property name="order_id" value="A-1002"/>
    <property name="log" value="logs/checkout.txt"/>
    <property name="status" value="201"/>
  </properties>
  <flakyError type="TimeoutError" message="checkout API took 30s">...</flakyError>
</testcase>

The old call: Meta.update(self, ...) since 0.9a8

@Suite(parameters=["admin"])
class LegacySuite:

    @test()
    def legacy(self, suite_parameter):
        # suite_parameter not passed
        Meta.update(self, name="Legacy")

TestJunkieMetaWarning: Meta.update(self, ...) wrote to parameter None, suite parameter None of <LegacySuite.legacy>, but the test runs as parameter None, suite parameter admin. Pass the running parameter= and suite_parameter=, or call Meta.update(**values) without self, which always writes to the running test

Assignees since 0.8a.8

Have a large QA team sharing the same framework? Test Junkie lets you assign suites and tests to specific team members via the owner property. In the HTML report, you'll see performance metrics broken down by assignee, and assignees are also readable from Listeners for custom reporting. See also features, components, and tags - the other categorization axes.

Both @Suite() and @test() support owner. Setting it on @Suite() assigns every test in that suite to the same owner by default; setting it again on a specific @test() overrides the owner for just that test.

You can run only a given owner's tests with runner.run(owners=[...]) - see running tests assigned to a specific person.

Features since 0.8a.8

Categorizing suites by the feature they cover doesn't just give you KPI metrics broken down by feature in the HTML report - it also lets you run feature-specific regression.

Features are defined via the feature property of the @Suite() decorator - one feature per suite, but a feature can be made up of multiple components. See also components, tags, and assignees - the other categorization axes.

Components since 0.8a.8

Categorizing tests by the component they cover doesn't just give you KPI metrics broken down by component in the HTML report - it also lets you run component-specific regression.

Components are defined via the component property of the @test() decorator - a feature can be made up of several components. See also features, tags, and assignees - the other categorization axes.

Tags since 0.8a.8

Similar to features and components, but tags are many-to-many: a single test can cover multiple areas of the platform - more than one component, even more than one feature. Ideally tests stay small and cover one thing, but in practice (especially with end-to-end tests) that's not always realistic. Tags let you tell Test Junkie, and your teammates, what a test actually covers at a glance.

Tags are supported only by the @test() decorator. Like features and components, they show up as KPI metrics in the HTML report, and let you run tag-specific regression. See also features, components, and assignees - the other categorization axes.

The tag_config argument to the Runner's run() method controls which tags to run tests for, and which to skip.

tag_config options

All four can be set at once. Skip conditions are checked first (skip_on_match_all or skip_on_match_any matching skips the test immediately), then run conditions (run_on_match_all or run_on_match_any matching runs it). If neither run condition is configured at all, a test that wasn't skipped runs by default - you don't need to set a run condition just to get normal execution. A skip triggers the on_skip event.

from test_junkie.runner import Runner

runner = Runner([ExampleSuite1, ExampleSuite2, ExampleSuite3])
runner.run(tag_config={"run_on_match_all": ["...", "..."],
                       "run_on_match_any": ["...", "..."],
                       "skip_on_match_all": ["...", "..."],
                       "skip_on_match_any": ["...", "..."]})

HTML report since 0.8a.8

Test Junkie tracks a number of metrics during test execution and can generate an HTML report from them.

Live demo

Click through a real report

Real report from a 0.9a8 run: retries, flaky tests, per-attempt metadata, links and attachments.

Open the live demo →

To generate a report like that:

from test_junkie.runner import Runner
runner = Runner(suites=[...], html_report="/path/to/file/report.html", monitor_resources=True)
runner.run()

monitor_resources=True turns on CPU & memory monitoring during test execution, rendered in the HTML report as a trend graph. See also XML report and JSON report for other output formats, and meta for attaching custom data that shows up in reports.

XML report since 0.8a.8

Test Junkie can also produce a basic, Jenkins-friendly XML report.

from test_junkie.runner import Runner
runner = Runner(suites=[...], xml_report="/path/to/file/report.xml")
runner.run()

See also HTML report for a richer, visual report, and JSON report for raw data you can process yourself.

Retried and flaky tests new since 0.9a8

<testcase name="pays_with_card" status="success">
  <flakyError type="TimeoutError"
      message="payment gateway took 30s">
    <stackTrace>Traceback (most recent call last):
  ...
TimeoutError: payment gateway took 30s
</stackTrace>
  </flakyError>
</testcase>
<testcase name="applies_coupon" status="failure">
  <failure type="TimeoutError"
      message="coupon service took 30s">...
  <rerunError type="TimeoutError"
      message="coupon service took 30s">...
  <rerunError type="TimeoutError"
      message="coupon service took 30s">...
</testcase>

JSON report since 0.8a.8

JSON reports are used under the hood for all of the other reports Test Junkie produces (including the HTML report). You can use the same data to slice results in whatever way is meaningful to you.

The report data becomes available on the Aggregator object runner.run() returns, once tests have finished executing:

from test_junkie.runner import Runner
runner = Runner(suites=[...])
aggregator = runner.run()

Raw metrics

You can access raw per-suite and per-test metrics directly from SuiteObject and TestObject:

suite_objects = runner.get_executed_suites()
for suite in suite_objects:
    test_objects = suite.get_test_objects()
    print(suite.metrics.get_metrics())
    for test in test_objects:
        print(test.metrics.get_metrics())

Aggregated reports

You can also access the same aggregated reports Test Junkie's own HTML report uses, via the Aggregator object:

print(aggregator.get_basic_report())        # overall + per-suite pass/fail totals
print(aggregator.get_report_by_features())  # grouped by feature -> component
print(aggregator.get_report_by_tags())      # grouped by tag
print(aggregator.get_report_by_owner())     # grouped by assignee
print(aggregator.get_report_by_suite())     # grouped by suite name

Truncation new since 0.9a8

When a test fails, Test Junkie keeps two pieces of text about it: the exception message and the traceback. Each has its own limit, so one failure that prints a 2 MB response can't make the console, the HTML report or the JSON report unreadable.

What to keep: top, bottom or middle

from test_junkie.objects import Limiter

# short headlines: keep the start of each message
Limiter.EXCEPTION_MESSAGE_LIMIT = 300
Limiter.EXCEPTION_MESSAGE_TRUNCATE = "bottom"

# more of the stack, both ends (the default)
Limiter.TRACEBACK_LIMIT = 10000
Limiter.TRACEBACK_TRUNCATE = Limiter.TRUNCATE_MIDDLE

Traceback (most recent call last):
  File "tests/checkout_suite.py", line 41, in pays_with_saved_card
    page.pay(card="4242")
  File "pages/checkout_page.py", line 88, in pay
[. . . 4,210 characters cut . . .]
  File "selenium/webdriver/remote/errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: pay button never became clickable

Good to know

• Only what is recorded is cut: the console, the reports and TestObject metrics. Listeners still get the full exception and traceback, so a listener that saves failures to a database keeps everything.
• A truncated exception keeps its type. isinstance(), retry_on and the error/failure split work exactly as before.
• --truncate MODE and run(truncate=) set the mode for both texts. In code they're separate. See limits for one run.
• Limiter.ACTIVE = False keeps everything uncut.

Runner() since 0.8a.8

Runner is the object that actually executes your tests - everything else in Test Junkie (decorators, listeners, rules) just describes what to run and how; Runner is what runs it. Construct one with your suites, then call run().

from test_junkie.runner import Runner

runner = Runner([LoginSuite, CheckoutSuite], html_report="reports/run.html")
runner.run(suite_multithreading_limit=2, test_multithreading_limit=10)

run()

Every argument is optional - calling runner.run() with nothing runs everything under the suites you constructed it with, unparallelized.

Other methods

Execution flow

The exact sequence Test Junkie calls each method — from the moment Python processes your decorator imports through the final line of runner.run(). Orange = hooks you write; grey = TJ-managed internals.

Running tests since 0.8a.8

Everything below is a Runner() call - the same object, just with a different slice of arguments for each situation. Combine them freely: features, components, owners, tag_config, and tests all filter the same pool of suites you constructed the Runner with.

Parallel execution

The most common starting point: run every suite you passed in, up to 5 suites and 5 tests in parallel at once. See parallel execution for how the limits interact.

from test_junkie.runner import Runner
from ... import LoginSuite, CheckoutSuite, ApiSuite

runner = Runner([LoginSuite, CheckoutSuite, ApiSuite])
runner.run(suite_multithreading_limit=5, test_multithreading_limit=5)

Regression for a feature

Out of the suites you constructed the runner with, only the ones matching the given features run - everything else is skipped.

runner = Runner(suites=[...])
runner.run(features=["Login"])

Regression for a component

Same idea, scoped to components instead of features.

runner = Runner(suites=[...])
runner.run(components=["OAuth"])

Regression for tags

Runs the tag_config match modes against every test in scope - useful for "everything critical, except known-flaky" style slices.

runner = Runner(suites=[...])
runner.run(tag_config={"run_on_match_all": ["component_a", "critical"],
                       "skip_on_match_any": ["trivial", "known_failure"]})

Tests assigned to a specific person

Scoped to owners - handy for "just run my stuff" during local development.

runner = Runner(suites=[...])
runner.run(owners=["John Doe", "Jane Doe"])

Specific tests

Pass the test functions directly if they're already imported:

runner = Runner(suites=[...])
runner.run(tests=[LoginSuite.positive_login, LoginSuite.negative_login])

...or use their names as strings if you'd rather not import them:

runner = Runner(suites=[...])
runner.run(tests=["positive_login", "negative_login"])

Canceling tests since 0.8a.8

If you're integrating Test Junkie into a bigger framework, you may want to programmatically stop test execution - a deploy got aborted, a CI job got killed, a user hit stop in your own tooling. Calling cancel() on the Runner does this gracefully: it triggers on_cancel and on_class_cancel rather than abruptly killing anything.

import threading
from test_junkie.runner import Runner
from ... import LoginSuite, CheckoutSuite, ApiSuite

runner = Runner([LoginSuite, CheckoutSuite, ApiSuite])
threading.Thread(target=runner.run, kwargs={"suite_multithreading_limit": 5}).start()

# elsewhere, once something tells you to abort:
runner.cancel()

Tests that are already running when you cancel finish their current run and aren't retried since 0.9a6 . Pressing Ctrl+C during tj run, or during run() on the main thread, cancels the same way. See Stopping a run with Ctrl+C.

How a cancelled run ends new since 0.9a8

• The run ends [CANCELLED] and exits with code 12, even if tests had already failed. Ctrl+C and Runner.cancel() end the same way.
• Suites the cancel came before list their tests as cancelled, in every mode, so the JSON and HTML totals match the console.
• A suite with a cancelled test is cancel, not success. A suite with a failed, errored or ignored test stays fail. A cancelled test doesn't count as unsuccessful.
• A KeyboardInterrupt raised by a hook or listener cancels the run like Ctrl+C, but @afterClass() and @afterGroup() still run.

SuiteObject() since 0.8a.8

Every listener event - on_in_progress, on_failure, on_complete, and the rest - hands you the suite's SuiteObject in its kwargs. It's your read access into what the suite is doing right now: how many tests it has, who owns it, what its current pass/fail counts are. Reach for it when a listener needs to make a decision based on the suite as a whole rather than just the one test that triggered the event - failing the build once a suite crosses an error threshold, or routing an alert to the right owner.

from test_junkie.listener import Listener

class MyTestListener(Listener):

    def __init__(self, **kwargs):
        Listener.__init__(self, **kwargs)

    def on_failure(self, **kwargs):
        properties = kwargs.get("properties")
        suite = properties["jm"]["jso"]
        # suite is the SuiteObject for whichever suite just failed a test
        if suite.has_unsuccessful_tests() and suite.get_owner() == "artur":
            send_alert(suite.get_class_name(), suite.get_owner())

Methods

Every getter takes no arguments unless noted. Jump straight to a row via its link in the sidebar-linked sections below, or scan the table.

get_data_by_tags() output

The one method above worth seeing a full example of - everything else returns a plain value. Each tag gets its own bucket with the same shape as _totals_; a test with no tags is grouped under null.

{
  "_totals_": {
    "performance": [0.0, 0.0, 0.001, 0.0, 0.0, 0.0],
    "exceptions": ["AssertionError()"],
    "retries": [2, 2, 1],
    "total": 6, "success": 2, "skip": 0,
    "fail": 4, "cancel": 0, "ignore": 0, "error": 0
  },
  "login": { "... same shape, scoped to tests tagged 'login' ..." },
  "smoke": { "... same shape, scoped to tests tagged 'smoke' ..." }
}

TestObject() since 0.8a.8

Every listener event hands you the test's TestObject alongside the SuiteObject it belongs to. Where SuiteObject answers questions about the suite as a whole, TestObject answers them for the one test that triggered the event - who owns it, whether it's about to retry, whether it's parameterized. Reach for it when a listener's behavior should depend on the specific test, not just the suite around it.

from test_junkie.listener import Listener

class MyTestListener(Listener):

    def __init__(self, **kwargs):
        Listener.__init__(self, **kwargs)

    def on_failure(self, **kwargs):
        properties = kwargs.get("properties")
        test = properties["jm"]["jto"]
        # test is the TestObject for whichever test just failed
        owner = str(test.get_owner())
        if test.is_qualified_for_retry():
            log.info("%s will retry, notifying %s" % (test.get_function_name(), owner))

@beforeTest() and @afterTest() hooks, and Rules, can get the same test too, as a read-only view: its getters return copies, so nothing they do changes the run.

Methods