Tutorials / CI and the command line

Running Test Junkie in CI

Everything a CI job needs is in the tj command: settings committed with the project, exit codes the build can act on, reports to keep, and failures marked on the pull request.

1. Commit the settings with the project

Save where the tests are and how many threads to use in a tj.cfg at the root of the repository. tj run finds it from any folder in the project, locally and in CI:

tj config update -s tests -T 4 -S 2 --config ./tj.cfg
git add tj.cfg

If you keep your tooling in pyproject.toml, a [tool.test_junkie] table works too (Python 3.11+, or with tomli installed):

[tool.test_junkie]
sources = ["tests"]
test-multithreading-limit = 4

2. The workflow

name: Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v6
        with:
          python-version: "3.13"
      - run: pip install test-junkie -r requirements.txt

      # fail fast if a test was added without an owner
      - name: Metadata check
        run: tj audit suites --fail-on-gaps owners

      - name: Tests
        env:
          TEST_JUNKIE_HOME: ${{ runner.temp }}/tj_home  # keep config and temp files out of the workspace
        run: tj run --retry 2 --xml-report reports/ --html-report reports/ --json-report reports/

      - name: Keep the reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-reports
          path: reports/

• --retry 2 gives every failing test a second run in CI, against a shared environment that has hiccups, whatever its retry= says. Locally, tj run --no-retry shows the real failure rate.
• Reports go into one folder: reports/report.xml (JUnit, for CI test tabs), report.html (the dashboard) and report.json (for scripts).
• Annotations need nothing: under GitHub Actions, tj run prints an ::error line for every failed test, and GitHub marks the failing line in the pull request.

3. Exit codes

The step fails when tj run exits with anything but 0:

tj audit --fail-on-gaps exits 1 when tests are missing the metadata you listed, and 0 otherwise.

4. Make failures easy to read

CI logs aren't a terminal, so tj run prints plain lines without the live progress bars, and keeps what tests print and log out of the way: it's only shown under the tests that failed, next to their traceback. For a shorter log, -q prints only the problems and the result line.