Tutorials / Core features

Metadata and targeting

Every @Suite and @test can carry metadata: owner, feature, component, tags, priority. It's more than documentation: pass the same values back to the runner and they become filters, so you run exactly the subset you need without touching the tests.

Annotating suites and tests

checkout_suite.pypython
@Suite(feature="Checkout", owner="checkout-team")
class CheckoutSuite:

    @test(component="payment", tags=["smoke", "regression"], priority=1)
    def card_payment(self):
        ...

    @test(component="cart", tags=["regression"], priority=2, owner="cart-team")
    def add_to_cart(self):
        ...
  • feature is set on the suite; component, tags and priority on tests.
  • owner can be set on both: a test without its own owner= takes the suite's.

Execution priority

priority= orders tests; it doesn't filter them. Lower numbers run first. Within a suite the order is:

  • Tests with a priority=, lowest number first.
  • Then tests without a priority that may run in parallel (the default).
  • Then tests without a priority marked parallelized=False.
checkout_suite.pypython
@Suite()
class CheckoutSuite:

    @test(priority=1)
    def login(self): ...                     # 1st

    @test(priority=2)
    def add_to_cart(self): ...               # 2nd

    @test()
    def search(self): ...                    # 3rd: no priority

    @test(parallelized=False)
    def logout(self): ...                    # last: no priority, not parallel

@Suite(priority=...) orders suites the same way, and @Suite(order=TestOrder.RANDOM) shuffles a suite's tests instead (see random order and --seed).

Targeting at run time

Pass the same values to run(). Tests that don't match are reported as skipped:

run_ci.pypython
from test_junkie.runner import Runner

runner = Runner([CheckoutSuite, LoginSuite, SearchSuite])

# CI smoke pass: smoke-tagged tests in the payment and cart components
runner.run(components=["payment", "cart"], tag_config={"run_on_match_any": ["smoke"]})

# triage: only what the checkout team owns
runner.run(owners=["checkout-team"])

# nightly: everything
runner.run()

The same filters on the command line:

terminalbash
tj run -c payment cart -k smoke
tj run -o checkout-team
tj run -f Checkout -l smoke regression    # feature Checkout, tests tagged smoke AND regression

Tag filters

tag_config has four keys, and any combination can be used at once:

run.pypython
runner.run(tag_config={
    "run_on_match_any": ["smoke", "critical"],     # run if it has ANY of these tags
    "run_on_match_all": ["regression", "payment"], # run only if it has ALL of these tags
    "skip_on_match_any": ["wip"],                  # skip if it has ANY of these tags
    "skip_on_match_all": ["slow", "flaky"],        # skip only if it has ALL of these tags
})

On the command line they are -k / --tags-any, -l / --tags-all, -g / --skip-tags-any and -j / --skip-tags-all.

Auditing your metadata

Filters only work if the metadata is there. tj audit shows how tests break down by owner, feature, component or tag without running them, and --fail-on-gaps fails a CI build when tests are missing metadata:

terminalbash
tj audit owners
tj audit suites --fail-on-gaps owners,tags

Runtime metadata: Meta.update()

For values only known while the test runs (an order ID, a build number), call Meta.update() inside the test. The values end up in the test's metadata and in the reports.

Test without parameters

checkout_suite.pypython
from test_junkie.meta import Meta


@Suite()
class CheckoutSuite:

    @test(component="payment")
    def card_payment(self):
        order = checkout(card="4111111111111111")
        Meta.update(self, order_id=order["id"], amount=order["total"])
        assert order["status"] == "confirmed"

Parameterized test

Pass parameter= and/or suite_parameter= too, so the values go to the variant that is running. Without them they go to a record that belongs to no variant, and parallel variants overwrite each other there:

checkout_suite.pypython
@Suite(parameters=[{"env": "staging"}, {"env": "prod"}])
class CheckoutSuite:

    @test(parameters=[{"card": "visa"}, {"card": "amex"}])
    def card_payment(self, parameter, suite_parameter):
        order = checkout(card=parameter["card"], env=suite_parameter["env"])
        Meta.update(self, parameter=parameter, suite_parameter=suite_parameter,
                    order_id=order["id"], amount=order["total"])
        assert order["status"] == "confirmed"
Pass the parameters the test takes
A test with only test-level parameters passes parameter=parameter; with only suite parameters, suite_parameter=suite_parameter; with both, both.

Read the values after the run with test_object.get_meta(parameter, class_parameter), or see them in the HTML report.