Tutorials / Core features

Multi-layer parametrization

Parametrize at the suite level and the test level. Write one test method, run it across every combination of environments, browsers or user roles, and get per-variant results, retries and reports.

Suite-level parameters

parameters= on @Suite runs the suite once per entry. Any hook or test that declares a suite_parameter argument receives the current entry:

checkout_suite.pypython
@Suite(parameters=[
    {"env": "https://staging.shop.example"},
    {"env": "https://shop.example"},
])
class CheckoutSuite:

    @beforeClass()
    def setup(self, suite_parameter):
        self.base_url = suite_parameter["env"]

    @test()
    def add_to_cart(self, suite_parameter):
        # runs twice: once against staging, once against production
        checkout(self.base_url)

Test-level parameters

parameters= on @test runs that test once per entry, within each suite run. The current entry is passed as parameter:

checkout_suite.pypython
@Suite()
class CheckoutSuite:

    @test(parameters=[
        {"browser": "chrome"},
        {"browser": "firefox"},
        {"browser": "safari"},
    ])
    def checkout_works(self, parameter):
        driver = new_driver(parameter["browser"])
        ...

Parameters can also come from a function that builds the list when the run starts, e.g. from a CSV file or a database: see data-driven testing.

Both layers together: every combination

With parameters= on both, Test Junkie runs every combination. Two environments × three browsers gives six variants from one method:

checkout_suite.pypython
@Suite(parameters=[
    {"env": "https://staging.shop.example"},
    {"env": "https://shop.example"},
])
class CheckoutSuite:

    @test(parameters=[
        {"browser": "chrome"},
        {"browser": "firefox"},
        {"browser": "safari"},
    ])
    def checkout_works(self, parameter, suite_parameter):
        driver = new_driver(parameter["browser"])
        checkout(driver, suite_parameter["env"])

# 6 variants: staging/chrome, staging/firefox, staging/safari,
#             production/chrome, production/firefox, production/safari
parameter and suite_parameter are passed by name
Test Junkie looks at the function's arguments and passes the current values to the ones named parameter and suite_parameter. The names are fixed: read parameter["browser"], there's no browser argument.

Which functions take part

Whether a test runs once or once per suite parameter depends on its own arguments:

  • Tests that declare suite_parameter run once per suite parameter. A test that doesn't declare it runs once in total, however many suite parameters there are.
  • @beforeClass and @afterClass run once per suite parameter whether or not they declare it; declare it to read the value.
  • @beforeTest and @afterTest run around every test run. If they declare suite_parameter, it is None around a test that doesn't take suite parameters. They can't declare parameter.
checkout_suite.pypython
@Suite(parameters=[{"env": "staging"}, {"env": "prod"}])
class CheckoutSuite:

    @beforeClass()
    def setup(self, suite_parameter):
        # runs twice: staging, then prod
        self.base_url = urls[suite_parameter["env"]]

    @afterClass()
    def teardown(self):
        # no suite_parameter, still runs twice: once per suite run
        cleanup()

    @beforeTest()
    def before_each(self, suite_parameter):
        # None before homepage_loads, which doesn't take suite parameters
        if suite_parameter is not None:
            log("starting a test on {}".format(suite_parameter["env"]))

    @test()
    def homepage_loads(self):
        # no suite_parameter: runs once in total
        ...

    @test()
    def checkout_works(self, suite_parameter):
        # runs twice: staging and prod
        checkout(self.base_url)
Check for None in shared hooks
Without the if suite_parameter is not None check above, suite_parameter["env"] raises a TypeError before homepage_loads, and that test is reported as an error without running.

Per-variant results and retries

Each variant has its own result. If the production/safari variant fails, the others are unaffected, and retries apply per variant: only the variants that failed run again.

checkout_suite.pypython
@test(
    retry=2,
    retry_on=[ConnectionError],
    parameters=[{"browser": "chrome"}, {"browser": "safari"}],
)
def checkout_works(self, parameter):
    ...

# if the safari variant raises a ConnectionError, only it runs again;
# the chrome variant's result stands

In the console and the reports every variant is listed with its values, e.g. CheckoutSuite.checkout_works [suite: {'env': 'prod'}, {'browser': 'safari'}].