1. Create test configurations which you can reuse and check in-to your codebase
2. Run tests using those configurations or set values explicitly via terminal at runtime
3. Auditing & searching through existing tests which can be integrated with Git hooks or to get more insights into your current test coverage

Test Junkie enforces suite based test architecture. Thus all tests must be defined within a class and that class must be decorated with @Suite(). See Basic Usage for an example. This decorator supports many different properties which allow you to optimize execution of your tests.

from test_junkie.decorators import Suite

@Suite()
class ExampleTestSuite:
    ...

Use this decorator to prioritize execution of a decorated function at the very beginning of a test @Suite().

If exception is thrown with in such decorated function, it will trigger on_ignore event for all of the tests in scope of the current @Suite() and none of the tests will run.

from test_junkie.decorators import beforeClass

...
@beforeClass()
def before_class():
    ...

This decorator will prioritize execution of a decorated function before every @test() case in the @Suite().

If exception is thrown with in such decorated function, it will trigger either the on_failure or the on_error event for the test in scope.

from test_junkie.decorators import beforeTest

...
@beforeTest()
def before_test():
...

This decorator tells Test Junkie that the function that is getting decorated is a test and all tests must be defined within a class. See Basic Usage for an example.

If code produces exception in the decorated function, it will trigger either the on_failure or the on_error event. Otherwise if the test is successful, on_success event will trigger.

from test_junkie.decorators import test

...
@test()
def example_test():
...

Use this decorator to de-prioritize execution of a decorated function for the very end of a @test() case.

If code produces exception in the decorated function, it will trigger either the on_failure or the on_error event.

@afterTest() runs even when @test() produces an exception. If @test() has already thrown an exception and @afterTest() produces one as well, in the event listener you can access both of the tracebacks and exception objects but in the general console output, only the @test() traceback will be displayed.

from test_junkie.decorators import afterTest

...
@afterTest()
def after_test():
...

Use this decorator to de-prioritize execution of a decorated function for the very end of a test suite. If suite retry or parameters are configured, @afterClass() will run for each retry/suite parameter.

If code produces exception in the decorated function, it won't have any effects on the status of the tests and it wont show up in the console output. However, depending on the exception thrown, either the on_after_class_failure or on_after_class_error event will get triggered and the ability to capture the exception is there if required.

from test_junkie.decorators import afterClass

...
@afterClass()
def after_class():
...

As soon as any one of the suites in the group is about to start running its tests, this group rule will be executed. In the event that an exception is produced, all of the suites in the group will be ignored and they will not be retried.

@GroupRules()
def group_rules(self):

    from ... import ExampleSuite1
    from ... import ExampleSuite2
    @beforeGroup([ExampleSuite1, ExampleSuite2])
    def before_group_a():
        # your code
        pass

    from ... import ExampleSuite3
    from ... import ExampleSuite4
    @beforeGroup([ExampleSuite3, ExampleSuite4])
    def before_group_b():
        # your code
        pass

As soon as all of the suites in the group are done running, this group rule will be executed. This rule applies to suites that were used to initiate the Runner instance. Meaning, even if you have 20 suites, and running only 1 of them and that 1 suite is part of an @afterGroup() rule that was configured for 20 suites, this rule will still get executed upon completion of that 1 suite. The other 19 suites will never run, thus the condition is considered satisfied.

If code produces exception in the decorated function, it won't have any effects on the status of the tests and it wont show up in the console output. However, depending on the exception thrown, either on_after_group_failure or on_after_group_error event will get triggered and the ability to capture the exception is there if required.

@GroupRules()
def group_rules(self):

    from ... import ExampleSuite1
    from ... import ExampleSuite2
    @afterGroup([ExampleSuite1, ExampleSuite2])
    def after_group_a():
        # your code
        pass

    from ... import ExampleSuite3
    from ... import ExampleSuite4
    @afterGroup([ExampleSuite3, ExampleSuite4])
    def after_group_b():
        # your code
        pass

Test Junkie extends skipping functionality at the test and at the suite level which can be leveraged using either the @test() or @Suite() decorator property: skip which can take a boolean or a function object.

If the boolean or the return of the function object is equal to True, the test or the entire suite will be skipped and, respectively, on_skip or on_class_skip event will fire.

from test_junkie.decorators import Suite, test

def custom_skip_function(meta):
    # meta is optional
    # do your processing
    return True  # or False but must return a boolean

@Suite(skip=True)
or
@Suite(skip=custom_skip_function)
class ExampleSuite:

    @test(skip=True)
    or
    @test(skip=custom_skip_function)
    def example_test(self):
        pass

In addition, at the @test() level, you can choose to skip @beforeTest() & @afterTest() decorators for individual tests in the suite.

In the similar manner you can skip before_test() & after_test() Rules if you have those defined for the given suite.

...
@test(skip_before_test=True,
      skip_after_test=True,
      skip_before_test_rule=True,
      skip_after_test_rule=True)
def example_test(self):
    pass
...

With Test Junkie you can retry tests in a generic way at test and suite level. In addition, you can further control retries based on the type of exceptions!

Suite retries - once initial execution of tests is completed and assuming there are unsuccessful tests and suite retry is higher than 1, Test Junkie will re-run this suite again up to the number of retries that you have set or until all of the tests pass (whichever comes first). With suite retries, @beforeClass() & @afterClass() will run again and test level retries will apply again.

Test retries - once initial execution of a test is completed and assuming that test was unsuccessful, it will be retried immediately up to the number of retries that you have set or until the test passes (whichever comes first).

from test_junkie.decorators import Suite, test

@Suite(retry=2)  # default is 1
class ExampleSuite:

    @test(retry=2)  # default is 1
    def example_test(self):
        # this test will run 4 times in total
        # suite_retry x test_retry = total test retries
        assert True is False

retry on specific exception

...
@test(retry=2, retry_on=[AssertionError])
def example_test(self):
    # will be retried
    assert True is False
...

no retry on specific exception

...
@test(retry=2, no_retry_on=[AssertionError])
def example_test(self):
    # won't be retried
    assert True is False
...

Suite parametrization - In order to parametrize a suite, you need to use the parameters property of the @Suite() decorator. Parametrization at suite level can be configured to apply to all of the specially decorated functions, like:

1. @beforeClass()
2. @beforeTest()
3. @test()
4. @afterTest()
5. @afterClass()

@Suite(parameters=[1, 2])
class ExampleSuite:

    @beforeClass()
    def before_class(self, suite_parameter):
        print(suite_parameter)

    @beforeTest()
    def before_test(self, suite_parameter):
        print(suite_parameter)

Test parametrization - While you can, certainly, use suite level parameters in the test, you can, also, parameterize the test with its own set of parameters. In order to parametrize a test, you need to use the parameters property of the @test() decorator and add parameter to the function's signature.

Tests can run with test and suite level parameters at the same time. Without any parameters at all or either only with suite or test level parameters as you see fit.

@Suite(parameters=[1, 2])
class ExampleSuite:

    @test(parameters=[10, 20])
    def example_test(self, parameter, suite_parameter):
        print(parameter)
        print(suite_parameter)

Inputs - for both @Suite() & @test(), parameters property takes a list object of any data types. However, it can also take a function or a method object - in which case it will run the object when time comes to execute the suite. This, coupled with parallel execution, can potentially save you a lot of time when running test suites which have slow running functions that generate parameters.

In addition, parameterized tests can be executed in parallel meaning that each parameter of that test can be running in parallel. This option is only available for test cases and not test suites. By default, this is turned off for every single test but you can set parallelized_parameters property of the @test() decorator to True in order to enable it.

...
@test(parallelized_parameters=True, parameters=[...])
def example_test(parameter):
    ...

When running in multi-threaded mode, you have the option to disable it for specific suites and/or tests. Set parallelized property of the @test() or @Suite() decorator to False and it will get executed sequentially with other similar suites/tests after all parallelized suites/tests have finished.

@Suite(parallelized=False)
class ExampleSuite:
    @test(parallelized=False)
    def example_test():
    ...

It's not uncommon for tests to run into conflicts if executed at the same time and you can see a hypothetical example of it in the advanced example. Test Junkie allows to configure specific suites/tests so they never get executed at the same time by leverage the pr (parallel restriction) property of the @test() or @Suite() decorator. It accepts either a list of class objects (for suite decorator) or method objects (for test decorator).

@Suite(pr=[TestSuiteA])
class TestSuiteB:
    @test(pr=[TestSuiteC.example_test])
    def example_test():
    ...

test related events

• on_in_progress
• on_success
• on_failure
• on_error
• on_ignore
• on_cancel
• on_skip
• on_complete

suite related events

• on_class_in_progress
• on_before_class_failure
• on_before_class_error
• on_after_class_failure
• on_after_class_error
• on_class_skip
• on_class_cancel
• on_class_ignore
• on_class_complete
• on_before_group_failure
• on_before_group_error
• on_after_group_failure
• on_after_group_error

In order to create a listener, create a new subclass of a Listener. After that, overwrite desired functions (events) to use in your framework. Every event that you overwrite, must accept **kwargs in the function signature.

Listeners can be assigned to specific suites and are supported by the @Suite() decorator, much like the Rules.

from test_junkie.listener import Listener

class MyTestListener(Listener):

    def __init__(self, **kwargs):

        Listener.__init__(self, **kwargs)
    ...

Event is triggered as soon as Test Junkie begins processing the @test(). If test is parameterized, this event will be trigger for each of the parameters as they get executed.

...
def on_in_progress(self, **kwargs):
    # your code
    print(kwargs)
    ...

Event is triggered after test has successfully executed, that means @beforeTest() (if applicable), @test(), and @afterTest() (if applicable) have ran without producing an exception.

...
def on_success(self, **kwargs):
    # your code
    print(kwargs)
    ...

Event is triggered after test has produced AssertionError. AssertionError must be unhandled and thrown during the code execution in functions decorated with either @beforeTest(), @test(), or @afterTest()

...
def on_failure(self, **kwargs):
    # exception & traceback available
    # your code
    print(kwargs)
    ...

Event is triggered after test has produced any exception other than AssertionError. Exception must be unhandled and thrown during the code execution in functions decorated with either @beforeTest(), @test(), or @afterTest()

...
def on_error(self, **kwargs):
    # exception & traceback available
    # your code
    print(kwargs)
    ...

Event is triggered when a function decorated with @beforeClass() produces an exception or when incorrect inputs are passed in for the @test() decorator properties.

...
def on_ignore(self, **kwargs):
    # exception & traceback available
    # your code
    print(kwargs)
    ...

Event is triggered sometime after test execution has been canceled for an active suite(s). For all of the remaining tests in the suite, this event will fire.

...
def on_cancel(self, **kwargs):
    # your code
    print(kwargs)
    ...

Event is triggered when tests are skipped. Skip is supported by the @test() decorator. Tests can, also, be skipped when regression is triggered for specific Features, Components, Tags, or Assignees.

...
def on_skip(self, **kwargs):
    # your code
    print(kwargs)
    ...

Event is triggered regardless of the outcome of the @test(). If test is parameterized, this event will be trigger for each of the parameters as they get processed and the tests finish.

...
def on_complete(self, **kwargs):
    # your code
    print(kwargs)
    ...

Event is triggered as soon as Test Junkie starts processing a @Suite().

...
def on_class_in_progress(self, **kwargs):
    # your code
    print(kwargs)
    ...

Event is triggered only when a function decorated with @beforeClass() produces AssertionError. It will, also, trigger on_ignore for all of the tests in scope of that suite.

...
def on_before_class_failure(self, **kwargs):
    # exception & traceback available
    # your code
    print(kwargs)
    ...

Event is triggered only when a function decorated with @beforeClass() produces exception other than AssertionError. It will, also, trigger on_ignore for all of the tests in scope of that suite.

...
def on_before_class_error(self, **kwargs):
    # exception & traceback available
    # your code
    print(kwargs)
    ...

Event is triggered only when a function decorated with @afterClass() produces AssertionError.

...
def on_after_class_failure(self, **kwargs):
    # exception & traceback available
    # your code
    print(kwargs)
    ...

Event is triggered only when a function decorated with @afterClass() produces exception other than AssertionError.

...
def on_after_class_error(self, **kwargs):
    # exception & traceback available
    # your code
    print(kwargs)
    ...

Event is triggered when suites are skipped. Skip is supported by the @Suite() decorator. Suites can, also, be skipped when regression is triggered for specific Features, Components, Tags, or Assignees.

...
def on_class_skip(self, **kwargs):
    # your code
    print(kwargs)
    ...

Event is triggered sometime after test execution has been canceled. This event will trigger for all of the test suites that were still in queue and not in progress at that time.

...
def on_class_cancel(self, **kwargs):
    # your code
    print(kwargs)
    ...

Event is triggered when Test Junkie detects bad arguments being used for @Suite() properties.

For example, if you pass in empty parameters list, it does not make sense to run any tests in the suite because its assumed that either the setup functions or the tests rely on those parameters and sense they are empty the test scenarios will not be able to complete thus Test Junkie will ignore the @Suite().

...
def on_class_ignore(self, **kwargs):
    # exception & traceback available
    # your code
    print(kwargs)
    ...

Event is triggered when Test Junkie is done running all of the tests within the @Suite() that includes retrying any failed tests and running all suite level parameters, if applicable. If class was skipped or canceled, this even wont trigger.

...
def on_class_complete(self, **kwargs):
    # your code
    print(kwargs)
    ...

Event is triggered when a @beforeGroup() rule produces an AssertionError.

...
def on_before_group_failure(self, **kwargs):
    # your code
    print(kwargs)
    ...

Event is triggered when a @beforeGroup() rule produces an exception other than AssertionError.

...
def on_before_group_error(self, **kwargs):
    # your code
    print(kwargs)
    ...

Event is triggered when an @afterGroup() rule produces an AssertionError.

...
def on_after_group_failure(self, **kwargs):
    # your code
    print(kwargs)
    ...

Event is triggered when an @afterGroup() rule produces an exception other than AssertionError.

...
def on_after_group_error(self, **kwargs):
    # your code
    print(kwargs)
    ...

Bellow is an example of how to access the SuiteObject from the on_failure event. You can use the same method in any of the events thought.

Remember that when calling any of the available methods to get data. When you assign the return value to a variable and if you later modify that variable, you may be modifying the actual value in the SuiteObject that Test Junkie is working with when running the tests. It is critical for the proper functionality of the TJ that those values are modified only by TJ, therefore, make a copy of the return value, especially if you are planning to manipulate it later.

from test_junkie.listener import Listener

class MyTestListener(Listener):

    def __init__(self, **kwargs):

        Listener.__init__(self, **kwargs)

    def on_failure(self, **kwargs):
        properties = kwargs.get("properties")
        test_junkie_suite_object = properties["jm"]["jso"]
        # ^ this is your TJ Suite Object, you can now use any of the
        # listed methods down bellow to get desired data from it
        # DO NOT OVERRIDE ANY OF ITS PROPERTIES
        # Always make a copy of the value you are getting.
        # For example, if return is an integer - this is how it would look
        total_tests = int(test_junkie_suite_object.get_test_count())
        # wrapping the return in the int(), makes new value instead of making
        # a pointer to the value returned otherwise
        # now you can add/subtract etc from the total_tests variable without
        # accidentally effecting TJ
...

available methods

• get_test_count
• get_class_name
• get_class_instance
• get_class_object
• get_class_module
• get_test_objects
• get_test_function_names
• get_test_components
• get_test_tags
• get_feature
• get_priority
• get_runtime

• get_retry_limit
• get_meta
• get_unsuccessful_tests
• has_unsuccessful_tests
• get_status
• get_owner
• get_parallel_restrictions
• get_parameters
• is_parallelized
• get_data_by_tags

Return Type: Integer
Function: get_test_count()
Example call: SuiteObject.get_test_count()

This call does not account for parameterized tests. Aka If you have 1 test with 10 different parameters and one test without any parameters in the same suite, this call will return 2.

2

Return Type: String
Function: get_class_name()
Example call: SuiteObject.get_class_name()

Use to get the name of the test suite.

ExampleSuite

Return Type: Class
Function: get_class_instance()
Example call: SuiteObject.get_class_instance()

Use to get the actual instance of the suite that you defined and all of its functions and variables. You can therefore manipulate and even impact the tests in that instance to a degree.

< your.test.framework.suites.ExampleSuite.ExampleSuite object at 0x000002166D868070 >

Return Type: Class
Function: get_class_object()
Example call: SuiteObject.get_class_object()

While get_class_instance is used to get the instance of the class, this method is used to just get the class object.

< your.test.framework.suites.ExampleSuite.ExampleSuite >

Return Type: String
Function: get_class_module()
Example call: SuiteObject.get_class_module()

Use to get the module of the test suite class.

your.test.framework.suites.ExampleSuite

Return Type: List
Function: get_test_objects()
Example call: SuiteObject.get_test_objects()

Use to get the list of the test functions within a test suite.

[< ExampleSuite.example_1 >, < ExampleSuite.example_2 >]

Return Type: List
Function: get_test_function_names()
Example call: SuiteObject.get_test_function_names()

Similar to get_test_objects but instead returns a list of strings, where strings are the names of the test functions.

['example_1', 'example_2']

Return Type: Set
Function: get_test_components()
Example call: SuiteObject.get_test_components()

Use to get a set of all the components under test in a suite. If you did not define any components for at least 1 test, None will always be returned, in addition to the components that you have defined. None is treated as a component, just undefined - this is OK.

{'Example', None}

Return Type: Set
Function: get_test_tags()
Example call: SuiteObject.get_test_tags()

Use to get a set of all the tags under test in a suite.

{'smoke', 'login'}

Return Type: String
Function: get_feature()
Example call: SuiteObject.get_feature()

Use to get the feature under test in the suite. Suite can only have one feature defines. But features can be made up of multiple components.

ExampleFeature

Return Type: Integer OR NoneType
Function: get_priority()
Example call: SuiteObject.get_priority()

Returns the priority of the suite. If priority is not defined for the suite, returns None.

1
None

Return Type: Float
Function: get_runtime()
Example call: SuiteObject.get_runtime()

Returns the time in seconds for how long a suite has been running for.

8.082839965820312

Return Type: Integer
Function: get_retry_limit()
Example call: SuiteObject.get_retry_limit()

Use to find out the retry limit for the suite.

2

Return Type: Dictionary
Function: get_meta()
Example call: SuiteObject.get_meta()

Used to retrieve the meta information of the suite if such was defined.

{"some_key": "some value"}

Return Type: List
Function: get_unsuccessful_tests()
Example call: SuiteObject.get_unsuccessful_tests()

Returns functions of the unsuccessful tests in the suite at the time.

 [<ExampleSuite.example_1>]

Return Type: Boolean
Function: has_unsuccessful_tests()
Example call: SuiteObject.has_unsuccessful_tests()

Use to find out if a suite has any tests that did not run successfully.

True

Return Type: String
Function: get_status()
Example call: SuiteObject.get_status()

Used to get the status of the suite at the time.

fail

Return Type: String
Function: get_owner()
Example call: SuiteObject.get_owner()

Owner is defined via meta, therefore you can obtain this value also with get_meta call and then access it via dictionary. However, this function will return the exact owner as a string.

Artur Spirin

Return Type: List
Function: get_parallel_restrictions()
Example call: SuiteObject.get_parallel_restrictions()

Used to get the list of suites that this suite should not be running with at the same time.

[]

Return Type: Boolean
Function: get_parameters()
Example call: SuiteObject.get_parameters()

Used to get suite parameters if such were defined.

["user1@example.com", "user2@example.com"]

Return Type: Boolean
Function: is_parallelized()
Example call: SuiteObject.is_parallelized()

Use to find out if a suite can run in parallel with other suites.

True

Return Type: Boolean
Function: get_data_by_tags()
Example call: SuiteObject.get_data_by_tags()

Test Junkie tracks a number of metrics for each suite, you can access most of them via this call.

{
   "_totals_":{
      "performance":[
         0.0,
         0.0,
         0.0009999275207519531,
         0.0,
         0.0,
         0.0,
         0.0,
         0.0,
         0.0
      ],
      "exceptions":[
         "AssertionError()",
         "AssertionError()",
         "AssertionError()",
         "AssertionError()",
         "AssertionError()",
         "AssertionError()",
         "AssertionError()"
      ],
      "retries":[
         2,
         2,
         2,
         1,
         1,
         1
      ],
      "total":6,
      "success":2,
      "skip":0,
      "fail":4,
      "cancel":0,
      "ignore":0,
      "error":0
   },
   "login":{
      "performance":[
         0.0,
         0.0,
         0.0009999275207519531,
         0.0,
         0.0,
         0.0,
         0.0
      ],
      "exceptions":[
         "AssertionError()",
         "AssertionError()",
         "AssertionError()",
         "AssertionError()",
         "AssertionError()",
         "AssertionError()",
         "AssertionError()"
      ],
      "retries":[
         2,
         2,
         2,
         1
      ],
      "total":4,
      "success":0,
      "skip":0,
      "fail":4,
      "cancel":0,
      "ignore":0,
      "error":0
   },
   "smoke":{
      "performance":[
         0.0,
         0.0
      ],
      "exceptions":[

      ],
      "retries":[
         1,
         1
      ],
      "total":2,
      "success":2,
      "skip":0,
      "fail":0,
      "cancel":0,
      "ignore":0,
      "error":0
   }
}