▸ Zero plugins. Full arsenal.
The building blocks every framework has — clean, decorator-based, zero boilerplate.
@Suite & @test
Organize tests in plain Python classes. No base class to extend — your IDE, linter, and debugger work as-is.
Lifecycle Hooks
@beforeClass, @afterClass, @beforeTest, @afterTest — each fires at the right scope, declared directly on the class.
Automatic Retries
retry=N on any test re-runs it on failure. Extendable with exception-type control when you need more precision.
Built-in Reports
HTML, XML, and JSON reports generated after every run. No plugins, no config files, no post-processing step.
Capabilities built into the core — not available through plugins in other frameworks.
GroupRules
Shared LifecycleDefine driver setup, teardown, and hooks once and attach them to many suites — no inheritance, no copy-paste, no conftest proliferation.
Rules subclass defines before_class, after_class, before_test, and after_test hooks; any suite that passes rules= inherits them — one change updates every suite.
@beforeClass and @afterClass — the Rules class wraps around them, not replacing them.
from test_junkie.rules import Rules class BrowserRules(Rules): def after_test(self, **kwargs): logs = Browser.get_log("browser") errors = [m["message"] for m in logs] if errors: raise AssertionError(str(errors)) Browser.shutdown() @Suite(rules=BrowserRules) class CheckoutSuite: ... @Suite(rules=BrowserRules) class LoginSuite: ...
Multi-layer Parametrization
CoverageParametrize at the suite level and the test level independently — write one test, run it across every combination of environments, browsers, and user roles.
@Suite(parameters=[ {"env": "staging"}, {"env": "prod"}, ]) class CheckoutSuite: @test(parameters=[ {"browser": "chrome"}, {"browser": "firefox"}, {"browser": "safari"}, ]) def test_checkout(self, parameter, suite_parameter): ... # 6 variants, 1 method
Built-in Parallel Execution
PerformanceNo plugin, no external orchestrator — run suites and tests simultaneously at the thread level with independent concurrency limits at each tier.
-T 10 -S 2 runs 10 suites concurrently, each with 2 parallel tests — suite and test thread limits are controlled separately.
# tj run -s checkout.py -T 10 -S 2 @Suite(parallelized=True) class CheckoutSuite: @test(parallelized=True) def test_add_to_cart(self): ... @test(parallelized=False — db writes) def test_place_order(self): ...
Exception-aware Retries
ReliabilityRetry only on specific exception types — not blindly on any failure. Stop masking real bugs with policies that retry everything.
retry_on=[ConnectionError, TimeoutError] silently retries infrastructure noise; an AssertionError always surfaces immediately.
no_retry_on is an explicit block list — security and data-integrity assertions can always fail fast, regardless of the retry count.
@test( retry=3, retry_on=[ ConnectionError, TimeoutError, ], no_retry_on=[ AssertionError, ], ) def test_payment_gateway(self): ... # infra flakiness retries quietly # real failures always surface
Structured Metadata & CI Targeting
ObservabilityFirst-class owner, component, and tags on every test for CI filtering; priority= controls execution order — lower number runs first.
tj run --tags smoke in CI, full suite locally — no test selection scripts to maintain as the repo grows.
meta(bug="TJ-42") inside a running test to attach a ticket to the failure dynamically — no post-run correlation needed.
@test( owner="checkout-team", component="payment", priority=1, tags=["smoke", "regression"], ) def test_checkout(self): meta(bug="TJ-123") ... # tj run --tags smoke
Accessible Objects & Listeners
ProgrammaticRun results are plain Python objects and per-suite listeners fire live — build Slack alerts, Jira tickets, or custom CI gates without touching a plugin system.
runner.summary.suites gives typed objects with exception text, timing, retry count, and owner — everything to build your own reporting on top.
Listener(on_failure=my_fn) fires the moment a test finishes — your notification pipeline runs live alongside the suite, not after it.
runner = Runner([CheckoutSuite]) runner.run() for suite in runner.summary.suites: for test in suite.tests: if test.is_failed(): slack.post( test.get_exception()) # your reporting, not ours
Minimal test file to full production setup. CLI command to fully programmatic. It's all the same API — add what you need, leave out what you don't.
from test_junkie.runner import Runner from test_junkie.decorators import Suite, test @Suite() class LoginSuite: @test() def test_valid_login(self): assert login("admin", "pass") @test() def test_invalid_password(self): assert login("admin", "wrong") is False Runner([LoginSuite]).run()
@Suite( parameters=[{"env": "staging"}, {"env": "prod"}], parallelized=True, listener=AlertListener, ) class CheckoutSuite: @beforeClass() def setup(self, suite_parameter): self.driver = new_driver(suite_parameter["env"]) @test( owner="checkout-team", component="payment", priority=1, tags=["smoke"], retry=3, retry_on=[ConnectionError], no_retry_on=[AssertionError], parameters=[{"browser": "chrome"}, {"browser": "safari"}], parallelized=True, ) def test_checkout(self, parameter, suite_parameter): ...
# Run all tests in a directory $ tj run -s tests/ # CI: smoke tests only $ tj run -s tests/ --tags smoke # Parallel: 10 suite threads, 2 test threads $ tj run -s tests/ -T 10 -S 2 # Filter by owner and component $ tj run -s tests/ --owners checkout-team --components payment
from test_junkie.runner import Runner from test_junkie.constants import TestCategory runner = Runner([CheckoutSuite, LoginSuite]) runner.run() for suite in runner.get_executed_suites(): for test_obj in suite.get_test_objects(): metrics = test_obj.metrics.get_metrics() for _, by_param in metrics.items(): for _, data in by_param.items(): if data["status"] == TestCategory.FAIL: slack.notify(data["exceptions"][-1]) runner.get_html_report("./report.html") runner.get_xml_report("./report.xml")
Feature-by-feature against pytest, unittest, and Robot Framework — what ships in the box versus what requires plugins, workarounds, or simply isn't possible.
| Feature | test_junkie | pytest | unittest | Robot Framework |
|---|---|---|---|---|
| Execution & Resilience | ||||
| Parallel execution |
✓ Built-in
Thread-based; suite concurrency and test concurrency set independently; parallelized=False opts any test out without touching others
|
~ pytest-xdist Process-based workers; distributes test collection across processes; no dual-level thread limits; no per-test opt-out |
✗ Not supported
Sequential only; unittest-parallel exists as a third-party tool
|
~ Plugin (pabot)
Subprocess-based via robotframework-pabot; suite-level by default, test-level with --testlevelsplit; PabotLib for inter-process resource locking
|
| Test retries |
✓ Built-in
retry=N at test or suite level; on parametrized tests only the failing variants retry — not the full parameter set
|
~ pytest-rerunfailures
--reruns N; retries on any failure; entire parametrized test reruns if any variant fails
|
✗ Not supported |
~ Plugin
robotframework-retryfailed adds tag-driven per-test retry counts; --rerunfailed + rebot --merge for multi-pass reruns; no built-in retry decorator
|
| Exception-type retry controlUnique |
✓ Built-in
retry_on=[ExceptionClass] retries only on matched exception types; no_retry_on=[ExceptionClass] blocks specific classes from ever retrying
|
✗ Not supported
pytest-rerunfailures --only-rerun REGEX matches on exception message text, not exception class; approximate and easily wrong
|
✗ Not supported | ✗ Not supported Neither native retry nor any plugin filters retries by exception type; no equivalent exists in the RF ecosystem |
| Test Design | ||||
| Parametrized execution |
✓ Built-in
Dict-based params via @test(parameters=[...]); accepts a callable as provider for lazy/dynamic generation
|
✓ Built-in
@pytest.mark.parametrize; mature and widely used; stacking decorators produces a cartesian product at test level
|
~ Partial
self.subTest() context manager records sub-results but lifecycle hooks don't re-run; no decorator; variants can't be retried independently
|
~ Partial
Test Templates with inline data rows or FOR loops; robotframework-datadriver drives templates from CSV/Excel; no dict-based params or callable provider
|
| Suite-level × test-level cross-multiplyUnique |
✓ Built-in
@Suite(parameters=[...]) × @test(parameters=[...]) cross-multiply; 2 environments × 3 browsers = 6 independent variants, each tracked and retried separately
|
✗ Not supported
No class-level parametrize decorator; indirect fixture parametrization via conftest.py is possible but has no cross-multiply model and no per-variant retry
|
✗ Not supported |
✗ Not supported
No native suite-level × test-level Cartesian product; robotframework-datadriver operates at test level only; all variants must be written or generated explicitly
|
| Shared lifecycle across suitesUnique |
✓ Built-in
@GroupRules attaches shared before/after hooks to a named set of suites across any files; includes @beforeGroup / @afterGroup events
|
~ Partial
conftest.py fixtures scoped to session / module / class; scope is directory-based, not a named class set; no @beforeGroup equivalent
|
~ Partial
setUpModule() / tearDownModule() at module level only; no cross-file class grouping without inheritance
|
~ Partial
__init__.robot init files apply Suite/Test Setup to all suites in a directory tree; Resource files share keywords across suites; directory-scoped, not attachable to a named cross-directory set
|
| Metadata & Targeting | ||||
| Structured test metadata |
✓ Built-in
First-class owner, component, feature, priority, tags; priority also controls execution order in parallel runs
|
~ Partial
@pytest.mark.X free-form string markers; no first-class owner / component / priority; markers must be registered in pytest.ini; allure-pytest plugin adds structure
|
✗ Not supported Tests identified by class + method name only; no metadata or tagging system |
~ Partial
Built-in [Tags] and suite-wide Test Tags; [Metadata] for free name-value pairs; owner / component / feature / priority encoded as tag strings by convention (owner:john), not structured first-class fields
|
| Execution priority order |
✓ Built-in
priority= on @Suite and @test; lower number runs first; three tiers: priority-set (ascending) → parallel / no priority → sequential / no priority
|
~ Plugin
pytest-ordering; @pytest.mark.order(N) or --randomly-seed; no built-in priority= field
|
✗ Not supported
Alphabetical order by default; sortTestMethodsUsing for custom sort; no priority field
|
✗ Not supported
File / alphabetical order by default; --randomize for randomization; no built-in priority field
|
| CLI targeting & run filters |
✓ Built-in
--tags, --owners, --components, --features; AND / OR match modes; same filters available in Runner.run() programmatically; priority= controls execution order, not filtering
|
~ Partial
-m "marker" + -k "expression"; no built-in --owner / --component; multi-dimension filtering needs plugins or custom conftest
|
~ Partial
Fully-qualified name selection only (python -m unittest Suite.test_name); no metadata-based filtering
|
~ Partial
robot --include TAG, --exclude TAG, --test NAME, --suite NAME; AND/OR/NOT tag logic; because owner/component/priority are not structured fields, no dedicated flags exist for them
|
| Runtime metadata updatesUnique |
✓ Built-in
Meta.update() inside a running test can change owner, tags, component, linked bug ticket, or expected result dynamically mid-run
|
✗ Not supported
request.node.user_properties.append(...) is append-only; markers cannot be added or changed once the test is running
|
✗ Not supported |
~ Partial
Set Tags / Remove Tags BuiltIn keywords modify a test's tags during execution; no equivalent for attaching arbitrary key-value pairs to the result record mid-run
|
| Results & Observability | ||||
| HTML / XML / JSON reports | ✓ Built-in All three formats included; HTML dashboard includes CPU & memory graphs per run; zero configuration, zero plugins |
~ Partial
JUnit XML built-in (--junitxml); HTML requires pytest-html; JSON requires pytest-json-report; no system resource graphs
|
✗ Text only
TextTestRunner terminal output; xmlrunner third-party for XML; no HTML or JSON built-in
|
~ Partial
Native log.html + report.html + output.xml every run; XUnit XML via --xunit; no native JSON format; no resource graphs
|
| Per-suite event listeners |
✓ Built-in
Listener class attached per suite; 21 named events covering test and suite lifecycle; fires live during the run — not post-process
|
~ Partial
conftest.py hooks (pytest_runtest_logreport etc.); global scope, not per-class; requires understanding the plugin protocol
|
~ Partial
Subclass TestResult with addSuccess / addFailure / addError; no named event system; requires custom runner wiring
|
✓ Built-in
Listener API v2/v3 in core; events: start_suite, end_suite, start_test, end_test, start_keyword, and more; v3 passes live model objects; attached via --listener MyListener
|
| Programmatic result access |
✓ Built-in
runner.get_executed_suites() returns typed SuiteObject / TestObject with timing, retry count, exception data, owner, tags, and parameter sets
|
✗ Not in core
Primary interface is CLI; pytest-json-report plugin writes a JSON file; no typed result objects in core
|
~ Partial
TestResult.failures / .errors / .skipped — list of (TestCase, str) tuples; raw strings, no timing or structured exception
|
✓ Built-in
robot.api.ExecutionResult('output.xml') + ResultVisitor in core; full access to suite/test status, timing, messages; live access during run via v3 listener API model objects
|
| Project Health | ||||
| Maintenance status | ✓ Active MIT; Python 3.9–3.14; no plugins needed for parallel, parametrized, or retry workflows | ✓ Active BSD; Python 3.8+; the most widely used Python testing framework; 1,000+ community plugins |
✓ Stdlib
Part of CPython; zero external dependencies; stable but feature-frozen since subTest in Python 3.4
|
✓ Active Apache 2.0; RF 7.x; large enterprise ecosystem and standard library; keyword-driven DSL — tests written in custom syntax, not Python; extensive community library support |
1,000 tests, 1 ms test body, Python 3.12, median of 100 runs. No plugins required.