Tutorials / Selenium WebDriver series

Part 5: Catching JavaScript errors in Selenium tests

JavaScript errors are some of the cheapest bugs to catch in UI tests: the browser already logs them. One Test Junkie rule checks the browser console after every test and fails the test that caused an error, without touching a single test.

The rule

A Rules class holds hooks that Test Junkie runs around the tests of every suite it's attached to. after_test runs after each test, in the same thread, so Browser.get_driver() returns the browser that test used:

tests/TestRules.pypython
from test_junkie.rules import Rules

from src.pages.Browser import Browser


class TestRules(Rules):

    # third-party scripts that fail now and then, and aren't the site's own bugs
    IGNORED = ("buttons.github.io",)

    def after_test(self, **kwargs):
        # runs after every test in the suite, in the test's own thread - so it sees that test's browser
        if not Browser.is_open():
            return  # the test failed before it opened one
        try:
            logs = Browser.get_driver().get_log("browser")
        finally:
            Browser.shutdown()  # always close the browser, even when the check below fails the test
        errors = [entry["message"] for entry in logs if entry["level"] == "SEVERE"
                  and not any(ignored in entry["message"] for ignored in TestRules.IGNORED)]
        if errors:
            name = kwargs["test"].get_function_name()
            raise AssertionError("{} JavaScript error(s) during {}: {}".format(len(errors), name, errors))
  • Only SEVERE messages count: errors, not warnings or console.log output.
  • IGNORED skips third-party scripts that fail now and then and aren't the site's own bugs. On this site that's the GitHub star button, which caught us in testing.
  • The browser is closed in finally, so it's closed even when the check fails the test.
  • kwargs["test"] is the test that just ran, for a clear failure message.

Attach it to the suites

Add rules=TestRules to both suites, and remove their @afterTest hooks: a suite's own @afterTest runs before the rule's after_test, so if it still closed the browser, the rule would have nothing left to check. The rule closes the browser now.

tests/suites/NavigationSuite.pypython
from tests.TestRules import TestRules


@Suite(parameters=[HomePage, DocumentationPage, AboutPage], rules=TestRules, feature="Navigation")
class NavigationSuite:
    ...  # the tests from part 4, without the @afterTest hook
tests/suites/DocumentationSuite.pypython
from tests.TestRules import TestRules


@Suite(rules=TestRules, feature="Documentation")
class DocumentationSuite:
    ...  # the tests from part 4, without the @afterTest hook

What a JavaScript error looks like

Before IGNORED was added, a run where the GitHub button's script didn't load failed like this:

tj runtext
Problems 1 ─────────────────────────────────────────────────────────────────────

[FAIL] NavigationSuite.header_navigation  [suite: HomePage, about]
       tests/TestRules.py:19 · 1 run · 2.27s

  Traceback
  │ tests/TestRules.py:19 in after_test
  │     raise AssertionError("{} JavaScript error(s) during {}: {}".format(len(errors), name, errors))
  │ AssertionError: 1 JavaScript error(s) during header_navigation:
  │ ['https://buttons.github.io/buttons.js - Failed to load resource: net::ERR_SSL_PROTOCOL_ERROR']

The failure names the test, the page it started from and the link it clicked, so it's clear where to look.

Where to go from here

  • Take a screenshot when a test fails, in the same rule: shared setup and teardown shows how.
  • Run the suites in CI on every pull request: running Test Junkie in CI.
  • Add a page: a page object (part 3) and its start page in NavigationSuite's parameters, and every link from it is tested.

Series

« Part 4: Writing tests All tutorials »