test_runner.py

Overview

The [test_runner.py](/projects/286/67339) file is a comprehensive test suite for various aspects of the `pytest` testing framework, focusing primarily on the test execution, setup and teardown mechanisms, reporting, and integration of specific pytest features such as [importorskip](/projects/286/67385), outcome exceptions, and environment variable management during tests.

This file contains a collection of test classes and standalone test functions that validate the internal workings and user-facing behaviors of pytest's test runner system. It ensures that test setup and teardown states are correctly handled, that test outcomes (pass, fail, skip) are properly reported, and that pytest hooks and exceptions behave as expected.

Classes and Their Functionality

1. TestSetupState

This class contains tests that verify the correctness of pytest's internal `SetupState` management, which is responsible for setting up test environments, managing finalizers, and tearing down tests.

• Key Methods:

  • test_setup(self, pytester: Pytester) -> None
    Tests that setup adds correct finalizers and teardown properly cleans up resources.

  • test_teardown_exact_stack_empty(self, pytester: Pytester) -> None
    Ensures that multiple calls to teardown_exact do not cause errors when the internal stack is empty.

  • test_setup_fails_and_failure_is_cached(self, pytester: Pytester) -> None
    Verifies that if setup fails, the failure is cached and subsequent setups raise the same error.

  • test_teardown_multiple_one_fails(self, pytester: Pytester) -> None
    Checks that if multiple finalizers are registered and one fails during teardown, the exception is raised and other finalizers still run.

  • test_teardown_multiple_fail(self, pytester: Pytester) -> None
    Tests that multiple failing finalizers raise an ExceptionGroup containing all exceptions.

  • test_teardown_multiple_scopes_one_fails(self, pytester: Pytester) -> None
    Validates teardown behavior when finalizers are registered at different test scopes (e.g., module, function).

  • test_teardown_multiple_scopes_several_fail(self, pytester: Pytester) -> None
    Tests multiple failing finalizers across scopes, expecting nested ExceptionGroup.

  • test_cached_exception_doesnt_get_longer(self, pytester: Pytester) -> None
    Regression test ensuring cached exceptions do not grow in size during repeated setup failures.

2. BaseFunctionalTests

This class contains functional tests that simulate real test runs using `pytester` to validate outcomes and reporting.

• Key Methods:

  • test_passfunction(self, pytester: Pytester) -> None
    Tests that a passing test function is reported as passed.

  • test_failfunction(self, pytester: Pytester) -> None
    Tests that a failing test function is reported as failed.

  • test_skipfunction(self, pytester: Pytester) -> None
    Tests that a test function that calls pytest.skip is reported as skipped.

  • test_skip_in_setup_function(self, pytester: Pytester) -> None
    Tests skip behavior when pytest.skip is called in a setup function.

  • test_failure_in_setup_function(self, pytester: Pytester) -> None
    Tests failure reporting when an exception is raised in a setup function.

  • test_failure_in_teardown_function(self, pytester: Pytester) -> None
    Tests failure reporting when an exception is raised in a teardown function.

  • test_custom_failure_repr(self, pytester: Pytester) -> None
    Tests that a custom failure representation overrides the default representation.

  • test_teardown_final_returncode(self, pytester: Pytester) -> None
    Checks that a failure in teardown results in a non-zero pytest exit code.

  • test_logstart_logfinish_hooks(self, pytester: Pytester) -> None
    Verifies that pytest's pytest_runtest_logstart and pytest_runtest_logfinish hooks are called correctly.

  • test_exact_teardown_issue90(self, pytester: Pytester) -> None
    Regression test for correct exception info handling during teardown.

  • test_exact_teardown_issue1206(self, pytester: Pytester) -> None
    Tests error handling when teardown_method has the wrong signature.

  • test_failure_in_setup_function_ignores_custom_repr(self, pytester: Pytester) -> None
    Ensures failure in setup does not use custom failure representation.

  • test_systemexit_does_not_bail_out(self, pytester: Pytester) -> None
    Ensures SystemExit exceptions raised in tests are caught and reported as failures, not exit pytest.

  • test_exit_propagates(self, pytester: Pytester) -> None
    Checks that pytest.exit.Exception is properly propagated and not swallowed.

3. TestExecutionNonForked(BaseFunctionalTests)

Subclass of `BaseFunctionalTests` that overrides the test runner to use [runtestprotocol](/projects/286/67543) directly without forking subprocesses.

• Additional Method:

  • test_keyboardinterrupt_propagates(self, pytester: Pytester) -> None
    Ensures a KeyboardInterrupt raised in a test raises up and is not swallowed.

4. TestSessionReports

Tests around session-level report collection.

• Key Method:

  • test_collect_result(self, pytester: Pytester) -> None
    Tests that pytest can collect results from a module containing functions and classes, and that the collected report contains correct location and result information.

5. TestReportContents

Tests the user-level API of pytest's `TestReport` objects.

• Key Methods:

  • test_longreprtext_pass(self, pytester: Pytester) -> None
    Verifies that the long representation text of a passing test is empty.

  • test_longreprtext_skip(self, pytester: Pytester) -> None
    Tests that longreprtext can handle skip reports that are not strings.

  • test_longreprtext_collect_skip(self, pytester: Pytester) -> None
    Similar to above but for CollectReport skip cases.

  • test_longreprtext_failure(self, pytester: Pytester) -> None
    Tests that failure reports contain assert failure details.

  • test_captured_text(self, pytester: Pytester) -> None
    Checks that captured stdout and stderr are properly recorded in setup, call, and teardown phases.

  • test_no_captured_text(self, pytester: Pytester) -> None
    Verifies that tests with no captured output have empty capture attributes.

  • test_longrepr_type(self, pytester: Pytester) -> None
    Confirms that failure longrepr is represented as an ExceptionChainRepr object.

6. TestImportOrSkipExcType

Tests related to the behavior of [pytest.importorskip](/projects/286/67385) and its interaction with import errors and warnings.

• Tests that modules which don't exist cause skip without warnings.

• Tests modules that raise ImportError during import cause warnings.

• Tests suppression of warnings with the exc_type parameter.

• Tests integration of warnings in pytest output.

Important Functions and Tests (Standalone)

• test_report_extra_parameters(reporttype: type[reports.BaseReport])
  Tests that pytest report types accept extra parameters.

• test_callinfo()
  Tests the creation and string representation of runner.CallInfo objects from successful and failed calls.

• test_outcomeexception_exceptionattributes() and test_outcomeexception_passes_except_Exception()
  Tests the behavior of OutcomeException.

• test_pytest_exit() and test_pytest_fail()
  Tests pytest.exit and pytest.fail raising the appropriate exceptions.

• Various tests for pytest.importorskip with different scenarios including minversion checks, module-level skips, and custom skip reasons.

• test_pytest_cmdline_main()
  Tests running pytest programmatically via pytest.cmdline.main.

• Tests related to unicode handling in failure reports, exception printing, and error source retrieval.

• test_store_except_info_on_error()
  Ensures exception information is stored in sys.last_* variables after test failures.

• test_current_test_env_var()
  Ensures the environment variable PYTEST_CURRENT_TEST is set correctly during setup, call, and teardown phases.

• Tests for pytest's exit status codes when no tests are collected or when tests fail.

• Tests for failure reporting in setup and teardown phases, including correct traceback presentation.

Implementation Details and Algorithms

• The test methods use the pytester fixture extensively, which is a utility for creating test files, running pytest, and inspecting the results programmatically.

• Use of ExceptionGroup (Python 3.11+ or backported) to capture multiple exceptions occurring during teardown.

• Use of partial functions and finalizers to simulate teardown scenarios with multiple cleanup steps.

• Tests simulate pytest internals like SetupState, CallInfo, and report objects to validate their behavior under various conditions.

• Integration tests check the interaction of pytest with system environment variables, exceptions, and output capturing mechanisms.

• Some tests check backward compatibility and regression issues (e.g., issue #12204, #11706, #11523).

Interactions with Other Parts of the System

• pytest internals: This file tests core pytest internals such as runner, reports, outcomes, and SetupState.

• pytest hooks: Several tests verify pytest hook implementations and their correct invocation order (pytest_runtest_setup, pytest_runtest_call, pytest_runtest_teardown).

• Exception handling: Validates pytest's mechanisms for capturing, representing, and propagating exceptions during test setup, call, and teardown.

• Command-line execution: Tests programmatic invocation of pytest and the resulting exit codes.

• Environment variables: Ensures pytest sets and cleans up environment variables related to the currently running test.

• Test reporting: Validates how test outcomes are reported and how captured output is included in reports.

• Import system: Tests the importorskip helper that conditionally imports modules or skips tests when dependencies are missing.

Overall, this file ensures the robustness of pytest's test execution and reporting machinery, critical for reliable test runs.

Usage Examples

# Example: Running a test function and checking it passes
def test_passfunction_example(pytester: Pytester) -> None:
    reports = pytester.runitem("def test_func(): pass")
    rep = reports[1]
    assert rep.passed
    assert rep.outcome == "passed"

# Example: Adding a finalizer and checking it runs during teardown
def test_finalizer_runs(pytester: Pytester) -> None:
    item = pytester.getitem("def test_func(): pass")
    ss = item.session._setupstate
    ss.setup(item)
    called = []
    ss.addfinalizer(lambda: called.append(True), item)
    ss.teardown_exact(None)
    assert called == [True]

Mermaid Class Diagram

classDiagram
    class TestSetupState {
        +test_setup(pytester)
        +test_teardown_exact_stack_empty(pytester)
        +test_setup_fails_and_failure_is_cached(pytester)
        +test_teardown_multiple_one_fails(pytester)
        +test_teardown_multiple_fail(pytester)
        +test_teardown_multiple_scopes_one_fails(pytester)
        +test_teardown_multiple_scopes_several_fail(pytester)
        +test_cached_exception_doesnt_get_longer(pytester)
    }

    class BaseFunctionalTests {
        +test_passfunction(pytester)
        +test_failfunction(pytester)
        +test_skipfunction(pytester)
        +test_skip_in_setup_function(pytester)
        +test_failure_in_setup_function(pytester)
        +test_failure_in_teardown_function(pytester)
        +test_custom_failure_repr(pytester)
        +test_teardown_final_returncode(pytester)
        +test_logstart_logfinish_hooks(pytester)
        +test_exact_teardown_issue90(pytester)
        +test_exact_teardown_issue1206(pytester)
        +test_failure_in_setup_function_ignores_custom_repr(pytester)
        +test_systemexit_does_not_bail_out(pytester)
        +test_exit_propagates(pytester)
    }

    class TestExecutionNonForked {
        +getrunner()
        +test_keyboardinterrupt_propagates(pytester)
    }
    TestExecutionNonForked --|> BaseFunctionalTests

    class TestSessionReports {
        +test_collect_result(pytester)
    }

    class TestReportContents {
        +test_longreprtext_pass(pytester)
        +test_longreprtext_skip(pytester)
        +test_longreprtext_collect_skip(pytester)
        +test_longreprtext_failure(pytester)
        +test_captured_text(pytester)
        +test_no_captured_text(pytester)
        +test_longrepr_type(pytester)
    }

    class TestImportOrSkipExcType {
        +test_no_warning()
        +test_import_error_with_warning(pytester)
        +test_import_error_suppress_warning(pytester)
        +test_warning_integration(pytester)
    }

Summary

[test_runner.py](/projects/286/67339) is a vital internal test module for the pytest framework itself, designed to rigorously verify the correctness of test execution, setup/teardown logic, reporting, exception handling, and helper functions like [importorskip](/projects/286/67385). By testing both pytest internals and user-facing behaviors, it helps maintain pytest's reliability and feature correctness across versions and Python environments. It interacts closely with pytest's core classes and hooks, using the `pytester` utility to simulate test runs programmatically and capture their outcomes.