test_session.py

Overview

The [test_session.py](/projects/286/67371) file contains a comprehensive suite of tests designed to validate the behavior and robustness of pytest's session and collection mechanisms. It primarily leverages pytest's own testing utilities (`Pytester`, `MonkeyPatch`) to simulate various test scenarios and validate pytest's session lifecycle events, error handling, test collection, test execution order, plugin management, and command line options related to test session execution.

This file serves as a meta-test suite for pytest itself, ensuring that the pytest test session lifecycle and related features work correctly under diverse conditions, including syntax errors, import errors, plugin conflicts, and command line flags like [--exitfirst](/projects/286/67385), `--maxfail`, [--ignore](/projects/286/67223), and [--deselect](/projects/286/67223).

Classes and their Methods

Class: SessionTests

This class groups tests that verify fundamental pytest session behaviors, including test collection, event triggering, error reporting, and handling of edge cases like broken [__repr__](/projects/286/67223) implementations.

Methods:

• test_basic_testitem_events(self, pytester: Pytester) -> None
  Validates basic test item collection and outcomes (pass/fail), including function and class tests.

  • Uses pytester.makepyfile() to create a test module with several tests (passing, assertion failure, exception raising).

  • Runs the tests with inline_run() and asserts the expected passed, failed, and skipped counts.

  • Checks that the failed test node IDs match expected test function names.

  • Validates that pytest_itemcollected events are fired correctly.

• test_nested_import_error(self, pytester: Pytester) -> None
  Tests pytest's ability to report nested import errors inside a module.

  • Creates a module that imports another module which in turn imports a non-existent module.

  • Verifies that the import failure is captured and reported with the correct traceback.

• test_raises_output(self, pytester: Pytester) -> None
  Verifies that pytest.raises correctly reports a failure when the expected exception is not raised.

  • Runs a test with pytest.raises expecting a ValueError which does not occur.

  • Confirms that the failure message includes "DID NOT RAISE".

• test_syntax_error_module(self, pytester: Pytester) -> None
  Checks that syntax errors in test modules are detected and reported during collection.

• test_exit_first_problem(self, pytester: Pytester) -> None
  Tests the --exitfirst (-x) option which stops test execution after the first failure.

• test_maxfail(self, pytester: Pytester) -> None
  Tests the --maxfail option which stops test execution after a specified number of failures.

• test_broken_repr(self, pytester: Pytester) -> None
  Ensures pytest handles classes with broken repr implementations gracefully during test failure reporting.

• test_broken_repr_with_showlocals_verbose(self, pytester: Pytester) -> None
  Similar to above, but verifies that local variables are still shown correctly in verbose mode despite broken repr.

• test_skip_file_by_conftest(self, pytester: Pytester) -> None
  Verifies that files can be skipped during collection via a pytest_collect_file hook in conftest.py.

Class: TestNewSession(SessionTests)

This class extends `SessionTests` and adds tests focused on more complex session scenarios, such as execution order, collection-only mode, and interaction of multiple command-line options.

Methods:

• test_order_of_execution(self, pytester: Pytester) -> None
  Tests that test execution order matches expected behavior, including tests inside a class and standalone functions.

• test_collect_only_with_various_situations(self, pytester: Pytester) -> None
  Validates pytest's --collect-only mode and ensures correct collection events and failure reporting for syntax errors.

• test_minus_x_import_error(self, pytester: Pytester) -> None
  Tests that -x (exit after first failure) properly detects import errors during collection.

• test_minus_x_overridden_by_maxfail(self, pytester: Pytester) -> None
  Checks that --maxfail can override the -x option to allow more than one failure before stopping.

Standalone Test Functions

Several standalone test functions at module level verify pytest's plugin handling, command line options, and session attributes such as [shouldfail](/projects/286/67385) and [shouldstop](/projects/286/67210).

• test_plugin_specify(pytester: Pytester) -> None
  Tests that specifying a non-existent plugin via -p raises ImportError.

• test_plugin_already_exists(pytester: Pytester) -> None
  Tests loading an existing plugin and proper configuration/unconfiguration.

• test_exclude(pytester: Pytester) -> None
  Tests that --ignore option properly excludes directories containing syntax errors.

• test_exclude_glob(pytester: Pytester) -> None
  Similar to above, but using --ignore-glob with wildcard patterns.

• test_deselect(pytester: Pytester) -> None
  Tests --deselect option to skip specific tests by name or parameterized instances.

• test_sessionfinish_with_start(pytester: Pytester) -> None
  Ensures that pytest_sessionstart and pytest_sessionfinish hooks are called in correct order and that session state can be validated.

• test_collection_args_do_not_duplicate_modules(pytester: Pytester) -> None
  Tests that collecting the same test module multiple times via different arguments does not duplicate module collectors unless --keep-duplicates is used.

• test_rootdir_option_arg(pytester: Pytester, monkeypatch: MonkeyPatch, path: str) -> None
  Tests the --rootdir option's flexibility in specifying the root directory via relative or environment variable paths.

• test_rootdir_wrong_option_arg(pytester: Pytester) -> None
  Validates that an invalid --rootdir option value causes an appropriate error message.

• test_shouldfail_is_sticky(pytester: Pytester) -> None
  Tests that the session attribute shouldfail cannot be reset to False once set to True, ensuring consistent failure state.

• test_shouldstop_is_sticky(pytester: Pytester) -> None
  Similarly tests that shouldstop attribute is sticky and cannot be unset once marked.

Implementation Details and Algorithms

• The tests extensively use pytester's makepyfile, inline_run, inline_runsource, and runpytest APIs to dynamically create test files, run pytest sessions, and capture test results programmatically.

• Outcome assertions use listoutcomes() or countoutcomes() to verify the numbers of passed, failed, and skipped tests.

• The tests often inspect pytest's internal reporting mechanisms such as getcalls(), getreports(), and getfailedcollections() to validate collection and execution event sequences.

• Several tests simulate failure scenarios, such as syntax errors or import errors, by injecting malformed source code or imports of non-existent modules.

• Sticky attributes (shouldfail, shouldstop) are tested for immutability after being set to ensure pytest session state consistency.

• The test_order_of_execution verifies that test functions and class test methods run in the expected order by appending to shared lists and asserting their final contents.

• Tests around plugins verify that pytest correctly handles plugin loading errors and plugin lifecycle methods.

Interaction with Other System Components

• This file is part of pytest's own test suite and interacts primarily with pytest's core testing framework components.

• It uses the _pytest.pytester.Pytester utility for test file generation and pytest invocation, simulating real test runs.

• The tests interact with pytest session lifecycle hooks, test collection internals, and reporting objects.

• By exercising command line options like --exitfirst, --maxfail, --ignore, --deselect, and --rootdir, it indirectly tests pytest's command line parsing and session configuration subsystems.

• The plugin tests interact with pytest's plugin manager and configuration subsystems.

• The conftest.py creation and hook implementations test pytest's hook calling infrastructure.

Usage Examples

Example: Testing basic test item events

def test_basic_testitem_events(pytester: Pytester) -> None:
    tfile = pytester.makepyfile(
        '''
        def test_one():
            pass
        def test_fail():
            assert False
        '''
    )
    reprec = pytester.inline_run(tfile)
    passed, skipped, failed = reprec.listoutcomes()
    assert len(passed) == 1
    assert len(failed) == 1

Example: Using --maxfail option in tests

def test_maxfail(pytester: Pytester) -> None:
    reprec = pytester.inline_runsource(
        '''
        def test_one(): assert 0
        def test_two(): assert 0
        def test_three(): assert 0
        ''',
        '--maxfail=2',
    )
    passed, skipped, failed = reprec.countoutcomes()
    assert failed == 2

Mermaid Class Diagram

classDiagram
    class SessionTests {
        +test_basic_testitem_events(pytester: Pytester) void
        +test_nested_import_error(pytester: Pytester) void
        +test_raises_output(pytester: Pytester) void
        +test_syntax_error_module(pytester: Pytester) void
        +test_exit_first_problem(pytester: Pytester) void
        +test_maxfail(pytester: Pytester) void
        +test_broken_repr(pytester: Pytester) void
        +test_broken_repr_with_showlocals_verbose(pytester: Pytester) void
        +test_skip_file_by_conftest(pytester: Pytester) void
    }

    class TestNewSession {
        +test_order_of_execution(pytester: Pytester) void
        +test_collect_only_with_various_situations(pytester: Pytester) void
        +test_minus_x_import_error(pytester: Pytester) void
        +test_minus_x_overridden_by_maxfail(pytester: Pytester) void
    }

    SessionTests <|-- TestNewSession

Summary

[test_session.py](/projects/286/67371) is a critical part of pytest's own test infrastructure, ensuring that pytest's session management, test collection, execution order, error handling, and configuration options work correctly. It uses `Pytester` to dynamically create test scenarios and validate pytest's internal behaviors. The file is comprehensive and covers a wide variety of edge cases, contributing to pytest's robustness and reliability.