Test Execution and Reporting

This module orchestrates the execution lifecycle of tests within the system and manages detailed reporting of test outcomes. It is responsible for running tests through their setup, call (execution), and teardown phases, capturing results, handling exceptions, and communicating the test progress and results to the user via terminal output and structured XML reports.

Core Concepts and Purpose

Test Execution and Reporting addresses the fundamental need to:

Control the lifecycle of individual test items, ensuring setup and teardown procedures execute correctly around the test logic.
Capture detailed information about test outcomes, including pass/fail status, exceptions, captured output, and timing.
Provide clear, user-friendly feedback during and after test runs via terminal reporting.
Generate machine-readable test result artifacts (JUnit XML) for integration with CI/CD pipelines and other tools.

This module solves problems related to reliably running tests with proper isolation, reporting nuanced results, and integrating with other parts of the system that manage test discovery, fixtures, and plugins.

Module Workflow and Key Functionalities

Test Lifecycle Management

Test execution follows a structured protocol with three distinct phases:

Setup: Prepares the test environment, including fixture setup and any necessary preconditions.
Call: Executes the actual test function or method.
Teardown: Cleans up resources and environment changes made during setup or test execution.

The `runtestprotocol` function in [src/_pytest/runner.py](/projects/286/67339) manages these phases in order for each test item:

def runtestprotocol(item: Item, log: bool = True, nextitem: Item | None = None) -> list[TestReport]:
    rep = call_and_report(item, "setup", log)
    reports = [rep]
    if rep.passed:
        if not item.config.getoption("setuponly", False):
            reports.append(call_and_report(item, "call", log))
    reports.append(call_and_report(item, "teardown", log, nextitem=nextitem))
    return reports

call_and_report invokes the corresponding hook (pytest_runtest_setup, pytest_runtest_call, or pytest_runtest_teardown) and wraps the outcome in a TestReport.
The setup and teardown phases are carefully managed via a shared SetupState stack that ensures correct nesting and finalization of setup and teardown calls.
Environment variables such as PYTEST_CURRENT_TEST are updated to reflect the current test and phase, aiding external tooling and debugging.

Exception and Outcome Handling

The module captures exceptions during each phase and classifies outcomes as "passed", "failed", or "skipped".
Special handling exists for expected failures (xfail) and skips, with detailed representations maintained in TestReport objects for reporting.
Exceptions are wrapped in CallInfo instances that record timing and exception info, facilitating rich reporting and debugging support.

Reporting Infrastructure

Reporting is split into two main user-facing components:

Terminal Reporting (src/_pytest/terminal.py)
- Provides real-time feedback on test progress, including short status letters (e.g., ., F, s) and verbose output.
- Summarizes test session start, collection progress, individual test outcomes, failures, skips, and warnings.
- Supports configurable verbosity, coloring, and output styles.
- Integrates with pytest's hook system to receive and display test reports (pytest_runtest_logreport) and session events.
Example of terminal output decision logic:
```
def pytest_runtest_logreport(self, report: TestReport) -> None:
    res = TestShortLogReport(*self.config.hook.pytest_report_teststatus(report=report, config=self.config))
    category, letter, word = res.category, res.letter, res.word
    self._add_stats(category, [report])
    # Writes single letter or verbose word depending on verbosity
    self._tw.write(letter, **markup)
```

JUnit XML Reporting (src/_pytest/junitxml.py)

Generates XML reports conforming to JUnit schema for CI systems.
Accumulates test statistics, durations, and captures outputs and logs.
Writes detailed failure, skip, and error information into XML elements.
Supports customization of suite names, logging verbosity, and XML schema variants.
Registers as a pytest plugin and hooks into test report events to build the XML incrementally.

Example of handling a test report for XML:

def pytest_runtest_logreport(self, report: TestReport) -> None:
    if report.passed and report.when == "call":
        reporter = self._opentestcase(report)
        reporter.append_pass(report)
    elif report.failed:
        reporter = self._opentestcase(report)
        if report.when == "call":
            reporter.append_failure(report)
        else:
            reporter.append_error(report)
    elif report.skipped:
        reporter = self._opentestcase(report)
        reporter.append_skipped(report)
    self.update_testcase_duration(report)
    if report.when == "teardown":
        reporter.write_captured_output(report)
        self.finalize(report)

Interaction with Other System Components

Test Items and Collectors (src/_pytest/nodes.py, src/_pytest/python.py): The execution module receives Item instances representing individual tests discovered by the collection phase.
Fixtures (src/_pytest/fixtures.py): Setup and teardown phases execute fixture lifecycles, coordinating resource allocation and dependency injection.
Plugins and Hooks (src/_pytest/hookspec.py, src/_pytest/config/__init__.py): The execution and reporting phases leverage pytest's hook system for extensibility and integration.
Capturing Output (src/_pytest/capture.py, src/_pytest/logging.py): Captured stdout, stderr, and logs during test execution are collected and relayed to reports.
Assertion Rewriting (src/_pytest/assertion/rewrite.py): Enhanced assertion introspection enriches failure reports generated during test execution.
Terminal and XML Reporting (src/_pytest/terminal.py, src/_pytest/junitxml.py): These modules consume test reports produced by the execution phase and render user-visible summaries and artifacts.

Important Concepts and Design Patterns

Phased Test Execution: The split into setup, call, and teardown phases encapsulates test lifecycle management clearly, allowing hooks and reporting to be phase-aware.
Stack-Based SetupState: A stack tracks active nodes (session, modules, test items) during setup and teardown, ensuring proper nesting and finalizer execution.
CallInfo Wrapper: Wraps outcomes of calls with timing, exception info, and result encapsulation, enabling uniform handling of successes and failures.
TestReport Abstraction: A rich data structure capturing all relevant test outcome details, used throughout reporting.
Hook-Based Extensibility: Uses pytest's plugin hook system to provide customizable execution and reporting behavior.
Incremental XML Construction: JUnit XML reporting builds the XML tree incrementally as test reports arrive, accommodating parallel and interleaved test execution.
Terminal Progress Tracking: Maintains sets of reported node IDs to provide dynamic progress information and concise output.

Visual Diagram: Test Execution and Reporting Flow

sequenceDiagram
    participant Session
    participant SetupState
    participant Item
    participant Hooks as Pytest Hooks
    participant TerminalReporter
    participant JUnitXMLReporter

    Session->>SetupState: Initialize setup stack
    loop For each Test Item
        SetupState->>SetupState: Setup chain (session, module, item)
        Hooks->>Item: pytest_runtest_setup(item)
        Hooks->>Item: pytest_runtest_call(item)
        Hooks->>Item: pytest_runtest_teardown(item)

        Item->>Hooks: Generate CallInfo for each phase
        Hooks->>Hooks: pytest_runtest_makereport(call info)
        Hooks->>TerminalReporter: pytest_runtest_logreport(report)
        Hooks->>JUnitXMLReporter: pytest_runtest_logreport(report)

        SetupState->>SetupState: Teardown chain up to next item
    end
    Session->>TerminalReporter: pytest_sessionstart()
    Session->>TerminalReporter: pytest_sessionfinish()
    Session->>JUnitXMLReporter: pytest_sessionstart()
    Session->>JUnitXMLReporter: pytest_sessionfinish()

This documentation captures the core responsibilities and mechanisms of the Test Execution and Reporting module, illustrating its critical role in managing test lifecycles and delivering actionable feedback to users and CI systems.