Test Rerun and Stepwise Execution

This module provides features that enhance the test execution workflow by enabling reruns of failed tests first and supporting stepwise test execution. These capabilities improve developer efficiency by focusing testing on problematic tests and allowing a controlled, incremental test run that stops on failures and resumes from the last failure.

Core Concepts and Purpose

Rerun Failed Tests First: Supports command-line options to run only tests that failed in the last run (--lf or --last-failed), or run all tests but prioritize previously failed tests first (--ff or --failed-first). This reduces the time spent running passing tests when iterating on fixes.
Run New Tests First: Supports running tests from new files first (--nf or --new-first), enhancing feedback when adding new tests to a codebase.
Stepwise Execution: Provides a mechanism (--stepwise or --sw) to execute tests sequentially, stopping at the first failure, and resuming from that failure in the next run. Additional options --stepwise-skip and --stepwise-reset control skipping the first failure or resetting the stepwise state.
Caching Test Outcomes: Uses caching to persist information about failed tests and test order across test sessions, enabling these features to work consistently between runs.

These features address problems such as long test suite runtimes when many tests pass, difficulty in isolating failing tests quickly, and inefficient reruns during test-driven development.

How the Module Works

1. Caching Test Results for Rerun and Stepwise

The system uses the pytest cache directory to store data about failed tests and stepwise execution state. The cache is accessed via a `Cache` object (defined in `src/_pytest/cacheprovider.py`) which provides a persistent JSON-backed storage.

The last failed tests are stored at the cache key "cache/lastfailed".
The stepwise execution state is stored under "cache/stepwise".

The `Cache` class abstracts reading and writing these JSON-serializable values, automatically managing cache directories and files.

2. Plugins Implementing Rerun and Stepwise Behavior

The rerun and stepwise features are implemented as pytest plugins registered during configuration (`pytest_configure` hook):

LFPlugin (src/_pytest/cacheprovider.py): Manages --lf (last-failed) and --ff (failed-first) options.
NFPlugin (src/_pytest/cacheprovider.py): Manages --nf (new-first) option.
StepwisePlugin (src/_pytest/stepwise.py): Manages --sw (stepwise) and related options.

LFPlugin: Last-Failed and Failed-First

On test collection, LFPlugin inspects the cached last-failed test nodeids.
If --lf is active, it filters the test items to only those that failed previously.
If --ff is active, it reorders the test items to run failed tests first, followed by others.
After test runs, it updates the cache with tests that failed or passed to keep the last-failed set current.
The plugin also handles deselection of tests outside the last-failed set and reports statuses at collection finish.

NFPlugin: New-First

On test collection, NFPlugin compares collected test nodes against cached nodeids to identify new tests.
It reorders tests so that new tests run before existing ones, sorted by file modification time.
Updates the cache with all known nodeids after test runs for future reference.

StepwisePlugin: Stepwise Execution

Registers options: --sw enables stepwise mode; --sw-skip skips the first failure but stops on the next; --sw-reset resets the stepwise state.
On collection, the plugin reads cached stepwise state, including the last failed test nodeid and the previous test count.
If a last failed test exists and the test count matches, it skips tests before that failure, allowing continuation from the last failure.
During test execution, if a failure occurs, the plugin records the failed test nodeid and requests pytest to stop further testing (session.shouldstop).
If the failure is skipped via --sw-skip, it ignores the first failure and continues.
Updates cache with the current stepwise state at session finish.

Key Functional Workflows

Test Collection Modification

Last-Failed Plugin reorders or filters test items based on cached failures.
New-First Plugin reorders tests prioritizing new files.
Stepwise Plugin skips tests prior to the last failure if applicable.

def pytest_collection_modifyitems(self, config: Config, items: list[nodes.Item]) -> None:
    # StepwisePlugin example snippet:
    if self.cached_info.last_failed:
        # Find index of last failed test
        failed_index = next(
            (i for i, item in enumerate(items) if item.nodeid == self.cached_info.last_failed),
            None,
        )
        if failed_index is not None:
            # Skip tests before the last failure
            deselected = items[:failed_index]
            del items[:failed_index]
            config.hook.pytest_deselected(items=deselected)

Test Execution and Failure Handling

When a test fails, StepwisePlugin stores the nodeid of the failure and stops the session.
If --stepwise-skip is used, it skips the first failure and continues.

def pytest_runtest_logreport(self, report: TestReport) -> None:
    if report.failed:
        if self.skip:
            if report.nodeid == self.cached_info.last_failed:
                self.cached_info.last_failed = None
            self.skip = False
        else:
            self.cached_info.last_failed = report.nodeid
            self.session.shouldstop = "Test failed, continuing from this test next run."

Cache Update

At session finish, the plugins write updated failure or stepwise information back to the cache for use in subsequent runs.

Interactions with Other System Components

Cache System (src/_pytest/cacheprovider.py): Provides persistent JSON storage used by rerun and stepwise plugins to save and retrieve test failure data and stepwise state.
Test Collection (src/_pytest/nodes.py): The rerun and stepwise plugins modify the list of collected test items to control which tests are run or skipped.
Test Execution (src/_pytest/runner.py): Stepwise plugin interacts by signaling pytest to stop the test session upon failure.
Configuration and CLI (src/_pytest/config/__init__.py): Plugins register command-line options and are registered based on these options.

Important Concepts and Design Patterns

Persistent Caching: Relies heavily on caching test results and metadata between runs for incremental test execution.
Session Control via Hooks: Uses pytest hook system (pytest_collection_modifyitems, pytest_runtest_logreport, pytest_sessionfinish) to intervene at key lifecycle points.
Selective Test Deselection: Implements logic to deselect tests before the last failure for stepwise continuation.
Graceful Recovery: Handles changes in test suite size by invalidating stale cache to avoid confusing skipping behavior.
User Control via CLI Options: Offers multiple command-line flags with interdependencies, allowing flexible control over test rerun behaviors.

Mermaid Diagram: Sequence of Stepwise Test Execution

sequenceDiagram
    participant User
    participant Pytest CLI
    participant StepwisePlugin
    participant Cache
    participant TestCollection
    participant TestRunner

    User->>Pytest CLI: Runs pytest with --stepwise
    Pytest CLI->>StepwisePlugin: Register plugin and read options
    StepwisePlugin->>Cache: Load last failed test info
    StepwisePlugin->>TestCollection: Modify collected tests
    TestCollection-->>StepwisePlugin: Return filtered/skipped tests
    StepwisePlugin->>TestRunner: Run tests starting from last failure
    TestRunner->>StepwisePlugin: Report test results
    StepwisePlugin->>Cache: Update last failed test info if failure occurs
    StepwisePlugin->>TestRunner: Request stop on failure
    TestRunner-->>User: Test run stops on failure
    User->>Pytest CLI: Runs pytest again with --stepwise
    StepwisePlugin->>Cache: Load last failed test info
    StepwisePlugin->>TestCollection: Skip tests before last failure
    TestRunner->>StepwisePlugin: Run remaining tests
    StepwisePlugin->>Cache: Clear last failed info if tests pass

This documentation explains the rationale, mechanisms, and system integration for pytest’s test rerun and stepwise execution features, illuminating how these capabilities optimize test workflows by leveraging caching, selective test execution, and session control.