stepwise.py

Overview

The [stepwise.py](/projects/286/67474) module implements a pytest plugin that provides **stepwise test execution** functionality. This feature enables running tests sequentially, stopping immediately on the first failing test, and then resuming from that failure in subsequent test runs. It is designed to improve developer productivity by focusing test runs on the problematic parts of the test suite, reducing time spent rerunning already passing tests.

**Key features:**

• Stop on first failure: The test session halts as soon as a test fails.

• Resume from last failure: On the next run, tests before the last failed test are skipped.

• Configurable behaviors: Options to skip the first failure (--sw-skip) or reset the stepwise state (--sw-reset).

• Persistent caching: Caches failure info and test counts between sessions to enable incremental test execution.

• Integration with pytest’s lifecycle: Hooks into test collection, reporting, and session finish to control flow and update cache.

Classes and Functions

Constants

pytest_addoption(parser: Parser) -> None

**Purpose:** Registers command-line options for stepwise execution.

**Options added:**

• --sw / --stepwise: Enable stepwise mode (stop on failure and resume).

• --sw-skip / --stepwise-skip: Skip the first failing test once, then stop on the next failure. Enables --stepwise implicitly.

• --sw-reset / --stepwise-reset: Reset stepwise state and cache. Enables --stepwise implicitly.

**Parameters:**

• parser: pytest argument parser.

**Example usage:**

pytest --sw
pytest --stepwise-skip
pytest --stepwise-reset

pytest_configure(config: Config) -> None

**Purpose:** Configures the plugin based on command-line options.

• If --stepwise-skip or --stepwise-reset is enabled, implicitly enables --stepwise.

• Registers the StepwisePlugin if stepwise is enabled.

**Parameters:**

• config: pytest configuration object.

pytest_sessionfinish(session: Session) -> None

**Purpose:** Session finish hook stub to avoid cache updates in xdist workers.

• If this is a worker process (workerinput attribute present), the cache is not updated to avoid race conditions.

**Parameters:**

• session: pytest session object.

@dataclasses.dataclass StepwiseCacheInfo

**Purpose:** Data structure representing the cached state of stepwise execution.

**Attributes:**

**Properties:**

• last_cache_date (datetime): Parses last_cache_date_str into a datetime object.

**Class Methods:**

• empty() -> StepwiseCacheInfo: Creates an empty cache info object with no failures and current timestamp.

**Instance Methods:**

• update_date_to_now() -> None: Updates the cache date to the current time.

class StepwisePlugin

**Purpose:** Primary plugin class implementing stepwise test execution.

Initialization: __init__(self, config: Config)

• Stores pytest config, cache, and options.

• Loads cached stepwise information (last_failed, last_test_count, last_cache_date).

• Tracks whether to skip first failure (--sw-skip) or reset state (--sw-reset).

Methods:

_load_cached_info(self) -> StepwiseCacheInfo

• Attempts to read stepwise state from the cache.

• If cache is missing or corrupted, returns an empty cache info.

• Logs any cache reading errors to self.report_status.

pytest_sessionstart(self, session: Session) -> None

• Stores the current test session object for later control (e.g., stopping the session).

pytest_collection_modifyitems(self, config: Config, items: list[nodes.Item]) -> None

• Modifies collected test items to implement the stepwise skipping behavior.

**Logic:**

1. Updates last_test_count in cache info with current test count.

2. If --stepwise-reset is used:

   • Resets last_failed to None, skipping no tests.

3. If no cached last failed test:

   • Does not skip any tests.

4. If the test count has changed since last run:

   • Invalidates cache (resets last_failed), skipping no tests.

5. Otherwise:

   • Finds the index of the last failed test in the collected items.

   • If found, skips all tests before that index.

   • If not found, skips no tests.

6. Reports status messages for these decisions.

pytest_runtest_logreport(self, report: TestReport) -> None

• Called after each test phase report.

• If a test fails:

  • If --stepwise-skip is active:

    • If the failed test matches the cached last failure, clears cached failure.

    • Disables skipping for subsequent failures.

  • Else:

    • Records current failed test nodeid as last_failed.

    • Signals pytest to stop the session (session.shouldstop).

• If a test passes during the call phase:

  • Removes the test from cached failed test if it matches.

pytest_report_collectionfinish(self) -> list[str] | None

• Returns status messages for the pytest collection finish phase if verbosity is enabled.

• Messages describe the stepwise plugin’s decisions (e.g., skipping tests).

pytest_sessionfinish(self) -> None

• Updates the cache with the latest StepwiseCacheInfo at session end.

• Does not update cache if running in a xdist worker.

Important Implementation Details

• Cache Management:
  The plugin uses pytest’s cache system to persist stepwise test state between runs. This cache stores:

  • The node ID of the last failed test.

  • The number of tests collected in the last run.

  • A timestamp of when the cache was last updated.

• Test Skipping Logic:
  The plugin modifies the collected tests list to remove all tests before the last failed test, effectively skipping already passing tests.

• Failure Handling and Session Stop:
  On test failure, the plugin records the failure and requests pytest to stop running further tests, thus implementing the "stop on first failure" behavior.

• Skip First Failure Option:
  The --stepwise-skip option allows the first failure to be ignored once, continuing execution until the second failure.

• Reset Option:
  The --stepwise-reset option clears cached state to restart the stepwise behavior fresh.

• Safety Checks:
  The plugin invalidates the cache if the test count has changed since last run, avoiding confusing skipping behavior due to stale cache.

• Concurrency Handling:
  The plugin avoids cache writes in xdist worker nodes to prevent race conditions.

Interactions with the System

• pytest CLI and Configuration:
  Registers CLI options and registers itself as a plugin during pytest configuration.

• pytest Cache System:
  Reads and writes stepwise execution state persistently via pytest's cache provider.

• Test Collection:
  Modifies the list of test items during collection to skip tests that passed previously.

• Test Execution and Reporting:
  Monitors test results to detect failures and stops the test session accordingly.

• Parallel Execution (xdist):
  Avoids updating cache from worker nodes to prevent race conditions.

Usage Example

# Run tests stepwise, stopping on first failure and resuming from it next time
pytest --sw

# Skip the first failure once, then stop on the next failure
pytest --sw-skip

# Reset stepwise state and start over
pytest --sw-reset

Visual Diagram: Class Diagram of StepwisePlugin and StepwiseCacheInfo

classDiagram
    class StepwiseCacheInfo {
        +str|None last_failed
        +int|None last_test_count
        +str last_cache_date_str
        +datetime last_cache_date
        +classmethod empty() StepwiseCacheInfo
        +update_date_to_now() void
    }

    class StepwisePlugin {
        -Config config
        -Session|None session
        -list~str~ report_status
        -Cache cache
        -bool skip
        -bool reset
        -StepwiseCacheInfo cached_info
        +__init__(config: Config)
        +_load_cached_info() StepwiseCacheInfo
        +pytest_sessionstart(session: Session) void
        +pytest_collection_modifyitems(config: Config, items: list~nodes.Item~) void
        +pytest_runtest_logreport(report: TestReport) void
        +pytest_report_collectionfinish() list~str~|None
        +pytest_sessionfinish() void
    }

    StepwisePlugin --> StepwiseCacheInfo : uses

Summary

[stepwise.py](/projects/286/67474) is a pytest plugin module enabling stepwise test execution by stopping on the first failure and resuming from that failure on the next run. It leverages pytest's caching system and lifecycle hooks to modify test collection and control session flow, providing developers with a time-saving incremental testing workflow. The plugin is configurable, handles edge cases such as test suite changes and parallel execution, and integrates cleanly with pytest’s extensible architecture.