test_stepwise.py

Overview

The [test_stepwise.py](/projects/286/67334) file contains a comprehensive suite of automated tests for the **stepwise testing feature** in the pytest testing framework. The "stepwise" feature is designed to improve debugging efficiency by stopping the test run on the first failure and continuing from that point in subsequent runs, avoiding re-running previously passed tests unnecessarily.

This file verifies the correctness, robustness, and user experience of the stepwise feature, including interactions with pytest command-line options, cache management, error handling, and compatibility with other pytest features such as `xfail` and parallel test execution (`xdist`).

Detailed Explanation of Contents

The file is structured using `pytest` fixtures and test functions, with some internal utility functions. It primarily uses the `Pytester` testing helper from pytest to create temporary test environments and run pytest subprocesses.

Fixtures

Fixtures prepare reusable test environments for different scenarios.

stepwise_pytester(pytester: Pytester) -> Pytester

• Purpose: Sets up a test environment with a simple test suite that includes options to fail or pass certain tests conditionally. This simulates tests with failures to examine stepwise behavior.

• Key actions:

  • Adds --fail and --fail-last CLI options to control failure injection.

  • Creates two test files (test_a.py and test_b.py) with multiple tests.

  • Configures a local cache directory .cache to avoid interference with tox cache.

• Returns: The configured Pytester instance.

error_pytester(pytester: Pytester) -> Pytester

• Purpose: Sets up a test environment with a test file containing an error (missing fixture) and a subsequent passing test to verify stepwise behavior on errors.

• Returns: The configured Pytester instance.

broken_pytester(pytester: Pytester) -> Pytester

• Purpose: Sets up a test environment with one well-formed and one broken test file to test behavior on collection errors.

• Returns: The configured Pytester instance.

Utility Functions

_strip_resource_warnings(lines: Sequence[str]) -> Sequence[str]

• Purpose: Filters out unreliable ResourceWarning and similar warnings from output lines to avoid test flakiness caused by non-deterministic warnings.

• Parameters:

  • lines: Sequence of lines (strings) to filter.

• Returns: Filtered sequence without resource warnings.

• Usage: Called after test runs to sanitize stderr output before assertions.

Test Functions

Each test function uses the `Pytester` fixture to run pytest subprocesses with different options and asserts expected behavior.

test_run_without_stepwise(stepwise_pytester: Pytester)

• Purpose: Verifies that running tests without the stepwise option executes all tests and shows correct pass/fail results.

• Key checks: Confirms tests run in order and fail/pass as expected based on the --fail flag.

test_stepwise_output_summary(pytester: Pytester)

• Purpose: Tests the summary output of the stepwise execution, including skipping previously passed tests on subsequent runs.

• Key checks:

  • First run reports no previous failures.

  • Second run skips already passed tests and reports correct deselection count.

test_fail_and_continue_with_stepwise(stepwise_pytester: Pytester)

• Purpose: Tests the core stepwise behavior: stop on first failure, then continue from the failure on next run.

• Key checks:

  • First run stops after the failing test.

  • Second run reruns the failing test and continues with subsequent ones.

  • Verifies no resource warnings are reported.

test_run_with_skip_option(stepwise_pytester: Pytester, stepwise_skip: str)

• Purpose: Checks that the --stepwise-skip (and its alias --sw-skip) options work as expected to skip previously passed tests but allow continuation after failures.

• Key checks: Ensures the first failure is ignored (skipped), and the second failure stops the test run.

test_fail_on_errors(error_pytester: Pytester)

• Purpose: Verifies that errors during test execution (e.g., missing fixtures) cause the test run to stop as expected in stepwise mode.

test_change_testfile(stepwise_pytester: Pytester)

• Purpose: Ensures that changing the test file between runs resets the stepwise continuation and runs tests from the start for the new file.

test_stop_on_collection_errors(broken_pytester: Pytester, broken_first: bool)

• Purpose: Tests that collection errors (e.g., syntax errors) cause the test run to stop immediately, regardless of the order of files.

• Parameters:

  • broken_first: Boolean to test both ordering scenarios.

test_xfail_handling(pytester: Pytester, monkeypatch: MonkeyPatch)

• Purpose: Tests that:

  • Normal xfail tests are ignored and don't interrupt stepwise runs.

  • Strict xfail failures interrupt the session as failures.

• Implementation: Parametrizes test content with xfail(strict=True/False) and asserts output.

test_stepwise_skip_is_independent(pytester: Pytester)

• Purpose: Verifies that --stepwise-skip option works independently and does not stop on first failure but continues while recording failures.

test_sw_skip_help(pytester: Pytester)

• Purpose: Checks that the pytest help message mentions that --stepwise-skip implicitly enables --stepwise.

test_stepwise_xdist_dont_store_lastfailed(pytester: Pytester)

• Purpose: Tests that when running under xdist (parallel), stepwise does not store lastfail cache and interrupts as expected.

test_disabled_stepwise_xdist_dont_clear_cache(pytester: Pytester)

• Purpose: Ensures that if stepwise is disabled in xdist mode, the cache is not cleared.

test_do_not_reset_cache_if_disabled(pytester: Pytester)

• Purpose: Verifies that running pytest without --stepwise does not clear the stepwise cache, supporting workflows where developers run isolated tests manually between stepwise runs.

test_reset(pytester: Pytester)

• Purpose: Tests the --stepwise-reset option which clears state and causes all tests to run again from the start.

test_change_test_count(pytester: Pytester)

• Purpose: Validates that changes in the number of tests invalidate the stepwise cache, causing tests to run fully again.

test_cache_error(pytester: Pytester)

• Purpose: Tests robustness against corrupted cache files: the cache is discarded gracefully, and tests run as if no cache existed.

Important Implementation Details and Algorithms

• The tests simulate real-world usage of the stepwise feature by controlling test pass/fail behavior using CLI flags (--fail, --fail-last).

• The caching mechanism tested here stores the last failed test to resume from that point on subsequent runs.

• Tests ensure correct cache invalidation on changes (e.g., test count changes or file changes).

• ResourceWarning filtering prevents flaky test failures due to warnings not relevant to stepwise feature correctness.

• Parametrized tests and monkeypatching are used to simulate different pytest configurations and test outcomes.

• Interaction with pytest's Cache and STEPWISE_CACHE_DIR mechanisms is verified to ensure proper cache file management.

• The stepwise feature is tested for compatibility with xfail tests and pytest's parallel execution plugin (xdist).

Interaction with Other Parts of the System

• This file tests the pytest stepwise plugin/feature which interacts with:

  • The pytest CLI options and test collection/execution flow.

  • The pytest caching subsystem (Cache).

  • pytest's xdist plugin for distributed test execution.

  • pytest's test result reporting and skipping mechanisms.

• It uses the pytest internal testing helpers (Pytester) to create isolated test environments and simulate pytest runs.

• The stepwise cache directory (STEPWISE_CACHE_DIR) and cache files are manipulated and verified.

• The tests ensure that stepwise integrates seamlessly with other pytest features such as xfail, monkeypatch, and command-line options.

Usage Examples

Below are simplified usage scenarios demonstrated in the tests:

# Run tests normally without stepwise
pytest -v --fail

# Run tests with stepwise, stopping at first failure
pytest -v --stepwise --fail

# Fix failing test, rerun with stepwise to continue from failure
pytest -v --stepwise

# Skip previously passed tests but continue after failures
pytest -v --stepwise --stepwise-skip --fail --fail-last

# Reset stepwise state to rerun all tests
pytest -v --stepwise-reset

Mermaid Diagram

classDiagram
    class StepwiseTestSuite {
        +stepwise_pytester(pytester: Pytester) Pytester
        +error_pytester(pytester: Pytester) Pytester
        +broken_pytester(pytester: Pytester) Pytester
        +_strip_resource_warnings(lines: Sequence[str]) Sequence[str]
        +test_run_without_stepwise(stepwise_pytester: Pytester)
        +test_stepwise_output_summary(pytester: Pytester)
        +test_fail_and_continue_with_stepwise(stepwise_pytester: Pytester)
        +test_run_with_skip_option(stepwise_pytester: Pytester, stepwise_skip: str)
        +test_fail_on_errors(error_pytester: Pytester)
        +test_change_testfile(stepwise_pytester: Pytester)
        +test_stop_on_collection_errors(broken_pytester: Pytester, broken_first: bool)
        +test_xfail_handling(pytester: Pytester, monkeypatch: MonkeyPatch)
        +test_stepwise_skip_is_independent(pytester: Pytester)
        +test_sw_skip_help(pytester: Pytester)
        +test_stepwise_xdist_dont_store_lastfailed(pytester: Pytester)
        +test_disabled_stepwise_xdist_dont_clear_cache(pytester: Pytester)
        +test_do_not_reset_cache_if_disabled(pytester: Pytester)
        +test_reset(pytester: Pytester)
        +test_change_test_count(pytester: Pytester)
        +test_cache_error(pytester: Pytester)
    }

Summary

[test_stepwise.py](/projects/286/67334) is a critical quality assurance file for validating the stepwise testing feature in pytest. It ensures that:

• Stepwise correctly stops on failures and resumes appropriately.

• Cache management is robust, efficient, and error-tolerant.

• Compatibility with pytest's other features and plugins is maintained.

• User-facing CLI options behave as documented.

• The internal workflows of stepwise are reliable across various test scenarios.

This thorough test coverage helps maintain a smooth and predictable developer experience when using pytest's stepwise test execution.