Last-Failed and New-First Caching

Purpose

This subtopic addresses the need to optimize test execution by prioritizing tests based on their previous run outcomes or novelty. Specifically, it enables rerunning only tests that failed in the last session (`--lf` / `--last-failed`) or running tests from new or recently modified files first (`--nf` / `--new-first`). This targeted rerun approach saves time and resources by avoiding redundant execution of passing tests and focusing on areas likely to contain regressions or new code changes.

Functionality

Core Features

• Persistent Cache Storage:
  Uses a JSON-backed cache directory (.pytest_cache by default) to store metadata about test outcomes (lastfailed) and seen test node IDs (nodeids) across test sessions.

• Last-Failed (--lf) and Failed-First (--ff) Modes:

  • --lf runs only the tests that failed previously.

  • --ff runs all tests but schedules previously failed tests first, potentially reordering tests.
    This is implemented by maintaining a cache of failed test node IDs and filtering or reordering the collected tests accordingly.

• New-First (--nf) Mode:
  Runs tests from new or modified files first by tracking cached node IDs and comparing them with current ones. New tests (not seen before) are prioritized and sorted by file modification time to run the most recent changes first.

• Cache Management and Cleanup:
  Supports options to clear the cache at the start of a session and to display cache contents without running tests.

Key Workflows and Methods

• Caching Failures:
  The plugin hooks into test reporting phases (pytest_runtest_logreport and pytest_collectreport) to update the lastfailed dictionary, adding failed tests and removing passed/skipped ones.

• Test Collection Modification:
  During test collection (pytest_collection_modifyitems), the plugin intercepts the list of collected tests and either filters (for --lf) or reorders (for --ff and --nf) the list based on cached failure or newness information.

• Path Tracking for Last-Failed:
  To optimize collection, it identifies and prioritizes test files and directories containing previously failed tests, skipping files without failures during collection when applicable.

• Cache Directory Structure:
  The cache stores values (v/) and directories (d/) separately within the .pytest_cache directory, managing creation and cleanup transparently.

Illustrative Code Snippets

• Filtering collected tests for last-failed only:

  if self.config.getoption("lf"):
      items[:] = previously_failed
      config.hook.pytest_deselected(items=previously_passed)
  

• Updating lastfailed cache on test reports:

  def pytest_runtest_logreport(self, report: TestReport) -> None:
      if (report.when == "call" and report.passed) or report.skipped:
          self.lastfailed.pop(report.nodeid, None)
      elif report.failed:
          self.lastfailed[report.nodeid] = True
  

• Sorting new tests first by file modification time:

  def _get_increasing_order(self, items: Iterable[nodes.Item]) -> list[nodes.Item]:
      return sorted(items, key=lambda item: item.path.stat().st_mtime, reverse=True)
  

Integration

This caching mechanism is tightly integrated with the pytest test collection and execution phases:

• It hooks into test collection to reorder or filter test items before execution, thus influencing the test run order.

• It interacts with the test execution and reporting subsystem to update cache state based on test outcomes.

• It complements the Test Rerun and Stepwise Execution parent topic by providing the persistent data and logic needed to identify tests for rerun or prioritization.

• It leverages the Cache Fixture to expose persistent storage for plugins or tests to store arbitrary state.

• Interaction with other subtopics like Plugin System and Hooks is essential, as the caching behavior is implemented as plugins registering hooks such as pytest_collection_modifyitems, pytest_runtest_logreport, and pytest_collectreport.

• The cache directory management ensures that other pytest features and external plugins can safely store and retrieve persistent data without interfering with test execution.

This subtopic introduces the concrete caching layer and logic for "last-failed" and "new-first" test prioritization, which is not covered at a high level in the parent topic, providing the essential mechanism for persistent state and targeted test reruns.

Diagram

flowchart TD
    A[Start Test Run]
    B[Test Collection]
    C{Caching Mode?}
    C -->|--lf (Last-Failed)| D[Filter collected tests to last failed only]
    C -->|--ff (Failed-First)| E[Reorder tests: failed first, then others]
    C -->|--nf (New-First)| F[Reorder tests: new files first]
    C -->|None| G[Run tests as collected]
    D --> H[Run filtered tests]
    E --> H
    F --> H
    G --> H
    H --> I[Test Execution & Reporting]
    I --> J[Update cache with failures and passes]
    J --> K[Save cache to disk]
    K --> L[End Test Run]

This flowchart visualizes how the last-failed and new-first caching influences test collection and execution, and how test outcomes update the cache for future runs.