test_error_diffs.py

Overview

`test_error_diffs.py` is a test utility file designed to verify the correctness and clarity of error diffs produced by pytest assertions, especially those involving complex data structures. It contains a comprehensive set of test cases that assert expected output diffs for various comparison scenarios, such as lists, tuples, sets, dictionaries, strings, and custom classes (dataclasses and attrs classes).

This file ensures that pytest's error reporting for assertion failures involving equality checks uses the correct `+` (addition) and `-` (deletion) signs, and that the diffs are informative and accurately highlight the differences between actual and expected values. It also covers corner cases like "not in" string assertions and attribute-level differences in classes.

The tests are parameterized, making it easy to add or modify cases, and they rely on pytest's own testing infrastructure (`Pytester`) to run and validate the output.

Detailed Explanation

Global Variable: TESTCASES

• Type: list of pytest.param objects

• Purpose: Holds multiple test scenarios as parameters for the main test function. Each test case consists of:

  • code (str): A snippet of Python test code that will be run and deliberately fail an assertion.

  • expected (str): The expected output from pytest when running the failing test, showing the error diff with correct +/- usage.

  • id (str): A descriptive identifier for the test case.

**Usage:**

• Used as input to pytest.mark.parametrize to dynamically generate multiple test runs.

**Example snippet from a `pytest.param`:**

pytest.param(
    """
    def test_this():
        result =   [1, 4, 3]
        expected = [1, 2, 3]
        assert result == expected
    """,
    """
    >       assert result == expected
    E       assert [1, 4, 3] == [1, 2, 3]
    E         At index 1 diff: 4 != 2
    E         Full diff:
    E           [
    E               1,
    E         -     2,
    E         ?     ^
    E         +     4,
    E         ?     ^
    E               3,
    E           ]
    """,
    id="Compare lists, one item differs",
)

Function: test_error_diff

@pytest.mark.parametrize("code, expected", TESTCASES)
def test_error_diff(code: str, expected: str, pytester: Pytester) -> None:

**Purpose:** Runs the provided test code snippet and verifies that pytest's output matches the expected error diff exactly in terms of content and formatting.

**Parameters:**

• code (str): Python code string containing a test function expected to fail.

• expected (str): Expected output string from pytest showing the error diff.

• pytester (Pytester): pytest's testing fixture that provides utilities to create test files and run pytest programmatically.

**Return Value:** `None` — The function asserts internally and will raise an error if the actual output does not match expected.

**Implementation Details:**

1. Normalize Expected Output:
   The expected string is split into lines, and each line is stripped of leading whitespace (lstrip()). This normalization allows for flexible indentation in the expected string while matching actual output lines.

2. Create Test File:
   Uses pytester.makepyfile(code) to create a temporary Python test file containing the failing test.

3. Run Pytest:
   Executes pytest on the generated test file with verbose output (-vv) to capture detailed assertion diffs.

4. Check Output:
   Matches the stdout lines against expected_lines using fnmatch_lines, ensuring the error message and diff display are as expected.

5. Check Return Code:
   Asserts that the pytest exit code is 1, indicating a failure (as expected).

**Usage Example:**

This function is automatically run by pytest for each case in `TESTCASES`. To add a new test scenario, add a new `pytest.param` entry to `TESTCASES`.

Important Implementation Details

• The file leverages pytester — a pytest plugin fixture designed for testing pytest plugins or pytest itself.

• The focus is on validating the textual output of assertion failures, not the logic of the tested code snippets themselves.

• The error diffs include detailed context lines with + and - signs that highlight the exact differences between actual and expected values, including nested structures and attribute-level differences.

• The test cases cover a wide variety of data types and scenarios to ensure robust and consistent error diff formatting.

Interaction with Other Parts of the System

• This file is part of the pytest test suite or pytest plugin ecosystem.

• It specifically tests pytest’s own assertion rewriting and error reporting logic.

• It depends on the _pytest.pytester.Pytester fixture which is part of pytest’s internal testing utilities.

• The output format tested here corresponds to the enhanced assertion introspection feature of pytest, providing rich diffs on test failures.

• It indirectly helps maintain quality and usability of pytest’s error messages, which improves developer experience when debugging test failures across the pytest ecosystem.

Visual Diagram: Flowchart of Main Functions and Relationships

flowchart TD
    A[TESTCASES: list of pytest.param] --> B[test_error_diff]
    B --> C[pytester.makepyfile(code)]
    B --> D[pytester.runpytest(p, "-vv")]
    D --> E[Capture stdout]
    E --> F[Match stdout with expected output lines]
    B --> G[Assert pytest exit code == 1]

Summary

`test_error_diffs.py` is a highly focused testing utility that validates pytest's enhanced assertion error diffs, ensuring that output is accurate, clear, and consistent for a variety of data structures and assertion scenarios. It uses parameterized tests and pytest’s own testing tools to automate regression testing of error message formatting, crucial for developer debugging and test clarity.

Appendix: Key Concepts Covered by Test Cases

• List equality differences (single differing item, extra/missing item)

• Tuple equality differences

• Set membership differences

• Dictionary key and value differences

• String differences and substring assertions (not in)

• Comparison of dataclasses and attrs classes with attribute-level diffs

This comprehensive coverage ensures confidence that pytest’s error diff tool behaves correctly across typical Python types and use cases.