Test Skipping and Expected Failures

This module provides the functionality to selectively skip tests or mark them as expected to fail (xfail). It enables conditional test skipping and xfail marking based on runtime conditions, facilitating flexible test suite management and improved test reporting.

Core Concepts and Purpose

Test Skipping: Allows tests to be skipped entirely, either unconditionally or when certain conditions are met. Skipped tests do not run and are reported as skipped, often with a reason.
Expected Failures (xfail): Marks tests expected to fail under certain conditions. Tests marked xfail can be run or not run, and their outcome is reported differently depending on whether they fail as expected, unexpectedly pass, or fail with unexpected exceptions.
These features help maintain test suites by allowing known problematic tests to be excluded or specially marked without causing overall test suite failure.

How the Module Works

1. Configuration and Command-Line Options

The module adds support for the --runxfail CLI option which changes the behavior of xfail tests to be reported as normal tests rather than as expected failures.
It also registers the xfail_strict ini option, which controls whether unexpected passes of xfail tests should be treated as failures by default.

def pytest_addoption(parser: Parser) -> None:
    group.addoption(
        "--runxfail",
        action="store_true",
        dest="runxfail",
        default=False,
        help="Report the results of xfail tests as if they were not marked",
    )
    parser.addini(
        "xfail_strict",
        "Default for the strict parameter of xfail markers when not given explicitly (default: False)",
        default=False,
        type="bool",
    )

2. Markers for Skipping and Xfail

The module defines and documents three markers available for tests:
- skip(reason=...): unconditionally skip the test with an optional reason.
- skipif(condition, reason=...): skip the test if the condition evaluates to True.
- xfail(condition, reason=..., run=..., raises=..., strict=...): mark the test as expected to fail under conditions.

These markers can be applied in test code, enabling declarative control over test execution.

3. Evaluation of Skip and Xfail Conditions

The module evaluates skip and xfail markers on test items to decide whether to skip or xfail them.
Conditions can be strings (evaluated with eval() in a controlled namespace) or booleans.
If a condition evaluates to True, the test is skipped or marked xfail accordingly.
For skipping, if any skipif condition is True or skip is present, the test is skipped.
For xfail, if any condition is True or no condition is given (unconditional), the test is marked xfail with parameters such as run (whether to execute the test), raises (expected exception types), and strict (whether unexpected passes are failures).

def evaluate_condition(item: Item, mark: Mark, condition: object) -> tuple[bool, str]:
    # Evaluates string or boolean conditions, returns (result, reason)
    ...
def evaluate_skip_marks(item: Item) -> Skip | None:
    # Returns Skip if skip/skipif condition is met
    ...
def evaluate_xfail_marks(item: Item) -> Xfail | None:
    # Returns Xfail if xfail condition is met
    ...

4. Hook Implementations for Test Lifecycle Integration

pytest_runtest_setup: Runs before test execution to evaluate skip/xfail marks.
- If the test should be skipped, it raises a skip exception that stops test execution.
- If the test is xfail and run is False, it triggers xfail without running the test.
- Stores xfail evaluation in the test item's stash for later use.
pytest_runtest_call: Wraps the actual test call.
- Checks again for xfail with run=False to skip running if needed.
- Allows dynamic addition or refresh of xfail marks.
pytest_runtest_makereport: Modifies test reports after test execution.
- Converts xfail exceptions into skipped outcomes with wasxfail set.
- Marks tests that were xfail but passed unexpectedly as "xpassed," possibly failing the test if strict is enabled.
- Handles cases where xfail expects specific exceptions and treats other exceptions as failures.

@hookimpl(tryfirst=True)
def pytest_runtest_setup(item: Item) -> None:
    skipped = evaluate_skip_marks(item)
    if skipped:
        raise skip.Exception(skipped.reason, _use_item_location=True)

    item.stash[xfailed_key] = xfailed = evaluate_xfail_marks(item)
    if xfailed and not item.config.option.runxfail and not xfailed.run:
        xfail("[NOTRUN] " + xfailed.reason)

@hookimpl(wrapper=True)
def pytest_runtest_call(item: Item) -> Generator[None]:
    xfailed = item.stash.get(xfailed_key, None)
    if xfailed is None:
        item.stash[xfailed_key] = xfailed = evaluate_xfail_marks(item)

    if xfailed and not item.config.option.runxfail and not xfailed.run:
        xfail("[NOTRUN] " + xfailed.reason)

    try:
        return (yield)
    finally:
        xfailed = item.stash.get(xfailed_key, None)
        if xfailed is None:
            item.stash[xfailed_key] = xfailed = evaluate_xfail_marks(item)

@hookimpl(wrapper=True)
def pytest_runtest_makereport(
    item: Item, call: CallInfo[None]
) -> Generator[None, TestReport, TestReport]:
    rep = yield
    xfailed = item.stash.get(xfailed_key, None)
    if item.config.option.runxfail:
        pass  # don't interfere
    elif call.excinfo and isinstance(call.excinfo.value, xfail.Exception):
        rep.wasxfail = call.excinfo.value.msg
        rep.outcome = "skipped"
    elif not rep.skipped and xfailed:
        if call.excinfo:
            raises = xfailed.raises
            if raises is None or (
                (
                    isinstance(raises, (type, tuple))
                    and isinstance(call.excinfo.value, raises)
                )
                or (
                    isinstance(raises, AbstractRaises)
                    and raises.matches(call.excinfo.value)
                )
            ):
                rep.outcome = "skipped"
                rep.wasxfail = xfailed.reason
            else:
                rep.outcome = "failed"
        elif call.when == "call":
            if xfailed.strict:
                rep.outcome = "failed"
                rep.longrepr = "[XPASS(strict)] " + xfailed.reason
            else:
                rep.outcome = "passed"
                rep.wasxfail = xfailed.reason
    return rep

5. Reporting Test Status for Skipped and Xfail Tests

The module adjusts terminal and report outputs to reflect xfail and xpass test statuses, using special codes and labels:
- xfailed tests are marked with "XFAIL" and x symbol.
- Unexpectedly passing xfail tests are marked with "XPASS" and X symbol.

def pytest_report_teststatus(report: BaseReport) -> tuple[str, str, str] | None:
    if hasattr(report, "wasxfail"):
        if report.skipped:
            return "xfailed", "x", "XFAIL"
        elif report.passed:
            return "xpassed", "X", "XPASS"
    return None

Module Interactions and Relationships

pytest.nodes.Item: Represents individual test items; skip and xfail marks are evaluated and stored here.
pytest.config.Config: Provides access to command line options and ini configuration like runxfail and xfail_strict.
pytest.outcomes: Contains exceptions used to signal skipping (skip.Exception) and expected failure (xfail.Exception).
pytest.reports.TestReport: The test report object modified to reflect xfail/skipped status.
pytest.mark.structures.Mark: Used to access and interpret skip and xfail markers attached to test items.
pytest.stash.StashKey: Used for storing xfail evaluation results in each test item's stash for reuse during test phases.

This module's functionality is integrated tightly with the pytest test lifecycle hooks (`pytest_runtest_setup`, `pytest_runtest_call`, `pytest_runtest_makereport`), enabling it to influence test execution, control skipping, and shape reporting.

Important Concepts and Design Patterns

Marker-Based Conditional Logic: The use of markers (skip, skipif, xfail) allows tests to declaratively express skipping and expected failure logic, which is evaluated dynamically at runtime.
Evaluation Context and Safety: Conditions can be strings evaluated in a controlled namespace including os, sys, and platform, allowing expressive conditions without exposing unsafe globals.
Stashing State: Evaluation results of xfail marks are cached per test item in a stash to avoid redundant computation and to support dynamic changes during test execution.
Exception-Based Flow Control: Raising special exceptions (skip.Exception, xfail.Exception) is used to interrupt or alter normal test execution flow.
Hook Wrapping and Prioritization: The use of @hookimpl(tryfirst=True) and @hookimpl(wrapper=True) ensures this module's hooks run early and wrap phases to control skipping and xfail behavior effectively.
Flexible Reporting: Test reports are dynamically altered to reflect skip and xfail results, including customized outcomes and labels, integrating with pytest’s reporting framework.

Mermaid Sequence Diagram: Skipping and Xfail Evaluation and Reporting Flow

sequenceDiagram
    participant Config as Config & CLI Options
    participant Item as Test Item
    participant Setup as pytest_runtest_setup
    participant Call as pytest_runtest_call
    participant Report as pytest_runtest_makereport
    participant Outcome as Test Outcome Reporting

    Config->>Item: Provide config (runxfail, xfail_strict)
    Item->>Setup: Evaluate skip marks
    alt skip condition true
        Setup->>Setup: Raise skip.Exception (skip test)
        Setup-->>Outcome: Mark test as skipped
    else
        Item->>Setup: Evaluate xfail marks
        alt xfail with run=False and not runxfail
            Setup->>Setup: Trigger xfail (no run)
            Setup-->>Outcome: Mark test as xfail skipped
        else
            Setup-->>Call: Allow test run
            Call->>Call: Yield to test function
            Call-->>Report: Capture call result or exception
        end
    end

    Report->>Item: Retrieve xfail info from stash
    alt call raised xfail.Exception
        Report->>Outcome: Mark test as xfail skipped
    else if call raised expected exception or no exception
        Report->>Outcome: Mark test as xfail skipped or passed
    else if call raised unexpected exception or unexpected pass with strict
        Report->>Outcome: Mark test failed or xpassed with failure
    end

This detailed explanation covers the purpose, functionality, integration, and key design elements of the Test Skipping and Expected Failures module, emphasizing its role in controlling test execution flow and improving test suite robustness and reporting.