Xfail Handling

Purpose

Xfail Handling addresses the need to mark tests as *expected failures* within the pytest framework. This feature allows users to designate tests that are known to fail due to existing bugs, unimplemented features, or external constraints without causing the entire test suite to be considered broken. It supports nuanced control over these tests, including whether they should be executed, how failures are interpreted, and strictness in reporting unexpected passes.

Functionality

The core functionality revolves around the evaluation and enforcement of the `xfail` marker on test items. This marker can be applied with conditions and options that control its behavior:

• Condition evaluation: The system evaluates conditions (boolean or string expressions) to determine if the xfail marker applies to a particular test.

• Run control: The run option specifies if the test should be executed despite being marked as xfail. If run=False, the test is skipped but reported as an expected failure.

• Strictness: The strict option enforces whether an unexpected pass (XPASS) is treated as a failure, which is useful to track when a previously failing test starts passing.

• Exception filtering: The raises option allows specifying expected exception types that justify the failure; failures due to other exceptions are reported as true failures.

Key workflows include:

1. Marker evaluation during setup: Before running a test, pytest evaluates xfail markers by checking their conditions to decide whether the test is expected to fail.

2. Test execution interception: If a test is marked as xfail with run=False, pytest skips execution but reports the outcome as xfail.

3. Outcome modification on reporting: After test execution, pytest adjusts the test report based on whether the test failed or passed and the xfail marker's parameters, annotating reports as XFAIL or XPASS accordingly.

4. Interaction with command line options: The --runxfail option overrides xfail behavior to report xfail tests as normal, allowing full execution and standard reporting.

Condition Evaluation

Conditions for xfail can be:

• Boolean expressions: Evaluated directly to True/False.

• String expressions: Evaluated in a controlled namespace with access to standard modules (os, sys, platform) and test globals.

If evaluation fails, pytest raises a failure with a descriptive message.

Example Snippet: Evaluating Xfail Marks

def evaluate_xfail_marks(item: Item) -> Xfail | None:
    for mark in item.iter_markers(name="xfail"):
        run = mark.kwargs.get("run", True)
        strict = mark.kwargs.get("strict", item.config.getini("xfail_strict"))
        raises = mark.kwargs.get("raises", None)
        conditions = mark.args if "condition" not in mark.kwargs else (mark.kwargs["condition"],)
        if not conditions:
            reason = mark.kwargs.get("reason", "")
            return Xfail(reason, run, strict, raises)
        for condition in conditions:
            result, reason = evaluate_condition(item, mark, condition)
            if result:
                return Xfail(reason, run, strict, raises)
    return None

Relationship to Parent Topic and Other Subtopics

Xfail Handling is a specialized extension of test skipping and expected failure support within pytest. While the parent topic covers the broader scope of skipping tests and managing expected failures, this subtopic focuses specifically on the *expected failure* (`xfail`) marker's detailed handling.

It complements the **Skip Marker Evaluation** subtopic by managing the expected failure lifecycle after skip conditions have been resolved. Together, they form the complete set of test exclusion and expectation controls.

Xfail Handling also integrates tightly with:

• Test Execution and Reporting: Modifies test outcomes and reporting logic to reflect expected failures accurately.

• Plugin System and Hooks: Implements hook wrappers (pytest_runtest_call, pytest_runtest_makereport) to inject xfail behavior seamlessly into the test lifecycle.

• Test Discovery and Collection: Relies on marker collection on test items to identify xfail conditions.

This subtopic introduces the logic to interpret `run` and `strict` options, exception filtering via `raises`, and the handling of command line flags like `--runxfail`, which are not covered elsewhere.

Diagram: Xfail Handling Flowchart

flowchart TD
    Start[Start test setup]
    EvaluateXfail[Evaluate xfail marker]
    RunOption{run option?}
    RunTest[Execute test function]
    SkipTest[Skip test execution (xfail run=False)]
    ReportOutcome[Generate test report]
    OutcomeCheck{Test passed or failed?}
    ExceptionCheck{Exception matches raises?}
    MarkXfail[Mark as XFAIL]
    MarkXpass[Mark as XPASS]
    MarkFail[Mark as FAILURE]
    End[End of test handling]

    Start --> EvaluateXfail
    EvaluateXfail --> RunOption
    RunOption -->|False| SkipTest
    RunOption -->|True| RunTest
    SkipTest --> ReportOutcome
    RunTest --> OutcomeCheck
    OutcomeCheck -->|Failed| ExceptionCheck
    OutcomeCheck -->|Passed| MarkXpass
    ExceptionCheck -->|Matches| MarkXfail
    ExceptionCheck -->|No match| MarkFail
    MarkXfail --> ReportOutcome
    MarkXpass --> ReportOutcome
    MarkFail --> ReportOutcome
    ReportOutcome --> End

This flowchart illustrates the decision process pytest follows when handling tests marked as expected failures, balancing execution, skipping, and report annotation according to user configuration and test outcomes.