skipping.rst

Overview

This documentation file provides an extensive tutorial and reference guide on how to use **skip** and **xfail** markers and functions within the pytest testing framework. Its primary purpose is to instruct users on managing test cases that are either not expected to run successfully or should be conditionally ignored, enabling a clean and meaningful test suite report.

The file explains the rationale behind skipping tests (tests that are not applicable or cannot run under certain conditions) and expected failures (tests that are known to fail due to bugs, unimplemented features, or environmental issues). It covers usage patterns, parameter options, and best practices for marking tests to control test execution flow and improve test suite maintainability.

Contents include:

• How to skip tests unconditionally or conditionally.

• Using skip and skipif decorators and functions.

• Marking tests as expected to fail (xfail) with detailed options.

• Handling skipped tests and expected failures during test execution and reporting.

• Using markers with parameterized tests.

The document also explains how these markers affect pytest's test collection, execution, and reporting behaviors.

Detailed Explanations

1. Skipping Tests

Skipping tests means instructing pytest not to run certain tests under specific conditions. This can be useful for platform-dependent tests, missing resource conditions, or temporarily disabling unstable tests.

1.1 @pytest.mark.skip

Decorator to unconditionally skip a test function or method.

**Parameters:**

• reason (str): Explanation for skipping the test, shown in reports.

**Example:**

@pytest.mark.skip(reason="no way of currently testing this")
def test_the_unknown():
    ...

**Usage Notes:**

• The skip reason helps document why the test is skipped.

• The decorator prevents the test from running and pytest reports it as skipped.

1.2 pytest.skip(reason)

Imperative function to skip tests during runtime, useful when skip condition can't be evaluated at import time.

**Parameters:**

• reason (str): Explanation for skipping.

**Example:**

def test_function():
    if not valid_config():
        pytest.skip("unsupported configuration")

**Additional:**

Skipping whole modules by calling `pytest.skip(reason, allow_module_level=True)` at module level is supported.

1.3 @pytest.mark.skipif(condition, reason)

Decorator for conditional skipping — skips the test if `condition` evaluates to `True`.

**Parameters:**

• condition (bool): Expression evaluated at collection time.

• reason (str): Explanation for the skip.

**Example:**

import sys

@pytest.mark.skipif(sys.version_info < (3, 10), reason="requires python3.10 or higher")
def test_function():
    ...

**Usage Notes:**

• Can be applied to test functions, classes, or modules (pytestmark = pytest.mark.skipif(...)).

• Multiple skipif decorators on the same test will skip if any condition is True.

• Can share skipif markers across modules by importing them.

1.4 Skipping based on missing imports

Use `pytest.importorskip("module_name")` to skip tests if an import fails or version requirements are not met.

**Example:**

docutils = pytest.importorskip("docutils", minversion="0.3")

This skips tests if `docutils` is not installed or version < 0.3.

2. Expected Failures (xfail)

Mark tests that are expected to fail — useful for known bugs, unimplemented features, or flaky tests.

2.1 @pytest.mark.xfail

Decorator marking tests expected to fail.

**Parameters:**

• condition (bool, optional): Condition when the test is expected to fail.

• reason (str, optional): Reason for expected failure.

• raises (Exception or tuple, optional): Expected exception(s) causing the failure.

• run (bool, optional, default=True): Whether to execute the test or not.

• strict (bool, optional): Make unexpected passes (XPASS) cause test suite failure.

**Example:**

@pytest.mark.xfail(reason="known parser issue")
def test_function():
    ...

@pytest.mark.xfail(sys.platform == "win32", reason="bug in a 3rd party library")
def test_function():
    ...

@pytest.mark.xfail(raises=RuntimeError)
def test_function():
    ...

@pytest.mark.xfail(run=False)
def test_function():
    ...

@pytest.mark.xfail(strict=True)
def test_function():
    ...

2.2 pytest.xfail(reason)

Imperative call to mark a test as expected to fail within the test or setup function.

**Example:**

def test_function():
    if not valid_config():
        pytest.xfail("failing configuration (but should work)")

Note: When called, no further test code is executed as `pytest.xfail` raises a special exception internally.

2.3 Command line options

• Use pytest -rx to show extra info on xfailed, xpassed, and skipped tests.

• Use --runxfail to force running and reporting of xfail tests as normal tests.

3. Skip/Xfail with Parameterized Tests

Markers can be applied to individual test instances when using `pytest.mark.parametrize` by marking parameters explicitly.

**Example:**

import sys
import pytest

@pytest.mark.parametrize(
    ("n", "expected"),
    [
        (1, 2),
        pytest.param(1, 0, marks=pytest.mark.xfail),
        pytest.param(1, 3, marks=pytest.mark.xfail(reason="some bug")),
        (2, 3),
        (3, 4),
        (4, 5),
        pytest.param(
            10, 11, marks=pytest.mark.skipif(sys.version_info >= (3, 0), reason="py2k")
        ),
    ],
)
def test_increment(n, expected):
    assert n + 1 == expected

Implementation Details & Algorithms

• The skip decorator and function cause pytest to mark the test as skipped and not execute it.

• The skipif decorator evaluates the condition at collection/import time, skipping the test if True.

• The xfail decorator marks tests expected to fail but still runs them. Failures matching expected exceptions or conditions are reported as XFAIL without failing the suite.

• The pytest.xfail() function raises a special exception internally to immediately stop test execution and mark it as expected failure.

• Strict mode (strict=True) converts unexpected passes (XPASS) into test suite failures.

• Parametrized tests can selectively apply skip/xfail markers to individual parameter sets.

Interactions with Other Parts of the System

• This file documents usage of pytest's core marker and skip/xfail APIs.

• It references pytest command line options (-r, --runxfail) that control reporting and execution behavior.

• It interacts with pytest's test collection and execution machinery by influencing which tests are run, skipped, or expected to fail.

• It suggests organizing markers centrally in a shared module for large test suites, enhancing maintainability.

• It refers to related pytest concepts like test collection customization and import skipping.

Visual Diagram

The following Mermaid class diagram represents the core concepts and relationships of skip and xfail markers and functions as used in this file:

classDiagram
    class SkipMarker {
        +reason: str
        +condition: bool (optional)
        +__call__()
    }
    class XFailMarker {
        +condition: bool (optional)
        +reason: str (optional)
        +raises: Exception or tuple (optional)
        +run: bool = True
        +strict: bool = False
        +__call__()
    }
    class PytestFunctions {
        +skip(reason: str, allow_module_level: bool = False)
        +xfail(reason: str)
        +importorskip(module_name: str, minversion: str = None)
    }
    class TestFunction {
        +name: str
        +markers: list
        +run()
    }

    SkipMarker <|-- TestFunction : applied to
    XFailMarker <|-- TestFunction : applied to
    PytestFunctions ..> TestFunction : runtime control

Summary

This file serves as a comprehensive guide to using pytest's skip and xfail features, providing both declarative (decorator) and imperative (function call) methods to control test execution based on dynamic conditions or known test statuses. It helps maintain a clean test report by clearly signaling tests that are not run or are expected to fail, improving test suite reliability and developer productivity.