test_skipping.py

Overview

The `test_skipping.py` file is a comprehensive test suite designed to validate the behavior of pytest's skipping and expected failure (xfail) mechanisms. It focuses on verifying the correct evaluation and handling of `skip`, `skipif`, and `xfail` markers and functions within the pytest testing framework.

The file contains multiple test classes and standalone test functions that:

• Check marker evaluation logic (skipif, xfail) under various conditions.

• Test the integration of skip and xfail markers with pytest's internal test running protocol.

• Validate how skip and xfail markers behave with setup and teardown functions.

• Ensure that skip and xfail reporting, including reasons and strictness, is accurate and user-friendly.

• Examine edge cases such as boolean skipif conditions without reasons, dynamic marker application, and interaction with pytest namespaces.

• Confirm that pytest's skip and xfail features work correctly when applied at the class level, module level, and via fixtures.

• Test error handling when markers are misused or conditions raise exceptions.

• Validate the output and reporting formats related to skips and xfails in pytest runs.

This file heavily leverages pytest's own testing utilities (e.g., [Pytester](/projects/286/67470) fixture) to create inline test files, run pytest sessions, and assert expected outcomes.

Classes and Functions

Class: TestEvaluation

**Purpose:** Tests the evaluation logic of `skip` and `xfail` markers applied to test items. It verifies how conditions, reasons, and multiple markers are processed and how errors in marker usage are handled.

**Key Methods:**

• test_no_marker(self, pytester: Pytester) -> None
  Verifies that an unmarked test item is not skipped.

• test_marked_xfail_no_args(self, pytester: Pytester) -> None
  Tests a simple @pytest.mark.xfail without arguments leads to an xfail result with an empty reason.

• test_marked_skipif_no_args(self, pytester: Pytester) -> None
  Checks that a skipif marker without arguments causes skipping with an empty reason.

• test_marked_one_arg(self, pytester: Pytester) -> None
  Tests skipif with a condition string and ensures the reason is set to the condition description.

• test_marked_one_arg_with_reason(self, pytester: Pytester) -> None
  Verifies skipif marker with a condition and explicit reason overrides the default condition reason.

• test_marked_one_arg_twice and test_marked_one_arg_twice2
  Validate that multiple skipif markers on a test item are evaluated correctly, with the last true condition being used.

• test_marked_skipif_with_boolean_without_reason(self, pytester: Pytester) -> None
  Ensures that using a boolean skipif without a reason raises a pytest.fail.Exception.

• test_marked_skipif_with_invalid_boolean(self, pytester: Pytester) -> None
  Tests that exceptions raised when evaluating a boolean condition in skipif are properly reported as errors.

• test_skipif_class(self, pytester: Pytester) -> None
  Evaluates skipif markers applied at the class level, confirming class-wide skip behavior.

• test_skipif_markeval_namespace(self, pytester: Pytester) -> None
  Tests that the custom namespace returned by pytest_markeval_namespace is used for evaluating skip conditions.

• test_skipif_markeval_namespace_multiple(self, pytester: Pytester) -> None
  Tests that nested plugins' pytest_markeval_namespace override higher-level ones during skip evaluation.

• test_skipif_markeval_namespace_ValueError(self, pytester: Pytester) -> None
  Confirms that returning a non-dict from pytest_markeval_namespace raises a ValueError.

Class: TestXFail

**Purpose:** Focuses on testing the behavior of the `xfail` marker, including strictness, conditional evaluation, reporting formats, and dynamic application of xfail markers.

**Key Methods:**

• test_xfail_simple(self, pytester: Pytester, strict: bool) -> None
  Parametrized test verifying basic xfail behavior with strict set to True or False, ensuring the test is skipped and marked as xfail.

• test_xfail_xpassed(self, pytester: Pytester) -> None
  Checks that a test marked xfail which passes is reported as XPASS with the correct reason.

• test_xfail_using_platform(self, pytester: Pytester) -> None
  Ensures that platform-dependent xfail conditions are correctly evaluated.

• test_xfail_xpassed_strict(self, pytester: Pytester) -> None
  Tests that an XPASS under strict=True fails the test run and reports correctly.

• test_xfail_run_anyway(self, pytester: Pytester) -> None
  Validates running xfail tests when --runxfail is used, ensuring failing tests actually fail.

• test_xfail_run_with_skip_mark(self, pytester: Pytester, test_input, expected) -> None
  Parametrized test ensuring the interaction of xfail and skip markers during pytest runs with different command line options.

• test_xfail_evalfalse_but_fails(self, pytester: Pytester) -> None
  Tests that an xfail with condition False but a failing test does not count as xfail but as a failure.

• test_xfail_not_run_xfail_reporting(self, pytester: Pytester) -> None
  Verifies reporting of tests marked as xfail(run=False) showing NOTRUN status.

• test_xfail_xpass(self, pytester: Pytester) -> None
  Confirms that passing an xfail test is reported as XPASS.

• test_xfail_imperative(self, pytester: Pytester) -> None
  Tests the imperative usage of pytest.xfail() inside test functions.

• test_xfail_imperative_in_setup_function(self, pytester: Pytester) -> None
  Confirms that calling pytest.xfail() in setup functions works as expected.

• test_dynamic_xfail_set_during_funcarg_setup(self, pytester: Pytester) -> None
  Tests setting xfail dynamically during fixture setup.

• test_dynamic_xfail_set_during_runtest_failed(self, pytester: Pytester) -> None
  Verifies dynamic xfail marker application during test runtime failure.

• test_xfail_raises(self, expected, actual, matchline, pytester: Pytester) -> None
  Parametrized test checking raises parameter of xfail markers for various exception classes.

• test_strict_sanity(self, pytester: Pytester) -> None
  Sanity check ensuring strict xfail causes the correct behavior on failures.

• test_strict_xfail(self, pytester: Pytester, strict: bool) -> None
  Parametrized test verifying strict xfail behavior and correct test execution.

• test_xfail_markeval_namespace(self, pytester: Pytester) -> None
  Checks that xfail markers honor the custom namespace returned by pytest_markeval_namespace().

Class: TestXFailwithSetupTeardown

**Purpose:** Tests xfail behavior when setup or teardown functions raise exceptions.

**Key Methods:**

• test_failing_setup_issue9(self, pytester: Pytester) -> None
  Tests that a failing setup function combined with xfail test marks the test as xfailed.

• test_failing_teardown_issue9(self, pytester: Pytester) -> None
  Same as above but for teardown functions.

Class: TestSkip

**Purpose:** Tests various scenarios of the `skip` marker and [pytest.skip()](/projects/286/67266) function, including class-level skips, skips with reasons, and error cases.

**Key Methods:**

• test_skip_class(self, pytester: Pytester) -> None
  Verifies that class-level skip markers skip all tests in the class.

• test_skips_on_false_string(self, pytester: Pytester) -> None
  Checks that @pytest.mark.skip('False') skips the test (string treated as reason).

• test_arg_as_reason(self, pytester: Pytester) -> None
  Ensures string argument to skip is displayed as skip reason.

• test_skip_no_reason(self, pytester: Pytester) -> None
  Validates skip with no reason shows an unconditional skip message.

• test_only_skips_marked_test(self, pytester: Pytester) -> None
  Confirms only marked tests are skipped, others run normally.

• test_strict_and_skip(self, pytester: Pytester) -> None
  Tests skip behavior under --strict-markers option.

• test_wrong_skip_usage(self, pytester: Pytester) -> None
  Ensures misuse of skip (passing boolean condition) raises TypeError.

Class: TestSkipif

**Purpose:** Tests the `skipif` marker in various forms, including conditional skipping, reporting, and interaction with platform and multiple markers.

**Key Methods:**

• test_skipif_conditional(self, pytester: Pytester) -> None
  Confirms that a skipif condition triggers a skip exception with the correct message.

• test_skipif_reporting(self, pytester: Pytester, params) -> None
  Parametrized test verifying skip reporting output for varying skipif parameters.

• test_skipif_using_platform(self, pytester: Pytester) -> None
  Tests that platform-dependent skipif conditions work correctly.

• test_skipif_reporting_multiple(self, pytester: Pytester, marker, msg1, msg2) -> None
  Verifies reporting of multiple skipif or xfail markers on a test.

Standalone Test Functions

The file also contains numerous standalone test functions testing:

• Default skip reporting (test_skip_not_report_default)

• Class-level skipif behavior (test_skipif_class)

• Functional skipped reasons and folding (test_skipped_reasons_functional, test_skipped_folding)

• Output report characters (test_reportchars, test_reportchars_error, test_reportchars_all, test_reportchars_all_error)

• Error handling in skipif/xfail expressions (test_errors_in_xfail_skip_expressions)

• Interaction of skip and xfail with globals and namespaces (test_xfail_skipif_with_globals, test_default_markers)

• Module-level skips and skip with allow_module_level (test_module_level_skip_error, test_module_level_skip_with_allow_module_level)

• Marker application on non-Python test items (test_xfail_item, test_mark_xfail_item)

• Behavior with pytest.importorskip() and pytest.skip() with invalid parameters

• Summary list correctness after errors (test_summary_list_after_errors)

• Imperative skipping on xfail tests (test_imperativeskip_on_xfail_test)

• Boolean condition skipif and xfail tests (TestBooleanCondition class)

• Various edge cases around strictness and conditions for xfail

Important Implementation Details

• The tests use the Pytester fixture extensively, which allows creating test modules/files dynamically and running pytest on them, capturing results for assertion.

• The file tests both marker-based skipping/xfail and imperative skipping via pytest.skip() and pytest.xfail().

• It verifies both evaluation of conditions (including string expressions evaluated in a controlled namespace) and reporting formats (e.g., output lines like SKIPPED [1] test_file.py:2: reason).

• Some tests simulate nested plugins or conftest files to verify how namespace evaluation (pytest_markeval_namespace) affects skip/xfail behavior.

• Edge cases such as boolean conditions without reasons, exceptions raised during evaluation, and invalid marker parameters are checked to ensure robust error reporting.

• The file exercises pytest's internal test running protocol (runtestprotocol) to inspect test reports for expected skip/xfail states.

Interaction with Other Parts of the System

• Relies on internal pytest modules _pytest.pytester, _pytest.runner, and _pytest.skipping for test item creation, test running, and marker evaluation.

• Uses pytest core features such as markers (skip, skipif, xfail), test item and report APIs, and plugin hook pytest_markeval_namespace.

• Interacts with pytest CLI options (--runxfail, -rs, -rx, --strict-markers, etc.) to test different runtime behaviors.

• Tests integration with the pytest test report and terminal output reporting system.

• Exercises pytest's fixture and setup/teardown mechanisms to test skip/xfail behavior in those contexts.

Usage Examples

Since this is a test suite, usage is primarily for pytest developers or contributors to run and verify pytest's skipping and xfail functionality.

Example to run the tests in this file:

pytest test_skipping.py

Example of a snippet tested inside:

import pytest

@pytest.mark.skipif("hasattr(os, 'sep')", reason="OS does not have sep")
def test_func():
    pass

This should cause the test to be skipped with the reason `"OS does not have sep"`.

Mermaid Class Diagram of test_skipping.py

classDiagram
    class TestEvaluation {
        +test_no_marker(pytester)
        +test_marked_xfail_no_args(pytester)
        +test_marked_skipif_no_args(pytester)
        +test_marked_one_arg(pytester)
        +test_marked_one_arg_with_reason(pytester)
        +test_marked_one_arg_twice(pytester)
        +test_marked_one_arg_twice2(pytester)
        +test_marked_skipif_with_boolean_without_reason(pytester)
        +test_marked_skipif_with_invalid_boolean(pytester)
        +test_skipif_class(pytester)
        +test_skipif_markeval_namespace(pytester)
        +test_skipif_markeval_namespace_multiple(pytester)
        +test_skipif_markeval_namespace_ValueError(pytester)
    }

    class TestXFail {
        +test_xfail_simple(pytester, strict)
        +test_xfail_xpassed(pytester)
        +test_xfail_using_platform(pytester)
        +test_xfail_xpassed_strict(pytester)
        +test_xfail_run_anyway(pytester)
        +test_xfail_run_with_skip_mark(pytester, test_input, expected)
        +test_xfail_evalfalse_but_fails(pytester)
        +test_xfail_not_run_xfail_reporting(pytester)
        +test_xfail_not_run_no_setup_run(pytester)
        +test_xfail_xpass(pytester)
        +test_xfail_imperative(pytester)
        +test_xfail_imperative_in_setup_function(pytester)
        +test_dynamic_xfail_set_during_funcarg_setup(pytester)
        +test_dynamic_xfail_set_during_runtest_failed(pytester)
        +test_dynamic_xfail_set_during_runtest_passed_strict(pytester)
        +test_xfail_raises(expected, actual, matchline, pytester)
        +test_strict_sanity(pytester)
        +test_strict_xfail(pytester, strict)
        +test_strict_xfail_condition(pytester, strict)
        +test_xfail_condition_keyword(pytester, strict)
        +test_strict_xfail_default_from_file(pytester, strict_val)
        +test_xfail_markeval_namespace(pytester)
    }

    class TestXFailwithSetupTeardown {
        +test_failing_setup_issue9(pytester)
        +test_failing_teardown_issue9(pytester)
    }

    class TestSkip {
        +test_skip_class(pytester)
        +test_skips_on_false_string(pytester)
        +test_arg_as_reason(pytester)
        +test_skip_no_reason(pytester)
        +test_skip_with_reason(pytester)
        +test_only_skips_marked_test(pytester)
        +test_strict_and_skip(pytester)
        +test_wrong_skip_usage(pytester)
    }

    class TestSkipif {
        +test_skipif_conditional(pytester)
        +test_skipif_reporting(pytester, params)
        +test_skipif_using_platform(pytester)
        +test_skipif_reporting_multiple(pytester, marker, msg1, msg2)
    }

    TestEvaluation <|-- TestXFail
    TestEvaluation <|-- TestXFailwithSetupTeardown
    TestEvaluation <|-- TestSkip
    TestEvaluation <|-- TestSkipif

Summary

`test_skipping.py` is a detailed pytest test suite ensuring the correctness and robustness of pytest's skip and xfail features. It covers a wide range of cases, from simple marker usage to complex dynamic and conditional skipping/failing, including error handling and reporting accuracy. This file is crucial for maintaining pytest's core functionality related to test skipping and expected failures.

If you need further explanation on specific tests or integration points, feel free to ask!