test_mark.py

Overview

[test_mark.py](/projects/286/67332) is a comprehensive test suite for the `pytest` testing framework, focusing primarily on the functionality and behavior of **pytest markers** and **keyword selection** features. Markers in pytest are used to add metadata to test functions and classes, enabling conditional test execution, categorization, and parameterization.

This file contains numerous unit and functional tests that validate:

• The correct behavior of markers and marker decorators.

• The interaction of markers with parameterized tests.

• Marker expression parsing and evaluation via the -m option.

• Keyword-based test selection with the -k command line option.

• Edge cases and regression checks related to markers.

• Integration and consistency of markers across test collection and execution phases.

The tests utilize `pytest`'s own testing utilities, especially the [Pytester](/projects/286/67470) helper fixture, to dynamically create test files, run pytest programmatically, and assert outcomes.

Detailed Explanation of Classes and Functions

Class: TestMark

This class groups tests that check basic marker functionality, such as:

• Existence of marker-related attributes in pytest.

• Behavior of MarkGenerator and marker callability.

• Marker application on functions and classes.

• Restrictions on marker names starting with underscores.

Key Methods:

• test_pytest_exists_in_namespace_all(self, attr: str) -> None
  Verifies that the strings "mark" and "param" appear in pytest.__all__, ensuring they are part of the public API.

• test_pytest_mark_notcallable(self) -> None
  Confirms that a MarkGenerator instance cannot be called directly, expecting a TypeError.

• test_mark_with_param(self)
  Tests that applying a marker to a function or class returns the original object, while using .with_args returns a callable different from the original (used for parametrized marks).

• test_pytest_mark_name_starts_with_underscore(self) -> None
  Asserts that accessing marker names starting with an underscore raises an AttributeError.

Function: test_marked_class_run_twice(pytester: Pytester) -> None

Tests a regression where a file containing a class marked with `@pytest.mark.parametrize` runs twice unexpectedly. It creates a test file with a parameterized class and verifies that all tests pass exactly once.

Functions Testing Marker Configuration and Command-line Options

• test_ini_markers(pytester: Pytester) -> None
  Checks that markers declared in pytest.ini are recognized and available at runtime.

• test_markers_option(pytester: Pytester) -> None
  Validates that the --markers command-line option outputs all defined markers, including those without descriptions.

• test_ini_markers_whitespace(pytester: Pytester) -> None
  Ensures that marker names with whitespace in pytest.ini are parsed correctly.

• test_marker_without_description(pytester: Pytester) -> None
  Tests that markers without descriptions in a .cfg file or conftest.py do not cause errors when running with --strict-markers.

• test_markers_option_with_plugin_in_current_dir(pytester: Pytester) -> None
  Verifies that markers added dynamically by plugins are shown with --markers.

• test_mark_on_pseudo_function(pytester: Pytester) -> None
  Checks that marking a function with a callable that raises an exception does not break pytest.

Parameterized Tests for Marker Expression Parsing (-m option)

• test_strict_prohibits_unregistered_markers(pytester: Pytester, option_name: str)
  Confirms that the --strict-markers and --strict options cause pytest to error on unregistered markers.

• test_mark_option(expr: str, expected_passed: list[str | None], pytester: Pytester)
  Tests various marker expressions with -m to filter which tests run.

• test_mark_option_with_kwargs(expr: str, expected_passed: list[str | None], pytester: Pytester)
  Tests marker expressions with keyword arguments, verifying correct test selection.

• test_mark_option_custom(expr: str, expected_passed: list[str], pytester: Pytester)
  Tests custom markers added dynamically during collection and their filtering with -m.

Class: TestFunctional

Contains more integration/functional style tests verifying marker behavior in more complex scenarios, including:

• Marker merging in nested classes.

• Marker propagation via class inheritance and base classes.

• Closest marker retrieval with get_closest_marker().

• Dynamic marker application in fixtures.

• Keyword presence and interaction with session and node keywords.

• Marker order preservation with add_marker.

• Marker application on classmethod and staticmethod.

Notable Methods:

• assert_markers(self, items, **expected) -> None
  Helper method to assert expected markers on collected test items.

• test_mark_closest(self, pytester: Pytester) -> None
  Tests that the closest marker is correctly identified on test functions and classes.

• test_mark_dynamically_in_funcarg(self, pytester: Pytester) -> None
  Demonstrates adding markers dynamically during fixture setup.

Class: TestKeywordSelection

Tests related to the keyword expression selection feature (`-k` option):

• test_select_simple(self, pytester: Pytester) -> None
  Basic tests for selecting tests based on keywords in names.

• test_select_extra_keywords(self, pytester: Pytester, keyword)
  Tests adding custom extra keywords to tests and filtering with -k.

• test_keyword_extra(self, pytester: Pytester) -> None
  Tests filtering by arbitrary attributes added to test functions.

• test_no_magic_values(self, pytester: Pytester, keyword: str) -> None
  Ensures that magic names like dict or + do not match tests unexpectedly.

• test_no_match_directories_outside_the_suite(self, pytester: Pytester, monkeypatch: pytest.MonkeyPatch) -> None
  Ensures keyword selections do not match directories outside the test suite.

Class: TestMarkDecorator

Tests behavior of marker decorator instances:

• Equality comparisons between markers and strings.

• Attribute aliases for marker name, args, and kwargs.

Various Independent Test Functions

• Tests for pytest.param() id argument validation.

• Tests for marker expression error messages.

• Tests for parameterized test collection and marker application.

• Tests for handling empty parameter sets in parameterized tests.

• Tests for marker order and merging.

• Tests for markers on parameterized tests and fixtures.

• Test for marker method resolution order (MRO) and inheritance.

Important Implementation Details and Algorithms

• Marker Expression Parsing:
  The tests extensively verify that marker expressions passed to pytest -m are parsed and evaluated correctly, including logical operators (and, or, not), parentheses, and markers with keyword arguments.

• Marker Propagation:
  Tests ensure that markers applied at class level propagate to methods, taking into account inheritance hierarchies and nested classes.

• Dynamic Marker Application:
  Markers may be added dynamically during fixture setup or test collection, and tests verify that such markers are honored.

• Strict Marker Checking:
  With options like --strict-markers, unregistered markers cause pytest to error, and the tests verify this enforcement.

• Marker Ordering:
  The order in which markers are applied and stored is preserved, which can affect test behavior (e.g., skip markers applied last take precedence).

• Keyword Selection:
  The -k option supports complex boolean expressions over test function/class names and custom keywords attached to tests, with safeguards to prevent unintended matches on special names.

• Handling of Empty Parameter Sets:
  The tests verify behavior when parameterized tests have empty parameter sets, including skipping or failing at collection time depending on configuration.

Interaction with Other System Components

• Pytester Fixture:
  Almost all tests use the pytester.Pytester fixture to create temporary test files, configure pytest projects, and run pytest programmatically. This simulates real test runs in isolated environments.

• _pytest.mark Module:
  The file directly tests and uses components from _pytest.mark such as MarkGenerator, marker structures, and utilities like get_empty_parameterset_mark.

• Test Collection and Node APIs:
  Tests interact with pytest's collection mechanism, using pytest.inline_genitems and Node APIs to inspect collected test items and their markers.

• Configuration Parsing:
  The tests verify that markers configured via .ini files or pytest.ini are correctly recognized by pytest.

• Command-line Option Parsing:
  The test suite verifies behavior of pytest CLI options -m (marker expression) and -k (keyword expression), including error handling and output formatting.

Usage Examples

Example 1: Using Markers in Tests

import pytest

@pytest.mark.slow
def test_long_running():
    assert True

@pytest.mark.parametrize("x", [1, 2, 3])
def test_param(x):
    assert x in [1, 2, 3]

Example 2: Running Tests with Marker Expressions

Run only tests marked as `slow`:

pytest -m slow

Run tests marked `car` with `color='red'`:

pytest -m "car(color='red')"

Example 3: Filtering Tests by Keywords (-k)

Run tests whose names match `test_interface` or exclude those:

pytest -k interface
pytest -k "not interface"

Mermaid Class Diagram of TestMark Class and Related Test Classes

classDiagram
    class TestMark {
        +test_pytest_exists_in_namespace_all(attr: str)
        +test_pytest_mark_notcallable()
        +test_mark_with_param()
        +test_pytest_mark_name_starts_with_underscore()
    }

    class TestFunctional {
        +test_merging_markers_deep(pytester)
        +test_mark_decorator_subclass_does_not_propagate_to_base(pytester)
        +test_mark_should_not_pass_to_siebling_class(pytester)
        +test_mark_decorator_baseclasses_merged(pytester)
        +test_mark_closest(pytester)
        +test_mark_with_wrong_marker(pytester)
        +test_mark_dynamically_in_funcarg(pytester)
        +test_no_marker_match_on_unmarked_names(pytester)
        +test_keywords_at_node_level(pytester)
        +test_keyword_added_for_session(pytester)
        +assert_markers(items, **expected)
        +test_mark_from_parameters(pytester)
        +test_reevaluate_dynamic_expr(pytester)
    }

    class TestKeywordSelection {
        +test_select_simple(pytester)
        +test_select_extra_keywords(pytester, keyword)
        +test_keyword_extra(pytester)
        +test_no_magic_values(pytester, keyword)
        +test_no_match_directories_outside_the_suite(pytester, monkeypatch)
    }

    class TestMarkDecorator {
        +test__eq__(lhs, rhs, expected)
        +test_aliases()
    }

Summary

The [test_mark.py](/projects/286/67332) file is a critical part of the pytest test suite that ensures the robustness, correctness, and expected behavior of pytest's marker and keyword selection features. It covers a wide array of scenarios, from basic marker application to complex expression parsing and dynamic marker interactions with fixtures and test collection. This suite helps maintain pytest's flexible and powerful test selection capabilities, crucial for users managing large and complex test bases.

End of Documentation for test_mark.py