Skip Marker Evaluation

Purpose

Skip Marker Evaluation addresses the need to conditionally or unconditionally skip tests before they are executed. Within the broader topic of test skipping and expected failure handling, this subtopic focuses specifically on evaluating `skip` and `skipif` markers applied to test items, determining when a test should be bypassed based on user-defined conditions or static decisions. This evaluation helps avoid unnecessary test runs (e.g., on unsupported platforms or unmet dependencies), improving test suite efficiency and clarity.

Functionality

The core of Skip Marker Evaluation lies in analyzing the skip-related markers attached to a test item and deciding whether the test should be skipped, and if so, providing an appropriate reason.

Key steps and workflows include:

• Marker Iteration: The system scans for skipif and skip markers on a test item.

• Condition Extraction: For skipif, conditions can be passed as positional arguments or via a condition keyword argument.

• Condition Evaluation:

  • If the condition is a string, it is dynamically evaluated using eval() in a controlled namespace including standard modules (os, sys, platform) and test globals.

  • If the condition is a boolean, it is simply cast to bool().

  • Errors during evaluation result in test failure with detailed messages.

• Reason Handling:

  • If the skip condition is met, a reason string is extracted from the marker or synthesized from the condition.

  • For unconditional skips, the reason defaults to "unconditional skip".

• Skip Decision:

  • If any skipif condition evaluates to true, or if an unconditional skip marker exists, the test is flagged to be skipped.

• Integration with Test Setup:

  • The evaluation runs early in the test lifecycle during the pytest_runtest_setup hook, raising a skip exception to prevent test execution if conditions require.

Example snippet illustrating condition evaluation logic:

def evaluate_condition(item: Item, mark: Mark, condition: object) -> tuple[bool, str]:
    if isinstance(condition, str):
        # Evaluate string condition with safe globals
        result = eval(condition_code, globals_)
    else:
        # Evaluate boolean condition
        result = bool(condition)
    reason = mark.kwargs.get("reason") or f"condition: {condition}" if isinstance(condition, str) else fail(...)
    return result, reason

Integration

Skip Marker Evaluation integrates tightly with the parent topic by serving as the gatekeeper step that determines whether a test should be executed or skipped. It complements the handling of expected failures (`xfail`) by ensuring that tests flagged as skipped are not run unnecessarily.

• With Test Execution:
  The skip evaluation occurs in the pytest_runtest_setup hook, which runs before test execution phases, thus preventing skipped tests from wasting resources.

• With Markers and Configuration:
  Skip markers can be defined inline in test code or dynamically added by plugins. The evaluation uses the test item’s iter_markers method to discover relevant markers and respects configuration options and global namespaces.

• With Reporting:
  When a test is skipped due to this evaluation, detailed reasons are recorded and displayed in test reports, aiding diagnostics.

• With Other Subtopics:
  Unlike Xfail Handling, which manages expected failures and their outcomes post-execution, Skip Marker Evaluation acts pre-execution. It also differs from Test Skipping and Expected Failures by focusing solely on the evaluation logic of skips before tests run.

Diagram

A flowchart best illustrates the decision process for evaluating skip markers on a test item:

flowchart TD
    Start[Start Skip Evaluation]
    GetMarkers[Retrieve skipif & skip markers]
    CheckSkipif{Any skipif markers?}
    EvalConditions[Evaluate each skipif condition]
    ConditionTrue{Condition True?}
    ReturnSkipif[Return Skip(reason)]
    CheckSkip{Any skip markers?}
    ReturnSkip[Return Skip(reason)]
    NoSkip[Return None - no skip]
    
    Start --> GetMarkers
    GetMarkers --> CheckSkipif
    CheckSkipif -- Yes --> EvalConditions
    EvalConditions --> ConditionTrue
    ConditionTrue -- Yes --> ReturnSkipif
    ConditionTrue -- No --> CheckSkip
    CheckSkipif -- No --> CheckSkip
    CheckSkip -- Yes --> ReturnSkip
    CheckSkip -- No --> NoSkip

This process ensures that tests are skipped precisely according to user-defined conditions or unconditional skip declarations, optimizing test runs and improving clarity in test outcomes.