13522.bugfix.rst

Overview

This file documents a bug fix related to the `pytester` fixture in the pytest testing framework, specifically addressing issues with its behavior in subprocess mode when handling plugins. The fix resolves two main problems:

1. The pytester fixture in subprocess mode was ignoring all plugins except the first one specified in the pytester.plugins attribute.

2. The fixture silently ignored plugins specified as non-string types in pytester.plugins, which could lead to unexpected behavior without raising errors.

After the fix, the `pytester` fixture properly respects all plugins specified as strings and raises an error if a plugin is specified with a non-string type in subprocess mode. The documentation also advises users affected by this change to specify plugins by name (string) or alternatively use the `pytester.runpytest_inprocess` function for in-process test runs where plugin handling differs.

Detailed Explanation

Context: pytester Fixture and Plugins

• The pytester fixture is a testing utility provided by pytest to facilitate testing Pytest plugins and behaviors.

• It supports running tests in subprocess mode, which simulates a real pytest session in a separate process.

• Plugins can be passed to pytester via the pytester.plugins attribute, which should contain plugin names or plugin objects.

Bugs Fixed

1. Ignoring all but the first plugin

   • Before the fix, when running pytester in subprocess mode, only the first plugin in pytester.plugins was loaded.

   • This caused tests relying on multiple plugins to fail or behave incorrectly.

2. Silent ignoring of non-string plugins

   • Plugins not specified as strings (for example, plugin objects or other types) were silently ignored.

   • This behavior was problematic because it masked configuration errors and led to unexpected test behavior.

Behavior After Fix

• All plugins specified as strings in pytester.plugins are properly loaded in subprocess mode.

• If pytester.plugins contains non-string types, an error is raised to inform the user of the improper configuration.

• Users needing to specify plugins as objects or non-string values are encouraged to switch to pytester.runpytest_inprocess, which runs tests in-process and supports such plugin types.

Usage Guidance

• Specify plugins as strings by their plugin names when using pytester in subprocess mode to avoid errors.

• For tests requiring non-string plugin references, use pytester.runpytest_inprocess explicitly.

Example:

def test_with_plugins(pytester):
    # Correct: specify plugins by name (string)
    pytester.plugins = ["pytest_cov", "pytest_mock"]
    result = pytester.runpytest_subprocess()
    assert result.ret == 0
    
def test_with_object_plugins(pytester):
    # For plugin objects, use in-process run
    pytester.plugins = [MyPlugin()]
    result = pytester.runpytest_inprocess()
    assert result.ret == 0

Important Implementation Details

• The fix likely involves modifying the subprocess invocation logic in pytester to iterate over all plugins in pytester.plugins and validate their types.

• Non-string plugins trigger an error early, preventing silent misconfigurations.

• The subprocess mode now faithfully reproduces plugin loading as expected in real pytest runs.

Interaction With Other Parts of the System

• This bug fix affects the pytester testing utility within the pytest framework.

• pytester is commonly used in pytest's own test suite and by plugin developers to test plugin behavior.

• The fix improves reliability when running tests that involve multiple plugins or when plugins are specified dynamically.

• It ensures better alignment between subprocess test runs and the in-process behavior, facilitating consistent plugin loading.

• The guidance to use pytester.runpytest_inprocess for non-string plugins clarifies usage patterns and avoids confusion.

Visual Diagram

The following class diagram illustrates the relevant components related to `pytester` and plugin handling in the context of this fix:

classDiagram
    class Pytester {
        +plugins: List[str or object]
        +runpytest_subprocess()
        +runpytest_inprocess()
    }
    Pytester : +runpytest_subprocess() - subprocess mode runner, expects plugins as strings
    Pytester : +runpytest_inprocess() - in-process runner, accepts plugin objects
    class Plugin {
        <<interface>>
    }
    Pytester --> Plugin : uses plugins

• Pytester represents the pytest pytester fixture object.

• It has a plugins attribute that holds plugins (strings or objects).

• runpytest_subprocess() runs tests in a subprocess and requires plugins as strings.

• runpytest_inprocess() runs tests in-process and accepts plugin objects.

• The fix improves the subprocess mode's handling of plugins.

Summary

This bug fix improves the robustness and correctness of the `pytester` fixture's subprocess mode by:

• Ensuring all specified plugins are loaded, not just the first.

• Raising errors for invalid plugin types instead of silently ignoring them.

• Providing clear guidance for users on how to specify plugins and when to use in-process runs.

This enhances plugin testing fidelity and prevents subtle errors in plugin configuration during pytest plugin development and testing.