README.rst

Overview

This directory serves as a dedicated repository for smoke testing popular pytest plugins against the current pytest version. The main purpose is to quickly detect if any changes in pytest—whether intentional or accidental—cause these widely-used plugins to break.

Smoke testing here involves running minimal tests that exercise essential functionality of each plugin, ensuring compatibility without the overhead of full integration testing. This approach helps maintain ecosystem stability given the critical role plugins play in extending pytest functionality.

The folder contains:

• Dependency declarations for selected plugins.

• Minimalistic test scripts (smoke tests) that verify basic plugin operation.

• Supporting files to facilitate plugin testing workflows.

Selection Criteria for Plugins

Plugins included here are chosen based on two main criteria:

1. Wide Usage:
   The plugin is broadly adopted in the pytest community or industry. Consider if a breaking change in pytest affecting this plugin would disrupt a large number of test suites.

2. Minimal External Dependencies:
   Plugins that do not require complex or heavyweight external services, which keeps the smoke tests fast, reliable, and easy to maintain.

Implementation Details

• Each plugin is added as a dependency in the test environment configuration.

• A simple test or a set of minimal tests is created to invoke core features of the plugin. These smoke tests aim to confirm that the plugin loads correctly and basic plugin functionality works without errors.

• Existing tests may be reused or adapted to avoid duplication and reduce maintenance overhead.

• The tests run as part of the continuous integration pipeline for pytest, providing early feedback on potential compatibility issues.

Usage Example

To add a new plugin for smoke testing:

1. Add the plugin to the dependency list of this test suite.

2. Create a minimal smoke test script, for example:

   def test_plugin_smoke():
       # Import the plugin or invoke a simple feature
       import pytest_plugin_x
       result = pytest_plugin_x.some_basic_function()
       assert result is not None
   

3. Add the new test script to the test runner configuration.

4. Run the smoke tests to verify compatibility.

Interactions with the Rest of the System

• This folder is part of the pytest project’s test infrastructure.

• It supplements the main test suite by focusing on third-party plugin compatibility.

• Results from these smoke tests inform maintainers whether changes in pytest core may negatively impact the plugin ecosystem.

• It helps plugin maintainers and the pytest team coordinate to ensure smooth upgrades and avoid regressions.

Mermaid Diagram: Structure of Smoke Testing Setup

flowchart TD
    A[pytest Core] --> B[Smoke Test Runner]
    B --> C[Plugin Dependency Management]
    B --> D[Minimal Smoke Tests]
    D --> E{Individual Plugins}
    E --> F[Plugin A]
    E --> G[Plugin B]
    E --> H[Plugin C]
    style A fill:#f9f,stroke:#333,stroke-width:1px
    style B fill:#bbf,stroke:#333,stroke-width:1px
    style C fill:#bfb,stroke:#333,stroke-width:1px
    style D fill:#fbf,stroke:#333,stroke-width:1px
    style E fill:#ffb,stroke:#333,stroke-width:1px

Explanation:

• pytest Core: The central pytest codebase that undergoes changes.

• Smoke Test Runner: Executes the smoke tests against the current pytest version.

• Plugin Dependency Management: Handles installation and setup of plugin dependencies.

• Minimal Smoke Tests: Contains lightweight tests targeting core plugin functionality.

• Individual Plugins: Represents each plugin under test by the smoke tests.

Summary

This folder provides an essential safeguard for the pytest ecosystem by continuously verifying that popular plugins remain functional as pytest evolves. It prioritizes widely-used and lightweight plugins to maximize impact while maintaining efficiency. The smoke testing strategy enables quick detection of compatibility issues, thereby improving the stability and reliability of pytest and its plugin ecosystem.