update-plugin-list.py

Overview

`update-plugin-list.py` is a utility script designed to automate the retrieval, processing, and generation of a comprehensive list of `pytest` plugins available on PyPI. It fetches plugin metadata from the PyPI JSON API, filters and organizes the data, and then outputs a reStructuredText (RST) formatted file (`plugin_list.rst`) that documents available plugins, their statuses, last release dates, and dependency requirements.

This script is typically run on a weekly basis (often via a GitHub Action) to keep the plugin list up-to-date for inclusion in the official pytest documentation.

Functionality Summary

Uses a cached HTTP session to query PyPI for all packages starting with pytest- or pytest_ as well as some manually curated additional projects.
Filters out inactive projects based on their PyPI classifiers.
Collects metadata including plugin name, summary, last release date, development status, and pytest dependencies.
Generates a formatted RST file containing the plugin list both as a table (for HTML/web) and as detailed definitions (for LaTeX/PDF).
Caches HTTP requests locally to minimize repeated network calls and speed up subsequent runs.

Detailed Explanation of Components

Constants

FILE_HEAD
A multi-line string that forms the header of the generated RST file, including notes about the file's autogenerated nature and disclaimers regarding plugin endorsement.
DEVELOPMENT_STATUS_CLASSIFIERS
Tuple of PyPI development status classifier strings to categorize plugins (from Planning to Inactive).
ADDITIONAL_PROJECTS
Set of manually included projects that do not follow the usual pytest- or pytest_ naming convention but should be considered plugins.

Functions

`escape_rst(text: str) -> str`

Escapes special reStructuredText characters in a string to prevent unwanted formatting when rendered.

Parameters:
text (str): Input string potentially containing RST special characters.
Returns:
(str) The input string with certain characters escaped.

Usage Example:

escaped = escape_rst("Use * for emphasis and <tags> for HTML")
# Output: "Use \* for emphasis and \<tags\> for HTML"

`project_response_with_refresh(session: CachedSession, name: str, last_serial: int) -> OriginalResponse | CachedResponse`

Fetches the PyPI JSON metadata for a project by name, using the cached session. If the cached response's "last serial" value differs from the provided `last_serial`, forces a refresh to get the latest data.

Parameters:
- session (CachedSession): The HTTP session with caching enabled.
- name (str): PyPI project name.
- last_serial (int): The last known PyPI serial number for the project metadata.
Returns:
A requests_cache response object, either cached or freshly retrieved.
Notes:
Ensures the local cache is updated if PyPI has a newer release or metadata change.

`get_session() -> CachedSession`

Creates and configures a `CachedSession` using `requests-cache` with an SQLite backend, storing cached HTTP responses in the user's cache directory (`~/.cache/pytest-plugin-list/http_cache.sqlite3`).

Returns:
An instance of CachedSession ready for use in HTTP requests.
Implementation Details:
Uses platformdirs.user_cache_path to store the cache in a platform-appropriate location.

`pytest_plugin_projects_from_pypi(session: CachedSession) -> dict[str, int]`

Retrieves all PyPI projects from the simple API endpoint, filtered to those whose names start with `pytest-` or `pytest_` or are in `ADDITIONAL_PROJECTS`. Returns a dictionary mapping project names to their last serial number.

Parameters:
session (CachedSession): Cached HTTP session.
Returns:
Dictionary {project_name: last_serial}.
Notes:
The "last serial" is used to detect changes in project metadata/releases.

TypedDict

`PluginInfo`

A typed dictionary representing the key metadata fields tracked for each plugin.

Field	Type	Description
`name`	str	PyPI project name formatted as an RST link.
`summary`	str	Short description of the project.
`last_release`	str	Date of the most recent release (e.g., "Jan 01, 2024").
`status`	str	Development status extracted from classifiers.
`requires`	str	The required pytest version or requirement string.

Iterator / Generators

`iter_plugins() -> Iterator[PluginInfo]`

Main generator function that:

Creates a cached HTTP session.
Retrieves plugin projects and their last serials.
Iterates over each project, fetching detailed JSON metadata.
Skips 404 responses and inactive projects.
Extracts relevant metadata (status, last release date, requirements).
Yields PluginInfo dictionaries for each valid plugin.

Yields:
Instances of PluginInfo.

Usage Example:

for plugin in iter_plugins():
    print(plugin["name"], plugin["last_release"])

Algorithm Highlights:
- Uses packaging.version.parse to sort releases by version.
- Uses the latest release with available upload date as last_release.
- Filters out inactive plugins by checking PyPI classifiers.
- Looks for pytest in requires_dist to detect dependencies.

Output Generation

`plugin_definitions(plugins: Iterable[PluginInfo]) -> Iterator[str]`

Generates a detailed RST block for each plugin formatted to fit better on vertical (PDF) pages.

Parameters:
plugins (Iterable[PluginInfo]): Iterable of plugin metadata dicts.
Yields:
Multiline strings containing indented RST description blocks.

Example Output Fragment:

:pypi:`pytest-example`
   *last release*: Jan 01, 2024,
   *status*: Production/Stable,
   *requires*: pytest >= 6.0

   Example plugin summary text here.

Main Entrypoint

`main() -> None`

Coordinates the overall workflow:

Calls iter_plugins to collect all plugin metadata.
Determines output directory doc/en/reference.
Writes plugin_list.rst with header, plugin count, and plugin data.
Writes two formats:
- A table (using tabulate) for non-LaTeX outputs.
- Detailed definitions for LaTeX outputs.
Ensures the wcwidth import is referenced for tabulate compatibility.

Usage:
Run directly as a script (python update-plugin-list.py).
Side Effects:
Generates/overwrites doc/en/reference/plugin_list.rst.

Implementation Details and Algorithms

Caching Strategy:
Uses requests-cache with SQLite backend to locally cache HTTP responses, reducing load on PyPI and speeding up repeated runs.
Version Sorting:
Uses packaging.version.parse to robustly handle semantic version sorting, falling back gracefully on invalid versions.
Plugin Filtering:
- Only plugins starting with pytest- or pytest_ or in ADDITIONAL_PROJECTS are considered.
- Plugins marked as "Inactive" (Development Status :: 7 - Inactive) are excluded.
RST Escaping:
Basic escaping to avoid misinterpretation of special characters in plugin summaries.
Output Formatting:
Generates an RST table for web display and a more verbose block for PDF output, using the tabulate package and text wrapping functions.

Interactions with Other System Components

PyPI JSON API:
This script interacts primarily with PyPI's JSON API (https://pypi.org/pypi/{project}/json) and the simple API endpoint (https://pypi.org/simple) to discover and retrieve plugin metadata.
Documentation Build System:
The generated plugin_list.rst file is intended for inclusion in the pytest documentation site under doc/en/reference/.
GitHub Actions / CI:
Typically automated to run weekly via CI workflows to keep the plugin list fresh.
Caching Layer:
The requests-cache SQLite cache reduces the network traffic and speeds up repeated invocations.

File Structure and Workflow Diagram

flowchart TD
    A[Start: main()] --> B[iter_plugins()]
    B --> C[get_session()]
    C --> D[pytest_plugin_projects_from_pypi()]
    D --> E[For each plugin project]
    E --> F[project_response_with_refresh()]
    F --> G{Is status 404?}
    G -- Yes --> E
    G -- No --> H{Is plugin inactive?}
    H -- Yes --> E
    H -- No --> I[Extract metadata: status, summary, requires, last_release]
    I --> J[Yield PluginInfo dict]
    J --> E
    E --> K[Collect all PluginInfo into list]
    K --> L[Generate RST output]
    L --> M[Write plugin_list.rst]
    M --> N[End]

    style B fill:#f9f,stroke:#333,stroke-width:2px
    style L fill:#bbf,stroke:#333,stroke-width:2px

Summary

`update-plugin-list.py` is a specialized utility that automates the discovery and documentation of pytest plugins on PyPI. It efficiently queries, caches, filters, and formats plugin metadata into a maintainable, human-readable RST file for official pytest documentation. Its use of caching, version parsing, and careful output formatting makes it a robust and maintainable tool for keeping the plugin list current.

Appendix: Example Usage

Run the script directly to update the plugin list:

python update-plugin-list.py

This will create or update `doc/en/reference/plugin_list.rst` with the latest plugin data.

End of Documentation for update-plugin-list.py