check-pypi

Overview

`check-pypi` is a Python utility script intended to verify the presence and correctness of Python package distribution files (source distribution and wheels) after a build process, specifically for the `orjson` project. It checks that the release artifacts in a specified directory (`dist`) match the expected filenames based on the project's version and supported Python ABIs and platform tags.

This script ensures that:

• The source distribution (.tar.gz) file is present.

• All expected wheel (.whl) files are present, no wheels are missing, and no unexpected wheels exist.

If any expected files are missing or unexpected files are found, the script exits with a non-zero status, making it suitable for use as a CI/CD pipeline validation step to prevent incomplete or incorrect package uploads to PyPI.

Detailed Explanation

Script Execution and Input

• The script is designed to be executed from the command line.

• It expects one positional argument: the path to the distribution directory (dist) containing the built .tar.gz and .whl files.

Example command:

./check-pypi dist

Where `dist` is the folder containing build artifacts.

Key Functional Sections

Loading Project Version

pyproject_doc = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))
pyproject_version = pyproject_doc["project"]["version"]

• Reads the pyproject.toml file to extract the current project version.

• Uses the standard library tomllib (available in Python 3.11+) for TOML parsing.

• The version is used to construct the expected distribution filenames.

Defining Supported ABIs and Platform Tags

abis = (
    "cp39-cp39",
    "cp310-cp310",
    "cp311-cp311",
    "cp312-cp312",
    "cp313-cp313",
)

per_abi_tags = (
    "macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2",
    "manylinux_2_17_aarch64.manylinux2014_aarch64",
    ... # truncated for brevity
    "win_amd64",
)

• abis lists Python ABI tags supported by the built wheels (e.g., cp310-cp310 for CPython 3.10).

• per_abi_tags lists platform compatibility tags representing different OS and architecture targets.

• The cross-product of abis and per_abi_tags generates the bulk of expected wheel filenames.

Constructing Expected Wheel Filenames

prefix = f"orjson-{pyproject_version}"

wheels_matrix = set()

for abi in abis:
    for tag in per_abi_tags:
        wheels_matrix.add(f"{prefix}-{abi}-{tag}.whl")

• Uses nested loops to produce a complete list of wheel filenames expected for all ABI and platform combinations.

• These filenames follow the pattern:

  orjson-<version>-<abi>-<platform>.whl
  

Handling Prerelease and Unique Wheels

wheels_prerelease = {
    f"{prefix}-cp314-cp314-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl",
    ...
}

wheels_unique = {
    f"{prefix}-cp311-cp311-macosx_15_0_arm64.whl",
    ...
}

• wheels_prerelease contains wheels for a prerelease Python version (CPython 3.14).

• wheels_unique contains wheels with unique platform tags not covered by the main matrix.

• These two sets are combined with wheels_matrix to form the full expected set:

wheels_expected = wheels_matrix | wheels_unique | wheels_prerelease

Gathering Actual Distribution Files Found

wheels_queued = set(
    str(each).replace(f"{dist}/", "") for each in Path(dist).glob("*.whl")
)

• Scans the dist directory for .whl files.

• Strips the directory prefix from filenames for comparison.

Checking for Source Distribution Presence

sdist_path = Path(f"{dist}/{prefix}.tar.gz")
if sdist_path.exists():
    print("sdist present\n")
else:
    exit_code = 1
    print(f"Missing sdist:\n{sdist_path}\n")

• Verifies that the source distribution archive exists.

Comparing Expected vs Actual Wheels and Reporting

if wheels_expected == wheels_queued:
    print(f"Wheels as expected, {len(wheels_queued)} total\n")
else:
    exit_code = 1
    missing = "\n".join(sorted(wheels_expected - wheels_queued))
    additional = "\n".join(sorted(wheels_queued - wheels_expected))

    if missing:
        print(f"Missing wheels:\n{missing}\n")
    if additional:
        print(f"Unexpected wheels:\n{additional}\n")

• If the sets match exactly, a success message is printed.

• Otherwise, missing and unexpected wheels are reported.

• The script exits with exit_code 1 on error, 0 on success.

Important Implementation Details / Algorithms

• Set operations are heavily used to compare expected and actual wheels efficiently.

• File paths are normalized by stripping the distribution directory prefix to ensure consistent comparison.

• The script dynamically reads the project version to avoid hardcoding release versions.

• Wheel file naming follows PEP 425 / 600 compatibility tags conventions, combining Python ABI and platform tags.

Usage Example

Given a build directory `dist` containing the files:

orjson-3.10.15.tar.gz
orjson-3.10.15-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
orjson-3.10.15-cp311-cp311-macosx_15_0_arm64.whl
...

Run:

./check-pypi dist

Output will confirm if the source distribution and wheels are all present or report missing/unexpected files.

Interaction with Other System Components

• Reads pyproject.toml in the project root to get the current version.

• Verifies files in the distribution directory produced by the build process (e.g., python -m build or setup.py bdist_wheel).

• Intended to be part of the release pipeline, ensuring only complete and valid artifacts are uploaded to PyPI.

• Could be integrated into CI workflows (GitHub Actions, Jenkins) to gate releases.

Mermaid Diagram: Flowchart of Main Script Logic

flowchart TD
    A[Start: pass dist directory as argument] --> B[Read pyproject.toml version]
    B --> C[Define ABI and platform tags]
    C --> D[Generate expected wheel filenames set]
    D --> E[Scan dist directory for actual wheel files]
    E --> F[Check if source distribution (.tar.gz) exists]
    F --> G{Is sdist present?}
    G -- Yes --> H{Do expected wheels == actual wheels?}
    G -- No --> I[Print missing sdist, set exit_code=1]
    H -- Yes --> J[Print success message]
    H -- No --> K[Print missing and unexpected wheels, set exit_code=1]
    J --> L[Exit with code exit_code]
    I --> L
    K --> L

Summary

The `check-pypi` script is a crucial validation tool ensuring that all expected distribution artifacts for the `orjson` Python package are present before release. It uses version information from `pyproject.toml` and predefined ABI and platform tags to determine the full set of expected wheels and the source archive, comparing this with actual files found in the target distribution directory. This helps maintain release integrity and prevents incomplete uploads to PyPI.