compat.rs

Overview

The `compat.rs` file provides low-level compatibility utilities related to Python's C API internals, focusing primarily on reference counting and dictionary operations across multiple Python versions and build configurations. It contains constants and functions that help manage "immortal" Python objects (objects with special reference count handling) and exposes several unsafe extern "C" functions from the Python runtime to manipulate Python dictionaries and long integers efficiently.

These utilities are designed to abstract differences between Python versions (e.g., Python 3.12, 3.13, 3.14) and platform bit widths (32-bit vs 64-bit), enabling consistent behavior in a Rust environment interfacing with Python's native runtime (via `pyo3_ffi` bindings).

Detailed Explanation

Constants

The file defines several constants representing special reference count values used internally by CPython to mark "immortal" objects—Python objects that live for the entire program duration and have special reference counting semantics.

• _Py_IMMORTAL_REFCNT_LOCAL

  • Type: u32

  • Description: A constant used when Python is compiled with the GIL disabled (Py_GIL_DISABLED). It holds the maximum u32 value, representing an immortal reference count in this mode.

  • Condition: Defined only when Py_GIL_DISABLED is enabled.

• _Py_IMMORTAL_MINIMUM_REFCNT

  • Type: pyo3_ffi::Py_ssize_t

  • Description: For Python 3.14 on 32-bit platforms, this constant defines the minimum reference count value considered immortal. This is calculated as 1 << 30.

  • Condition: Defined only when targeting Python 3.14 and 32-bit.

• _Py_IMMORTAL_REFCNT

  • Type: pyo3_ffi::Py_ssize_t

  • Description: For Python 3.12 but not 3.14, this constant represents the immortal reference count. For 64-bit systems, it's the maximum 32-bit unsigned int; for 32-bit systems, it's the max value shifted right by 2 (lower 30 bits).

  • Condition: Defined only for Python 3.12 (not 3.14).

Function: _Py_IsImmortal

#[inline(always)]
#[allow(non_snake_case)]
pub(crate) unsafe fn _Py_IsImmortal(op: *mut pyo3_ffi::PyObject) -> core::ffi::c_int

• Purpose:
  Determines whether a given Python object is "immortal," i.e., has a special reference count indicating it should never be deallocated.

• Parameters:

  • op: Raw pointer to a Python object (*mut pyo3_ffi::PyObject).

• Returns:

  • core::ffi::c_int (typically 0 for false, non-zero for true).

• Behavior:
  The function inspects the internal reference count of the Python object, applying different logic depending on:

  • Python version (Py_3_12, Py_3_14)

  • Target pointer width (32-bit vs 64-bit)

  • Whether GIL is disabled (Py_GIL_DISABLED)

• For 64-bit systems (with GIL), the reference count is considered immortal if it is negative (a special hack in CPython).

• For 32-bit systems, the reference count is compared against _Py_IMMORTAL_REFCNT or _Py_IMMORTAL_MINIMUM_REFCNT depending on Python version.

• If GIL is disabled, it reads an atomic ob_ref_local counter and compares it against _Py_IMMORTAL_REFCNT_LOCAL.

• Usage Example:

unsafe {
    let is_immortal = _Py_IsImmortal(some_py_object_ptr);
    if is_immortal != 0 {
        println!("Object is immortal");
    }
}

Unsafe extern "C" function bindings

The file declares several external Python C API functions with conditional compilation based on Python version:

_PyDict_Next

pub fn _PyDict_Next(
    mp: *mut pyo3_ffi::PyObject,
    pos: *mut pyo3_ffi::Py_ssize_t,
    key: *mut *mut pyo3_ffi::PyObject,
    value: *mut *mut pyo3_ffi::PyObject,
    hash: *mut pyo3_ffi::Py_hash_t,
) -> core::ffi::c_int;

• Description:
  Iterates over a Python dictionary, retrieving key-value pairs and optionally the hash.

• Parameters:

  • mp: Pointer to the dictionary object.

  • pos: Pointer to the position index, updated on each call for iteration.

  • key: Output pointer to the key object.

  • value: Output pointer to the value object.

  • hash: Output pointer to the hash of the key (optional).

• Returns:

  • 1 if a key-value pair was found at the current position, 0 if iteration is complete.

• Usage:
  Used for low-level dictionary iteration in Rust code interfacing with Python dicts.

_PyDict_Contains_KnownHash (Python 3.10+)

pub fn _PyDict_Contains_KnownHash(
    op: *mut pyo3_ffi::PyObject,
    key: *mut pyo3_ffi::PyObject,
    hash: pyo3_ffi::Py_hash_t,
) -> core::ffi::c_int;

• Checks if the dictionary op contains key, using the precomputed hash.

_PyDict_SetItem_KnownHash / _PyDict_SetItem_KnownHash_LockHeld

• For Python < 3.13:

pub(crate) fn _PyDict_SetItem_KnownHash(
    mp: *mut pyo3_ffi::PyObject,
    name: *mut pyo3_ffi::PyObject,
    value: *mut pyo3_ffi::PyObject,
    hash: pyo3_ffi::Py_hash_t,
) -> core::ffi::c_int;

• For Python 3.13+:

pub(crate) fn _PyDict_SetItem_KnownHash_LockHeld(
    mp: *mut pyo3_ffi::PyDictObject,
    name: *mut pyo3_ffi::PyObject,
    value: *mut pyo3_ffi::PyObject,
    hash: pyo3_ffi::Py_hash_t,
) -> core::ffi::c_int;

• Inserts or updates an item in the dictionary using a known hash value. The 3.13+ version expects the GIL lock to be held.

_PyLong_AsByteArray

• Python 3.13+ version:

pub(crate) fn _PyLong_AsByteArray(
    v: *mut pyo3_ffi::PyLongObject,
    bytes: *mut core::ffi::c_uchar,
    n: pyo3_ffi::Py_ssize_t,
    little_endian: core::ffi::c_int,
    is_signed: core::ffi::c_int,
    with_exceptions: core::ffi::c_int,
) -> core::ffi::c_int;

• Pre-3.13 version:

pub(crate) fn _PyLong_AsByteArray(
    v: *mut pyo3_ffi::PyLongObject,
    bytes: *mut core::ffi::c_uchar,
    n: pyo3_ffi::Py_ssize_t,
    little_endian: core::ffi::c_int,
    is_signed: core::ffi::c_int,
) -> core::ffi::c_int;

• Converts a Python long integer to a byte array with specified endianness and signedness. The newer version supports an additional flag for exception handling.

Implementation Details and Algorithms

• The _Py_IsImmortal function encapsulates platform- and version-specific logic to check the immortality of Python objects by directly inspecting the object's reference count. The concept of immortal objects is an optimization in CPython to avoid unnecessary reference count operations on objects guaranteed to exist for the interpreter's lifetime.

• Constants like _Py_IMMORTAL_REFCNT are derived from CPython's internal macros and header logic, reflecting how CPython uses certain high reference counts or bit patterns to mark special objects.

• The unsafe extern functions are direct bindings to CPython's internal functions, exposing highly optimized dictionary and long integer operations that can be used when writing Rust extensions or embedding Python.

Interaction with Other System Components

• This file interacts closely with the pyo3_ffi crate, which provides raw FFI bindings to Python's C API.

• It is intended to be used internally by higher-level Rust abstractions that wrap Python objects, dictionaries, and integers, enabling efficient and version-correct behavior without duplicating Python's internal logic.

• The conditional compilation attributes ensure compatibility with different Python versions and build configurations, allowing the project to support multiple Python interpreters safely.

Visual Diagram

flowchart TD
    A[compat.rs] --> B[_Py_IsImmortal]
    A --> C[_PyDict_Next]
    A --> D[_PyDict_Contains_KnownHash]
    A --> E[_PyDict_SetItem_KnownHash / _LockHeld]
    A --> F[_PyLong_AsByteArray]

    subgraph Constants
        G[_Py_IMMORTAL_REFCNT_LOCAL]
        H[_Py_IMMORTAL_MINIMUM_REFCNT]
        I[_Py_IMMORTAL_REFCNT]
    end

    A --> Constants

    B -->|uses| G
    B -->|uses| H
    B -->|uses| I

    B -- Checks refcnt of --> J[PyObject]
    C -- Iterates --> K[PyDictObject]
    D -- Checks contains --> K
    E -- Sets item --> K
    F -- Converts --> L[PyLongObject]

Summary

The `compat.rs` file is a specialized compatibility layer bridging Rust code with Python's internal C API details. It mainly deals with immortal object reference counts and dictionary/long operations, providing version-aware, platform-specific constants and unsafe FFI function declarations. This enables safe and efficient interoperability with Python internals across multiple Python versions and configurations, crucial for projects embedding or extending Python in Rust.