util.rs

Overview

`util.rs` is a utility module in Rust designed to provide low-level macros and helper functions that facilitate safer and more ergonomic interaction with Python's C API via the `pyo3_ffi` bindings. The file primarily defines macros to simplify common operations such as type checking, reference counting, dictionary manipulation, and string handling in the context of Python objects. It also includes small utility functions for integer conversions with debug assertions.

This file serves as a foundational layer that abstracts and encapsulates unsafe operations and repetitive boilerplate patterns, ensuring consistency and reducing errors across the larger PyO3 codebase or any Rust code interfacing with Python internals.

Detailed Descriptions

Macros

Macros in this file are designed to encapsulate unsafe or complex operations on Python objects and types.

`is_type!`

Purpose: Checks if two raw pointers to Python objects/types are equal.
Signature: is_type!($obj_ptr:expr, $type_ptr:expr) -> bool

Usage:

if is_type!(obj_ptr, type_ptr) {
    // obj_ptr points to an object of type type_ptr
}

Implementation Detail: Performs pointer equality check inside an unsafe block.

`ob_type!`

Purpose: Retrieves the type pointer (ob_type) from a raw Python object pointer.
Signature: ob_type!($obj:expr) -> *mut pyo3_ffi::PyTypeObject
Usage:
```
let obj_type = ob_type!(obj_ptr);
```
Implementation Detail: Dereferences the raw pointer unsafely to access the ob_type field.

`is_class_by_type!`

Purpose: Checks if a given object type pointer equals a specified type pointer.
Signature: is_class_by_type!($ob_type:expr, $type_ptr:ident) -> bool

Usage:

if is_class_by_type!(obj_type, SOME_TYPE_PTR) {
    // do something
}

Implementation Detail: Unsafe pointer equality comparison.

`tp_flags!`

Purpose: Retrieves the tp_flags field from a Python type object pointer, handling differences in GIL state.
Signature: tp_flags!($ob_type:expr) -> usize
Conditional Compilation:
- If GIL is enabled (default), reads tp_flags directly.
- If GIL is disabled, loads tp_flags atomically with relaxed ordering.
Usage:
```
let flags = tp_flags!(type_ptr);
```
Implementation Detail: Uses unsafe dereferencing or atomic load depending on configuration.

`is_subclass_by_flag!`

Purpose: Checks if a type's flags contain a specific subclass flag.
Signature: is_subclass_by_flag!($tp_flags:expr, $flag:ident) -> bool

Usage:

if is_subclass_by_flag!(flags, Py_TPFLAGS_BASETYPE) {
    // is a base type subclass
}

Implementation Detail: Bitwise AND between flags and the pyo3_ffi flag constant.

`is_subclass_by_type!`

Purpose: Checks if a Python type object is a subclass of another by comparing the base type of its base object type.
Signature: is_subclass_by_type!($ob_type:expr, $type:ident) -> bool

Usage:

if is_subclass_by_type!(obj_type, BASE_TYPE_PTR) {
    // is subclass
}

Implementation Detail: Uses unsafe casting and pointer dereferencing to compare type hierarchies.

`err!`

Purpose: Shortcut to return a serde::ser::Error with a custom message.
Signature: err!($msg:expr) -> Err
Usage:
```
err!("An error occurred");
```
Implementation Detail: Returns early with a Serde serialization error.

`opt_enabled!` and `opt_disabled!`

Purpose: Check if a particular bitflag is enabled or disabled in a flags variable.
Signature:
- opt_enabled!($var:expr, $flag:expr) -> bool
- opt_disabled!($var:expr, $flag:expr) -> bool

Usage:

if opt_enabled!(options, FLAG_OPTION) {
    // option enabled
}

Implementation Detail: Bitwise AND and comparison with zero.

`unlikely!` and `likely!`

Purpose: Hints to the compiler about the likeliness of a branch to optimize performance.
Signature: unlikely!($exp:expr) -> bool, likely!($exp:expr) -> bool
Conditional Compilation: Uses core::intrinsics if the intrinsics feature is enabled; otherwise, passes through the expression.

Usage:

if unlikely!(some_condition) {
    // unlikely branch
}

Implementation Detail: Compiler hint intrinsics or noop.

`nonnull!`

Purpose: Converts a raw pointer to a non-null pointer wrapper without checking.
Signature: nonnull!($exp:expr) -> core::ptr::NonNull<T>
Usage:
```
let nn_ptr = nonnull!(raw_ptr);
```
Implementation Detail: Uses NonNull::new_unchecked inside unsafe.

`str_from_slice!`

Purpose: Creates a string slice from a raw pointer and size, assuming valid UTF-8.
Signature: str_from_slice!($ptr:expr, $size:expr) -> &str
Usage:
```
let s = str_from_slice!(ptr, length);
```
Implementation Detail: Uses from_utf8_unchecked on a slice created from raw parts.

Reference Counting Macros (`reverse_pydict_incref!`, `ffi!`, `call_method!`, etc.)

These macros wrap calls to Python C API functions (pyo3_ffi) to manipulate reference counts or call methods safely and ergonomically.
They account for differences in Python versions and GIL state with conditional compilation.
Examples:
- reverse_pydict_incref!($op) decrements refcount in a way that respects Python version and GIL status.
- ffi!($fn(...)) calls a pyo3_ffi function with 0-4 arguments.
- call_method!($obj1, $obj2) calls a Python object's method with zero or one argument.

`str_hash!`

Purpose: Retrieves the cached hash value of a Python ASCII string object.
Signature: str_hash!($op:expr) -> isize
Usage:
```
let hash = str_hash!(py_string_ptr);
```
Implementation Detail: Casts the pointer to PyASCIIObject and accesses the hash field.

Dictionary Macros (`pydict_contains!`, `pydict_next!`, `pydict_setitem!`)

These macros abstract dictionary operations with behavior adapted for different Python versions.
Examples:
- pydict_contains!(dict, key) checks if a dictionary contains a key.
- pydict_next!(dict, pos, key, value) iterates over dictionary items.
- pydict_setitem!(dict, key, value) inserts a key-value pair, asserting reference counts and hash validity.

`reserve_minimum!` and `reserve_pretty!`

Purpose: Reserve capacity in a writer/buffer to optimize allocations before writing.

Usage:

reserve_minimum!(writer);
reserve_pretty!(writer, val);

`assume!`

Purpose: Assert a condition in debug mode and hint the compiler to assume it unchecked in release.
Usage:
```
assume!(some_condition);
```

`unreachable_unchecked!`

Purpose: Marks code as unreachable to the compiler without overhead.
Usage:
```
unreachable_unchecked!();
```

Functions

`usize_to_isize`

Signature: pub(crate) fn usize_to_isize(val: usize) -> isize
Description: Converts a usize to isize with a debug assertion that the value fits within isize::MAX.
Parameters:
- val: The usize value to convert.
Returns: The value as isize.
Usage Example:
```
let signed = usize_to_isize(42_usize);
```
Implementation Details: Uses a debug assertion to prevent overflow in debug builds.

`isize_to_usize`

Signature: pub(crate) fn isize_to_usize(val: isize) -> usize
Description: Converts an isize to usize with a debug assertion that the value is non-negative.
Parameters:
- val: The isize value to convert.
Returns: The value as usize.

Usage Example:

let unsigned = isize_to_usize(42_isize);

Implementation Details: Uses a debug assertion to prevent negative values in debug builds.

Important Implementation Details

Unsafe Code Usage: Many macros use unsafe code blocks to directly dereference raw pointers or call C API functions. This is necessary due to the low-level nature of Python C API interaction.
Conditional Compilation: The file handles multiple Python versions and build configurations (e.g., GIL enabled/disabled) using Rust's #[cfg] attributes, ensuring compatibility and correctness across environments.
Reference Counting: The macros carefully handle Python reference counts, including special cases for 'immortal' objects and different Python versions, to avoid reference leaks or premature deallocation.
Performance Optimizations: The likely! and unlikely! macros provide branch prediction hints to the compiler, improving runtime performance where applicable.
Error Handling: The err! macro provides a concise way to return Serde serialization errors, facilitating error propagation in serialization contexts.

Interaction with Other Parts of the System

This file is a low-level utility module that supports higher-level Rust wrappers around Python objects and types.
It interacts closely with the pyo3_ffi crate, which provides raw FFI bindings to the Python C API.
Higher-level PyO3 modules rely on these macros and functions to implement safe Rust abstractions over Python internals.
The macros and functions are designed to be pub(crate), limiting their visibility to the crate, indicating they are internal helper utilities rather than part of the public API.
Macros like pydict_setitem! and call_method! are fundamental building blocks for implementing Python object protocols and behaviors in Rust.
The conditional compilation accommodates extensions or features (e.g., intrinsics) and Python interpreter variations, ensuring seamless integration with the Python runtime environment used by the application.

Visual Diagram

The following flowchart illustrates the relationships and workflow between the main macro categories and functions in `util.rs`. It shows how the macros abstract core Python C API operations and utility conversions, which are then used by higher-level Rust code.

flowchart TD
    A[Python C API (pyo3_ffi)] --> B[ffi! macro]
    B --> C[Reference Counting Macros]
    B --> D[Dictionary Macros]
    B --> E[Method Call Macros]
    B --> F[Type & Flag Macros]

    F --> G[is_type!]
    F --> H[ob_type!]
    F --> I[tp_flags!]
    F --> J[is_subclass_by_flag!]
    F --> K[is_subclass_by_type!]

    C --> L[reverse_pydict_incref!]
    C --> M[str_hash!]

    D --> N[pydict_contains!]
    D --> O[pydict_next!]
    D --> P[pydict_setitem!]

    E --> Q[call_method!]

    R[Utility Macros] --> S[opt_enabled! / opt_disabled!]
    R --> T[likely! / unlikely!]
    R --> U[nonnull!]
    R --> V[str_from_slice!]
    R --> W[assume! / unreachable_unchecked!]

    X[Helper Functions] --> Y[usize_to_isize()]
    X --> Z[isize_to_usize()]

    style A fill:#f9f,stroke:#333,stroke-width:2px
    style B fill:#bbf,stroke:#333
    style C fill:#bfb,stroke:#333
    style D fill:#fbf,stroke:#333
    style E fill:#ffb,stroke:#333
    style F fill:#fbb,stroke:#333
    style R fill:#aff,stroke:#333
    style X fill:#ffa,stroke:#333

Summary

util.rs is a critical internal utility module providing macros and functions that encapsulate unsafe operations with Python C API objects.
It supports type checking, reference counting, dictionary operations, and optimizations with careful attention to Python version and GIL state differences.
The macros enable safer and clearer Rust code interfacing with Python internals by abstracting repetitive and unsafe patterns.
The two small utility functions provide safe conversions between signed and unsigned integer types with debug-time correctness checks.
The file interacts closely with pyo3_ffi and is foundational for the PyO3 project's internal implementation.

**End of documentation for `util.rs`**

util.rs

Overview

Detailed Descriptions

Macros

is_type!

ob_type!

is_class_by_type!

tp_flags!

is_subclass_by_flag!

is_subclass_by_type!

err!

opt_enabled! and opt_disabled!

unlikely! and likely!

nonnull!

str_from_slice!

Reference Counting Macros (reverse_pydict_incref!, ffi!, call_method!, etc.)

str_hash!

Dictionary Macros (pydict_contains!, pydict_next!, pydict_setitem!)

reserve_minimum! and reserve_pretty!

assume!

unreachable_unchecked!

Functions

usize_to_isize

isize_to_usize

Important Implementation Details

Interaction with Other Parts of the System

Visual Diagram

Summary

`is_type!`

`ob_type!`

`is_class_by_type!`

`tp_flags!`

`is_subclass_by_flag!`

`is_subclass_by_type!`

`err!`

`opt_enabled!` and `opt_disabled!`

`unlikely!` and `likely!`

`nonnull!`

`str_from_slice!`

Reference Counting Macros (`reverse_pydict_incref!`, `ffi!`, `call_method!`, etc.)

`str_hash!`

Dictionary Macros (`pydict_contains!`, `pydict_next!`, `pydict_setitem!`)

`reserve_minimum!` and `reserve_pretty!`

`assume!`

`unreachable_unchecked!`

`usize_to_isize`

`isize_to_usize`