fragment.rs

Overview

The [fragment.rs](/projects/287/67759) file defines a low-level Rust extension module component for Python, specifically implementing a custom Python object type named `Fragment`. This type serves as a lightweight container wrapping a single Python object (`contents`). It is designed for efficient interaction with Python's C API, integrating closely with PyO3's FFI layer to extend Python with Rust-backed objects.

The file focuses on:

• Defining the Fragment struct, representing the Python object in Rust.

• Implementing the Python type object (PyTypeObject) for Fragment, enabling Python to recognize and instantiate this type.

• Providing constructor (tp_new) and destructor (tp_dealloc) functions for lifecycle management.

• Handling conditional compilation for Python versions and the Python Global Interpreter Lock (GIL) usage to optimize threading and reference counting.

This component is likely part of a larger project (possibly `orjson`), where `Fragment` objects hold intermediate or raw JSON fragments or similar data structures for high-performance serialization or manipulation.

Detailed Explanation

Struct: Fragment

#[repr(C)]
pub(crate) struct Fragment {
    // GIL-disabled runtime atomic/thread fields
    #[cfg(Py_GIL_DISABLED)]
    pub ob_tid: usize,
    #[cfg(all(Py_GIL_DISABLED, Py_3_14))]
    pub ob_flags: u16,
    #[cfg(all(Py_GIL_DISABLED, not(Py_3_14)))]
    pub _padding: u16,
    #[cfg(Py_GIL_DISABLED)]
    pub ob_mutex: pyo3_ffi::PyMutex,
    #[cfg(Py_GIL_DISABLED)]
    pub ob_gc_bits: u8,
    #[cfg(Py_GIL_DISABLED)]
    pub ob_ref_local: AtomicU32,
    #[cfg(Py_GIL_DISABLED)]
    pub ob_ref_shared: AtomicIsize,
    // Standard reference counting
    #[cfg(not(Py_GIL_DISABLED))]
    pub ob_refcnt: pyo3_ffi::Py_ssize_t,
    // Pointer to Python type object
    pub ob_type: *mut pyo3_ffi::PyTypeObject,
    // Pointer to contained Python object
    pub contents: *mut pyo3_ffi::PyObject,
}

• Purpose: Represents the Python-level Fragment object in Rust memory layout, compatible with Python's C API.

• Fields:

  • Conditional atomic/threading fields when Python's GIL is disabled to support multi-threaded reference counting.

  • ob_refcnt for classic reference counting when GIL is enabled.

  • ob_type points to the Python type descriptor.

  • contents holds the wrapped Python object.

**Implementation Detail:**

• Conditional compilation (cfg) is used extensively to support variations in Python versions (e.g., Python 3.14) and features (like GIL-disabled mode).

• Uses AtomicU32 and AtomicIsize for thread-safe reference counting in GIL-disabled mode.

• Uses repr(C) to ensure the Rust struct layout matches C struct layout, critical for FFI.

Function: raise_args_exception

#[cold]
#[inline(never)]
#[cfg_attr(feature = "optimize", optimize(size))]
fn raise_args_exception()

• Purpose: Raises a Python TypeError when the Fragment constructor receives incorrect arguments.

• Behavior: Constructs a Python string with the error message "orjson.Fragment() takes exactly 1 positional argument" and sets the Python error state.

• Usage: Called inside orjson_fragment_tp_new when argument validation fails.

Function: orjson_fragment_tp_new

#[unsafe(no_mangle)]
#[cold]
#[cfg_attr(feature = "optimize", optimize(size))]
pub(crate) unsafe extern "C" fn orjson_fragment_tp_new(
    _subtype: *mut PyTypeObject,
    args: *mut PyObject,
    kwds: *mut PyObject,
) -> *mut PyObject

• Purpose: Implements the tp_new slot for the Fragment type, i.e., the constructor called when Python code does Fragment(...).

• Parameters:

  • _subtype: Pointer to the Python type (usually Fragment).

  • args: Tuple of positional arguments.

  • kwds: Dictionary of keyword arguments.

• Returns: Pointer to a new Python object (Fragment) or null_mut() on error.

• Behavior:

  • Validates that exactly 1 positional argument is provided and no keyword arguments.

  • Extracts the single argument as contents.

  • Increments reference count of contents.

  • Allocates and initializes a new Fragment instance with appropriate fields.

  • Returns a raw pointer cast to PyObject.

• Safety: Marked unsafe due to raw pointer and FFI usage.

**Example usage in Python:**

f = orjson.Fragment(some_python_object)

Function: orjson_fragment_dealloc

#[unsafe(no_mangle)]
#[cold]
#[cfg_attr(feature = "optimize", optimize(size))]
pub(crate) unsafe extern "C" fn orjson_fragment_dealloc(object: *mut PyObject)

• Purpose: Implements the tp_dealloc slot for Fragment, called when Python garbage collector frees a Fragment object.

• Parameters:

  • object: Pointer to the Fragment Python object.

• Behavior:

  • Decrements the reference count of the contained contents object.

  • Deallocates the memory allocated for the Fragment struct itself.

• Safety: Uses raw pointer casts and manual memory deallocation.

Constant: FRAGMENT_TP_FLAGS

#[cfg(Py_GIL_DISABLED)]
const FRAGMENT_TP_FLAGS: AtomicCULong = AtomicCULong::new(Py_TPFLAGS_DEFAULT | pyo3_ffi::Py_TPFLAGS_IMMUTABLETYPE);

#[cfg(all(Py_3_10, not(Py_GIL_DISABLED)))]
const FRAGMENT_TP_FLAGS: core::ffi::c_ulong = Py_TPFLAGS_DEFAULT | pyo3_ffi::Py_TPFLAGS_IMMUTABLETYPE;

#[cfg(not(Py_3_10))]
const FRAGMENT_TP_FLAGS: core::ffi::c_ulong = Py_TPFLAGS_DEFAULT;

• Purpose: Defines the type flags for the Fragment Python type.

• Details:

  • Uses Py_TPFLAGS_DEFAULT as base flag.

  • Adds Py_TPFLAGS_IMMUTABLETYPE flag when applicable, indicating the type is immutable.

  • Conditional on Python version and GIL usage.

Function: orjson_fragmenttype_new

#[unsafe(no_mangle)]
#[cold]
#[cfg_attr(feature = "optimize", optimize(size))]
pub(crate) unsafe extern "C" fn orjson_fragmenttype_new() -> *mut PyTypeObject

• Purpose: Creates and initializes the Python type object (PyTypeObject) for Fragment.

• Returns: Pointer to the initialized PyTypeObject.

• Behavior:

  • Allocates a PyTypeObject on the heap with all fields set appropriately.

  • Sets key slots:

    • tp_name to "orjson.Fragment".

    • tp_basicsize to size of Fragment.

    • tp_dealloc to orjson_fragment_dealloc.

    • tp_new to orjson_fragment_tp_new.

    • tp_flags to FRAGMENT_TP_FLAGS.

  • Calls PyType_Ready to finalize the type object.

• Usage: Called during module initialization to register the Fragment type with Python runtime.

Important Implementation Details

• Thread Safety & GIL Handling:
  The file uses conditional compilation (#[cfg(Py_GIL_DISABLED)]) to support both GIL-enabled and GIL-disabled Python builds. In GIL-disabled mode, it uses atomic types (AtomicU32, AtomicIsize) for reference counting to maintain thread safety. This is critical for performance in multi-threaded environments.

• FFI and Memory Management:
  The Fragment struct and related functions interact directly with Python's internal C API through raw pointers, requiring careful manual management of reference counts (Py_INCREF, Py_DECREF) and memory allocation/deallocation.

• Minimal Python Object Implementation:
  The Fragment type is designed to be minimal, holding only a reference to one Python object (contents). It does not implement many Python type slots (e.g., no methods, no attributes, no rich comparisons), indicating a specialized internal use case.

• Performance Optimizations:
  Functions are marked with attributes such as #[cold], #[inline(never)], and conditional #[optimize(size)] to optimize the binary size and runtime performance, especially for error paths.

Interaction with Other System Components

• Integration with orjson or Similar Module:
  The prefix orjson_ in function names suggests this code is part of the orjson Rust module, which provides fast JSON serialization/deserialization for Python. The Fragment type likely encapsulates raw JSON fragments or intermediate data structures used during serialization.

• Use of pyo3_ffi Crate:
  The file relies heavily on pyo3_ffi for Python C API access, making it interoperable with the PyO3 ecosystem. This enables seamless bridging between Rust and Python.

• Interaction with Python Runtime:
  The Fragment type is registered as a new Python type with its own constructor and destructor, allowing Python code to instantiate and use these objects transparently.

• Reference to crate::typeref::FRAGMENT_TYPE:
  The ob_type field is assigned with crate::typeref::FRAGMENT_TYPE, indicating that the actual Python type object pointer is stored or referenced elsewhere in the crate, ensuring type consistency.

Usage Examples

Creating a Fragment Object in Python (conceptual)

import orjson

some_obj = {"key": "value"}

# Wrap a Python object inside a Fragment
fragment = orjson.Fragment(some_obj)

# fragment.contents would internally hold a reference to some_obj

Underlying Rust Construction (conceptual)

// Called when Python executes: Fragment(some_obj)
let py_obj_ptr = ...; // pointer to some_obj
let fragment_ptr = orjson_fragment_tp_new(fragment_type_ptr, args_tuple_ptr, null_mut());

Mermaid Diagram: Class Structure of Fragment

classDiagram
    class Fragment {
        +usize ob_tid
        +u16 ob_flags
        +PyMutex ob_mutex
        +u8 ob_gc_bits
        +AtomicU32 ob_ref_local
        +AtomicIsize ob_ref_shared
        +Py_ssize_t ob_refcnt
        +*PyTypeObject ob_type
        +*PyObject contents
    }

    class PyTypeObject {
        +PyVarObject ob_base
        +*c_char tp_name
        +isize tp_basicsize
        +Option<fn(*mut PyObject)> tp_dealloc
        +Option<fn(*mut PyTypeObject, *mut PyObject, *mut PyObject)> tp_new
        +c_ulong tp_flags
        // ...
    }

    Fragment --> PyTypeObject : ob_type points to
    Fragment --> PyObject : contents points to

Summary

The [fragment.rs](/projects/287/67759) file provides the Rust-side definition and Python binding for a custom Python type `Fragment` used in the `orjson` project or a similar Rust-Python integration context. It carefully manages Python object lifecycle via FFI, supports multi-threaded reference counting in GIL-disabled mode, and exposes minimal but essential Python type slots to integrate seamlessly with Python's runtime.

This file is crucial for representing lightweight, Rust-backed Python objects that encapsulate Python objects internally, enabling efficient serialization or manipulation workflows within the larger application.