Python Memory Allocator

Purpose

This subtopic addresses the challenge of managing memory allocation within the Rust components of the project while maintaining compatibility and integration with Python's memory management system. Specifically, it implements a custom global allocator in Rust that delegates all memory operations—allocation, deallocation, zeroed allocation, and reallocation—to Python's own memory APIs (`PyMem_Malloc`, `PyMem_Free`, `PyMem_Realloc`). This approach ensures that memory used by Rust code is consistent with Python's memory tracking and debugging facilities, reducing risks of memory leaks or corruption when Rust and Python objects interact closely, especially during deserialization.

Functionality

At its core, the subtopic provides a Rust struct implementing the `GlobalAlloc` trait, which overrides the default Rust allocator globally for the crate. The allocator uses unsafe calls into Python's C API (via `pyo3_ffi`) for all memory operations:

• Allocation (alloc): Requests a memory block of the requested size from Python's allocator.

• Deallocation (dealloc): Frees a previously allocated memory block via Python's memory free function.

• Zeroed Allocation (alloc_zeroed): Allocates memory and explicitly zeroes it by writing zeros to the returned pointer.

• Reallocation (realloc): Changes the size of an existing memory block using Python's realloc function.

By defining the allocator as a static global instance annotated with `#[global_allocator]`, all Rust code in the crate automatically uses this allocator without needing explicit memory management calls.

Key Code Interactions

unsafe impl GlobalAlloc for PyMemAllocator {
    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
        pyo3_ffi::PyMem_Malloc(layout.size()).cast::<u8>()
    }

    unsafe fn dealloc(&self, ptr: *mut u8, _layout: Layout) {
        pyo3_ffi::PyMem_Free(ptr.cast::<c_void>())
    }

    unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
        let len = layout.size();
        let ptr = pyo3_ffi::PyMem_Malloc(len).cast::<u8>();
        core::ptr::write_bytes(ptr, 0, len);
        ptr
    }

    unsafe fn realloc(&self, ptr: *mut u8, _layout: Layout, new_size: usize) -> *mut u8 {
        pyo3_ffi::PyMem_Realloc(ptr.cast::<c_void>(), new_size).cast::<u8>()
    }
}

The use of `unsafe` is necessary due to raw pointer manipulation and calls into external C APIs. The allocator is marked `Sync` to satisfy Rust's global allocator requirements, ensuring thread safety when used concurrently.

Integration

This allocator tightly integrates with the parent topic of **Memory and Resource Management** by providing a foundational mechanism for all memory operations within the Rust core of the project. Unlike the **Deserialization Key Caching** subtopic, which optimizes specific data structures, the Python Memory Allocator ensures all allocations conform to Python's memory model, enabling seamless interoperability between Python objects and Rust buffers.

Moreover, this allocator underpins performance and correctness across the entire Rust serialization and deserialization modules (`src/serialize` and `src/deserialize`) by safeguarding memory consistency. It complements Python Integration efforts by aligning Rust's low-level memory management with Python's runtime expectations, preventing memory mismatches that could lead to crashes or leaks.

By delegating memory management to Python's allocator:

• Memory profiling and debugging tools native to Python remain effective.

• Cross-language memory ownership semantics are simplified.

• The project avoids duplicating or conflicting allocators, improving maintainability.

Diagram

flowchart TD
    RustCode[All Rust Code]
    MemoryOps[Memory Operations (alloc, free, realloc)]
    PyMemAPI[Python Memory API (PyMem_Malloc, PyMem_Free, PyMem_Realloc)]

    RustCode -->|uses| MemoryOps
    MemoryOps -->|calls| PyMemAPI

This flowchart illustrates that all memory operations performed by Rust code are routed through the custom allocator which delegates calls to Python's memory API.

By centralizing memory allocation through Python's allocator, this subtopic ensures robust, consistent, and safe memory management critical to the project's seamless Python-Rust integration and high-performance JSON processing.