byteswriter.rs

Overview

`byteswriter.rs` defines a specialized utility for efficiently constructing and manipulating Python byte string objects (`PyBytesObject`) from Rust. At its core is the `BytesWriter` struct, which provides a mutable buffer interface tailored to grow a Python bytes object dynamically, supporting incremental writes with minimal overhead.

This file acts as a bridge between Rust's performant memory operations and Python's byte string API, leveraging unsafe Rust code and Python C API functions through FFI to allocate, resize, and finalize Python bytes buffers. It implements the `BufMut` trait from the `bytes` crate, allowing seamless writing of bytes and slices with direct memory access.

The utility is designed for internal use (`pub(crate)`) within a larger system that likely involves serialization, JSON processing, or Python interop where mutable, dynamically sized byte buffers are required.

Detailed Documentation

Constants

• BUFFER_LENGTH: usize = 1024

  Initial capacity (in bytes) allocated for the underlying Python bytes buffer. The buffer will grow dynamically as needed.

Struct: BytesWriter

A mutable writer for Python bytes objects (`PyBytesObject`) that supports efficient byte appending and resizing.

Fields

Associated Functions and Methods

default() -> Self

Creates a new `BytesWriter` with an initial capacity of `BUFFER_LENGTH`.

• Allocates a new Python bytes object of size 1024 (or BUFFER_LENGTH).

• The internal length is set to 0.

• Returns a fully initialized BytesWriter.

**Usage example:**

let mut writer = BytesWriter::default();

bytes_ptr(&mut self) -> NonNull<PyObject>

Returns a non-null raw pointer to the underlying Python object (`PyObject`).

• Useful for passing the Python bytes object back to Python or other FFI consumers.

• The pointer is safe to use as it is guaranteed non-null.

**Usage example:**

let py_obj_ptr = writer.bytes_ptr();
// Pass py_obj_ptr to Python API or other code

finish(&mut self, append: bool) -> NonNull<PyObject>

Finalizes the bytes buffer and returns the Python bytes object pointer.

• If append is true, appends a newline character (b'\n') before finalizing.

• Writes a null terminator (0) at the end of the buffer for safety.

• Updates the Python object size field (ob_size) to the current length.

• Resizes the Python bytes object buffer to match the final length.

• Returns the PyObject pointer for further use.

**Parameters:**

• append: Whether to append a newline byte before finishing.

**Returns:**

• NonNull<PyObject>: Pointer to the finalized Python bytes object.

**Usage example:**

let py_bytes = writer.finish(true);
// py_bytes now contains the bytes with a trailing newline

buffer_ptr(&self) -> *mut u8

Returns a raw mutable pointer to the next writable location in the internal buffer.

• Points to the byte at offset len inside the bytes buffer.

• Unsafe method, used internally for direct memory operations.

resize(&mut self, len: usize)

Resizes the Python bytes object buffer to the specified length.

• Updates internal capacity cap to the new length.

• Calls Python C API function _PyBytes_Resize to change the buffer size.

• May reallocate the buffer internally.

**Parameters:**

• len: New desired length of the Python bytes buffer.

grow(&mut self, len: usize)

Grows the buffer capacity to accommodate at least `len` bytes.

• Doubles the capacity repeatedly until it is large enough.

• Calls resize to apply the new capacity.

• Marked as #[cold] and #[inline(never)] to optimize for rare usage.

Trait Implementations

unsafe impl BufMut for BytesWriter

Implements the `BufMut` trait from the `bytes` crate, enabling writing bytes into the buffer.

Methods:

• unsafe fn advance_mut(&mut self, cnt: usize)

  Advances the internal length by cnt bytes after writing.

• fn chunk_mut(&mut self) -> &mut UninitSlice

  Returns a mutable slice of uninitialized memory representing the writable chunk.

• fn remaining_mut(&self) -> usize

  Returns the number of remaining bytes available for writing.

• fn put_u8(&mut self, value: u8)

  Writes a single byte into the buffer.

• fn put_bytes(&mut self, val: u8, cnt: usize)

  Writes cnt repeated bytes of val into the buffer.

• fn put_slice(&mut self, src: &[u8])

  Copies a byte slice src into the buffer.

**Note:** All methods assume that there is sufficient capacity in the buffer; otherwise, buffer growth must be managed externally.

Trait: WriteExt

A helper trait to extend writing capabilities with buffer pointer access and reservation.

• Provides default no-op implementations for as_mut_buffer_ptr and reserve.

Implementation: WriteExt for &mut BytesWriter

Extends `BytesWriter` with:

• as_mut_buffer_ptr() -> *mut u8

  Returns a raw mutable pointer to the current write location in the buffer.

• reserve(len: usize)

  Ensures there is capacity for at least len additional bytes.

  • If capacity is insufficient, calls grow to increase buffer size.

Important Implementation Details

• Unsafe code usage: The file heavily uses unsafe Rust features for raw pointer manipulation and FFI calls with the Python C API.

• Python bytes object management: The buffer is a Python bytes object, manipulated via the Python C API functions such as PyBytes_FromStringAndSize and _PyBytes_Resize. This allows the Rust code to create and resize Python bytes objects directly.

• Buffer growth strategy: The buffer doubles its capacity when more space is needed, a common amortized growth approach to avoid frequent reallocations.

• Null terminator: A null byte (0) is appended at the end when finishing, which is typical for C strings but here may serve as a safety or compatibility measure.

• BufMut trait integration: Implementing BufMut allows this writer to be used with libraries expecting the bytes crate's buffer writers, facilitating interoperability.

• Append newline option: The finish method optionally appends a newline (b'\n') before finalizing, which may be useful in certain serialization or text protocols.

• Use of helper function usize_to_isize: Converts usize to isize safely for API calls expecting signed sizes.

Interaction with Other System Components

• Python Interop: This file is part of a Rust module interfacing with Python, enabling efficient construction and manipulation of Python bytes objects from Rust code.

• crate::util::usize_to_isize: A utility function used for safe type conversion when calling Python C API functions.

• bytes crate: Uses BufMut and UninitSlice from the bytes crate to provide a standard Rust interface for buffer mutation.

• pyo3_ffi: Uses raw FFI bindings to the Python C API for bytes object creation and resizing.

• Likely used in serialization or JSON libraries: The hack and patch mentioned in the comments reference serde-rs/json issue, suggesting that BytesWriter is used for efficient JSON serialization to Python bytes.

Visual Diagram

classDiagram
    class BytesWriter {
        - cap: usize
        - len: usize
        - bytes: *mut PyBytesObject
        + default() BytesWriter
        + bytes_ptr() NonNull<PyObject>
        + finish(append: bool) NonNull<PyObject>
        - buffer_ptr() *mut u8
        + resize(len: usize)
        - grow(len: usize)
    }

    class WriteExt {
        <<trait>>
        + as_mut_buffer_ptr() *mut u8
        + reserve(len: usize)
    }

    WriteExt <|.. "&mut BytesWriter"
    BytesWriter ..> PyBytesObject : raw pointer

Summary

`byteswriter.rs` provides a performant, low-level Rust abstraction for building Python `bytes` objects incrementally. It combines direct memory manipulation, Python C API interaction, and Rust `BufMut` trait compatibility to enable efficient byte buffer writing in Python interop scenarios. Its design emphasizes minimal allocations, dynamic growth, and safe exposure of Python bytes objects to the rest of the system.