test_memory.py

Overview

`test_memory.py` is a test suite focused on verifying that the `orjson` library’s JSON serialization and deserialization functions (`loads` and `dumps`) do not cause memory leaks under various scenarios. The tests utilize the `pytest` framework and rely on the `psutil` library to monitor the process’s resident memory size (RSS) before and after repeated operations. By running intensive loops of JSON encoding/decoding and checking memory usage, these tests ensure `orjson` maintains efficient memory management and garbage collection.

The file also includes tests for serialization of complex data types, such as Python dataclasses, numpy arrays, pandas Series, and pytz timezone objects, covering a wide range of typical usage patterns. It uses a constant memory increase threshold to assert no significant leak occurs during orjson operations.

Classes and Functions

Constants and Fixtures

• FIXTURE: str
  A JSON string used as a standard test input for loads/dumps tests. Contains multiple data types including integers, floats, booleans, null, and unicode characters.

• MAX_INCREASE: int
  Set to 4 MiB (4194304 bytes), this is the maximum allowed increase in memory usage after running memory-intensive tests, used to detect memory leaks.

• DATACLASS_FIXTURE: list[Object]
  A list of 1000 Object dataclass instances, with randomized timestamps and members, used to test serialization of dataclasses.

Function: default(obj) -> str

• Purpose:
  A fallback serialization function used by orjson.dumps for unsupported types, converting the object to its string representation.

• Parameters:

  • obj: Any Python object that is not natively serializable by orjson.

• Returns:

  • str: The string representation of the object.

• Usage Example:

  orjson.dumps(data, default=default)
  

Dataclasses

Member

• Description:
  Represents a member entity with an id and an active status.

• Fields:

  • id: int — Unique identifier.

  • active: bool — Status flag.

Object

• Description:
  Represents an object with an ID, a timestamp, a name, and a list of members.

• Fields:

  • id: int — Unique identifier.

  • updated_at: datetime.datetime — Timestamp of last update.

  • name: str — Name string.

  • members: list[Member] — List of Member instances.

Class: Unsupported

• Description:
  Empty class used to test serialization failure in dumps() when the object type is unsupported.

Class: TestMemory

This class contains multiple pytest test methods to check memory behavior of `orjson` under various conditions. All tests that require `psutil` are skipped if the library is unavailable.

Methods:

test_memory_loads(self)

• Purpose:
  Tests for memory leaks when repeatedly deserializing the same JSON string using orjson.loads().

• Behavior:
  Loads the JSON string 10,000 times and asserts memory usage does not increase beyond threshold.

test_memory_loads_memoryview(self)

• Purpose:
  Similar to test_memory_loads, but uses a memoryview over the encoded JSON bytes to test memory usage in this input form.

test_memory_dumps(self)

• Purpose:
  Tests for memory leaks when repeatedly serializing a Python object using orjson.dumps().

test_memory_loads_exc(self)

• Purpose:
  Tests memory behavior when orjson.loads() raises JSONDecodeError exceptions repeatedly without garbage collection pauses.

• Details:
  Garbage collection is disabled to simulate stress conditions on memory.

test_memory_dumps_exc(self)

• Purpose:
  Tests memory behavior when orjson.dumps() raises JSONEncodeError exceptions repeatedly without garbage collection pauses, using an unsupported object.

test_memory_dumps_default(self)

• Purpose:
  Tests memory leaks when using the default serialization function to handle custom object serialization.

test_memory_dumps_dataclass(self)

• Purpose:
  Tests serialization of a large list of dataclass instances and checks for memory leaks.

test_memory_dumps_pytz_tzinfo(self)

• Purpose:
  Tests serialization of pytz timezone-aware datetime objects for memory leaks.

• Skip Conditions:
  Skipped if psutil or pytz are not installed.

test_memory_loads_keys(self)

• Purpose:
  Tests deserialization memory behavior with JSON objects containing a large number (1024) of keys, which could cause internal cache eviction.

test_memory_dumps_numpy(self)

• Purpose:
  Tests serialization of numpy arrays using orjson with the OPT_SERIALIZE_NUMPY option.

• Skip Conditions:
  Skipped if psutil or numpy are not installed.

test_memory_dumps_pandas(self)

• Purpose:
  Tests serialization of pandas Series objects containing numpy arrays for memory leaks.

• Skip Conditions:
  Skipped if psutil or pandas are not installed.

test_memory_dumps_fragment(self)

• Purpose:
  Tests serialization of orjson.Fragment objects repeatedly to check for memory leaks.

Important Implementation Details

• Memory Measurement:
  Memory usage is checked using psutil.Process().memory_info().rss, which returns the resident set size (physical memory in use by the process).

• Garbage Collection Control:
  In some tests, Python's garbage collector is explicitly disabled/enabled (gc.disable(), gc.enable()) to test memory behavior under stress without GC pauses.

• Memory Leak Threshold:
  The tests assert that the memory increase after multiple iterations is less than or equal to 4 MiB (MAX_INCREASE), which is considered an acceptable buffer.

• Conditional Skipping:
  Tests that require optional dependencies (psutil, numpy, pandas, pytz) are skipped if those modules are not installed, ensuring compatibility across environments.

• Use of Dataclasses:
  Complex nested structures like lists of dataclasses are serialized to test orjson’s handling of Python objects beyond primitives.

Interaction with Other Parts of the System

• orjson Library:
  This test file is tightly coupled with orjson’s API, testing its JSON serialization/deserialization functions and error handling.

• psutil:
  Used for process memory monitoring, critical for detecting leaks.

• pytest:
  Provides the testing framework, including decorators for skipping tests.

• numpy, pandas, pytz:
  Optional dependencies tested if available, to validate orjson’s support for serialization of these types.

• gc Module:
  Used to control garbage collection to simulate different runtime memory conditions.

• Local .util Module:
  Imports numpy and pandas from a local utility module, likely providing optional wrappers or safe imports.

Usage Example

Run all tests with pytest, ensuring `psutil` and optionally other dependencies are installed:

pip install pytest psutil orjson numpy pandas pytz
pytest test_memory.py

The test results will confirm that `orjson` does not leak memory under heavy serialization/deserialization workloads.

Mermaid Class Diagram

classDiagram
    class Member {
        +int id
        +bool active
    }
    class Object {
        +int id
        +datetime.datetime updated_at
        +str name
        +list~Member~ members
    }
    class Unsupported

    class TestMemory {
        +test_memory_loads()
        +test_memory_loads_memoryview()
        +test_memory_dumps()
        +test_memory_loads_exc()
        +test_memory_dumps_exc()
        +test_memory_dumps_default()
        +test_memory_dumps_dataclass()
        +test_memory_dumps_pytz_tzinfo()
        +test_memory_loads_keys()
        +test_memory_dumps_numpy()
        +test_memory_dumps_pandas()
        +test_memory_dumps_fragment()
    }

    Object o-- Member : contains

Summary

`test_memory.py` is a comprehensive test suite that systematically exercises `orjson`’s serialization and deserialization functions under different data scenarios and monitors memory usage to detect leaks. It leverages system monitoring tools and Python’s garbage collection controls to create rigorous memory tests, ensuring robustness of `orjson` for high-performance JSON processing in Python applications.