test_numpy.py

Overview

`test_numpy.py` is a comprehensive test suite designed to validate the integration and correct serialization of NumPy data types using the `orjson` JSON library. The file primarily ensures that various NumPy arrays and scalar types—across multiple dtypes and dimensions—are serialized correctly to JSON format when using the `orjson.OPT_SERIALIZE_NUMPY` option.

The tests cover:

• Serialization of 1D, 2D, 3D, and higher-dimensional NumPy arrays.

• Serialization of different NumPy scalar types (integers, floats, booleans).

• Serialization of NumPy datetime64 objects with different time units and edge cases.

• Handling of special float values (NaN, ±inf, -0.0).

• Behavior with non-C-contiguous arrays and unsupported dtypes.

• Equivalence validation between serialized NumPy arrays and their Python list representations.

• Endianness checks to ensure arrays with wrong byte order raise errors.

This ensures that `orjson` correctly supports NumPy serialization in all expected edge cases, providing robust JSON interoperability for applications using NumPy data structures.

File Contents and Detailed Explanation

Imports

• sys: Used to check system byte order in endianness tests.

• pytest: Testing framework used for marking and running tests.

• orjson: The JSON serialization library under test.

• numpy: The NumPy module imported from a relative utility.

Function: numpy_default(obj)

def numpy_default(obj):
    if isinstance(obj, numpy.ndarray):
        return obj.tolist()
    raise TypeError

• Purpose: A helper serialization fallback function that converts NumPy arrays to lists when default serialization fails.

• Parameters:

  • obj: The object to serialize.

• Returns: List representation of the NumPy array.

• Raises: TypeError if the object is not a NumPy array.

• Usage: Used in tests for serializing non-C-contiguous arrays by converting them to lists.

Class: TestNumpy

Decorated with `@pytest.mark.skipif(numpy is None, reason="numpy is not installed")` to skip all tests if NumPy is unavailable.

Contains numerous test methods that validate serialization of different NumPy types and structures using `orjson.dumps` with `option=orjson.OPT_SERIALIZE_NUMPY`.

Highlights of test methods:

• test_numpy_array_d1_uintp / test_numpy_array_d1_intp:

  • Test 1D arrays of platform-dependent unsigned and signed integer pointer types.

• test_numpy_array_d1_i64 / test_numpy_array_d1_u64 / test_numpy_array_d1_i8 / ...:

  • Test 1D arrays for int64, uint64, int8, uint8, int32, int16, uint16, uint32, float32, and float16.

• test_numpy_array_f16_roundtrip:

  • Serializes and deserializes a float16 array and checks for equality using numpy.array_equal.

• test_numpy_array_f16_edge / test_numpy_array_f32_edge / test_numpy_array_f64_edge:

  • Test serialization of special float values (inf, -inf, NaN, -0.0, 0.0, π) in different float precisions.

• test_numpy_array_d1_datetime64_years / ... / test_numpy_array_d1_datetime64_picoseconds:

  • Test serialization of numpy.datetime64 arrays with various granularities (year, month, day, hour, minute, second, millisecond, microsecond, nanosecond).

  • picoseconds unit is unsupported and raises TypeError.

• test_numpy_array_d2_i64 / test_numpy_array_d2_f64 / test_numpy_array_d3_i8 / ...:

  • Test multi-dimensional arrays (2D and 3D) of integers and floats.

• test_numpy_array_fortran / test_numpy_array_non_contiguous_message:

  • Tests serialization of Fortran-ordered (non-C contiguous) arrays.

  • Raises serialization error unless default=numpy_default is used.

• test_numpy_array_unsupported_dtype:

  • Tests serialization error on unsupported dtypes like complex single-precision floats.

• test_numpy_array_d1 to test_numpy_array_d4:

  • Tests serialization of arrays with dimensions from 1 to 4.

• test_numpy_array_4_stride:

  • Tests serialization of a 4-dimensional array generated with random values.

• test_numpy_array_dimension_zero:

  • Tests behavior of zero-dimensional arrays and arrays with zero-length dimensions.

• test_numpy_array_dimension_max:

  • Tests serialization of an array with the maximum supported 32 dimensions.

• test_numpy_scalar_*:

  • Tests serialization of scalar NumPy types for every integer and float size.

• test_numpy_bool:

  • Tests serialization of numpy boolean scalars inside dictionaries.

• test_numpy_datetime_*:

  • Tests serialization of single numpy.datetime64 scalars for various time granularities.

• test_numpy_datetime_naive_utc_* and test_numpy_datetime_naive_utc_utc_z_*:

  • Tests that datetime serialization respects the OPT_NAIVE_UTC and OPT_UTC_Z flags for timezone naive UTC and UTC 'Z' suffix.

• test_numpy_datetime_omit_microseconds_*:

  • Tests that microseconds can be omitted via OPT_OMIT_MICROSECONDS.

• test_numpy_datetime_nat:

  • Tests that serialization of numpy.datetime64("NaT") raises an error.

• test_numpy_repeated:

  • Verifies consistent serialization of the same array over multiple calls.

**Usage Example of a test method:**

def test_numpy_array_d1_i8(self):
    arr = numpy.array([-128, 127], numpy.int8)
    serialized = orjson.dumps(arr, option=orjson.OPT_SERIALIZE_NUMPY)
    assert serialized == b"[-128,127]"

Class: TestNumpyEquivalence

Also skipped if NumPy is not installed.

• Purpose: To validate equivalence between serialized NumPy arrays and their list counterparts.

• Contains a private helper method _test which asserts equality of serialized output for a NumPy array and the array converted to a list.

• Tests cover various unsigned and signed integer types and floating-point types.

• One float32 test is marked skipped due to a known conversion precision issue.

Class: NumpyEndianness

• Tests that arrays with byte orders that do not match the system's native byte order raise serialization errors.

• Uses sys.byteorder to determine the wrong byte order ('>' or '<') and creates an array with that.

• Serialization is expected to raise orjson.JSONEncodeError.

Important Implementation Details and Algorithms

• The file leverages orjson's OPT_SERIALIZE_NUMPY option to directly serialize NumPy arrays and scalars.

• Special cases like non-C contiguous arrays raise errors unless converted to lists using a custom default function.

• Serialization of numpy.datetime64 is carefully tested across units and edge cases, with unsupported units raising TypeError.

• Special floating-point values (inf, -inf, nan) are serialized to null in JSON.

• Equivalence tests verify that serialization of NumPy arrays produces the same JSON as serializing their Python list equivalents, ensuring correctness.

• Endianness tests confirm that arrays with incorrect byte order are not silently serialized, avoiding data corruption.

Interaction with Other Parts of the System

• This test file depends on:

  • The numpy utility module which is imported relatively (from .util import numpy).

  • The orjson library which provides fast JSON serialization with NumPy support.

  • pytest for running and marking tests.

• It validates the correctness of orjson's NumPy serialization integration and is part of the test suite ensuring the serialization module's robustness.

• The tests indirectly verify that consuming systems using orjson with NumPy can safely serialize NumPy arrays/scalars without data loss or errors.

• The numpy_default function provides a fallback serialization method and demonstrates how users can extend orjson serialization for complex types.

Visual Diagram

The file mainly consists of three test classes and one utility function. The diagram below shows their structure and relationships.

classDiagram
    class numpy_default {
        +obj: object
        +return: list or TypeError
    }

    class TestNumpy {
        +test_numpy_array_d1_uintp()
        +test_numpy_array_d1_intp()
        +test_numpy_array_d1_i64()
        +test_numpy_array_d1_u64()
        +test_numpy_array_d1_i8()
        +test_numpy_array_d1_u8()
        +test_numpy_array_d1_i32()
        +test_numpy_array_d1_i16()
        +test_numpy_array_d1_u16()
        +test_numpy_array_d1_u32()
        +test_numpy_array_d1_f32()
        +test_numpy_array_d1_f16()
        +test_numpy_array_f16_roundtrip()
        +test_numpy_array_f16_edge()
        +test_numpy_array_f32_edge()
        +test_numpy_array_f64_edge()
        +test_numpy_array_d1_bool()
        +test_numpy_array_d1_datetime64_years()
        +test_numpy_array_d1_datetime64_months()
        +test_numpy_array_d1_datetime64_days()
        +test_numpy_array_d1_datetime64_hours()
        +test_numpy_array_d1_datetime64_minutes()
        +test_numpy_array_d1_datetime64_seconds()
        +test_numpy_array_d1_datetime64_milliseconds()
        +test_numpy_array_d1_datetime64_microseconds()
        +test_numpy_array_d1_datetime64_nanoseconds()
        +test_numpy_array_d1_datetime64_picoseconds()
        +test_numpy_array_d2_i64()
        +test_numpy_array_d2_f64()
        +test_numpy_array_d3_i8()
        +test_numpy_array_d3_u8()
        +test_numpy_array_d3_i32()
        +test_numpy_array_d3_i64()
        +test_numpy_array_d3_f64()
        +test_numpy_array_fortran()
        +test_numpy_array_non_contiguous_message()
        +test_numpy_array_unsupported_dtype()
        +test_numpy_array_d1()
        +test_numpy_array_d2()
        +test_numpy_array_d3()
        +test_numpy_array_d4()
        +test_numpy_array_4_stride()
        +test_numpy_array_dimension_zero()
        +test_numpy_array_dimension_max()
        +test_numpy_scalar_int8()
        +test_numpy_scalar_int16()
        +test_numpy_scalar_int32()
        +test_numpy_scalar_int64()
        +test_numpy_scalar_uint8()
        +test_numpy_scalar_uint16()
        +test_numpy_scalar_uint32()
        +test_numpy_scalar_uint64()
        +test_numpy_scalar_float16()
        +test_numpy_scalar_float32()
        +test_numpy_scalar_float64()
        +test_numpy_bool()
        +test_numpy_datetime_year()
        +test_numpy_datetime_month()
        +test_numpy_datetime_day()
        +test_numpy_datetime_hour()
        +test_numpy_datetime_minute()
        +test_numpy_datetime_second()
        +test_numpy_datetime_milli()
        +test_numpy_datetime_micro()
        +test_numpy_datetime_nano()
        +test_numpy_datetime_naive_utc_year()
        +test_numpy_datetime_naive_utc_month()
        +test_numpy_datetime_naive_utc_day()
        +test_numpy_datetime_naive_utc_hour()
        +test_numpy_datetime_naive_utc_minute()
        +test_numpy_datetime_naive_utc_second()
        +test_numpy_datetime_naive_utc_milli()
        +test_numpy_datetime_naive_utc_micro()
        +test_numpy_datetime_naive_utc_nano()
        +test_numpy_datetime_naive_utc_utc_z_year()
        +test_numpy_datetime_naive_utc_utc_z_month()
        +test_numpy_datetime_naive_utc_utc_z_day()
        +test_numpy_datetime_naive_utc_utc_z_hour()
        +test_numpy_datetime_naive_utc_utc_z_minute()
        +test_numpy_datetime_naive_utc_utc_z_second()
        +test_numpy_datetime_naive_utc_utc_z_milli()
        +test_numpy_datetime_naive_utc_utc_z_micro()
        +test_numpy_datetime_naive_utc_utc_z_nano()
        +test_numpy_datetime_omit_microseconds_year()
        +test_numpy_datetime_omit_microseconds_month()
        +test_numpy_datetime_omit_microseconds_day()
        +test_numpy_datetime_omit_microseconds_hour()
        +test_numpy_datetime_omit_microseconds_minute()
        +test_numpy_datetime_omit_microseconds_second()
        +test_numpy_datetime_omit_microseconds_milli()
        +test_numpy_datetime_omit_microseconds_micro()
        +test_numpy_datetime_omit_microseconds_nano()
        +test_numpy_datetime_nat()
        +test_numpy_repeated()
    }

    class TestNumpyEquivalence {
        -_test(obj)
        +test_numpy_uint8()
        +test_numpy_uint16()
        +test_numpy_uint32()
        +test_numpy_uint64()
        +test_numpy_int8()
        +test_numpy_int16()
        +test_numpy_int32()
        +test_numpy_int64()
        +test_numpy_float32()
        +test_numpy_float64()
    }

    class NumpyEndianness {
        +test_numpy_array_dimension_zero()
    }

    TestNumpy --> numpy_default : uses (for fallback serialization)

Summary

`test_numpy.py` is a rigorous test suite ensuring that `orjson` can serialize NumPy arrays and scalars correctly across a wide range of scenarios, including edge cases with special values, datetime units, array memory layouts, and scalar types. This file is essential for maintaining the reliability and correctness of NumPy serialization support in the `orjson` library and helps prevent regressions or unsupported cases.

If you need further information or examples on specific tests or usage scenarios, please let me know.