util.py

Overview

The [util.py](/projects/287/67676) file serves as a utility module primarily focused on handling test fixture data for the project. It provides functions to read fixture files from disk in various formats (raw bytes, decoded strings, and deserialized JSON objects), caching the results to optimize repeated access during testing. Additionally, it includes conditional imports to support optional dependencies (`numpy` and `pandas`) depending on Python interpreter configuration, and a Pytest marker for tests that require fixture data.

This file plays a supportive role in the testing infrastructure by simplifying fixture management, improving test performance through caching, and ensuring compatibility with different Python runtime settings.

Detailed Description of Functions and Variables

Global Variables

• IS_FREETHREADING
  Type: bool
  Description: Indicates whether the Python interpreter is running with the Global Interpreter Lock (GIL) disabled (free-threading mode). Retrieved from the Python configuration variable "Py_GIL_DISABLED".
  Usage: Controls conditional importing of potentially non-thread-safe libraries like numpy and pandas.

• numpy
  Type: module or None
  Description: Attempts to import the numpy module if the interpreter is not in free-threading mode. Otherwise, remains None.
  Usage: Allows optional usage of numpy if available and compatible.

• pandas
  Type: module or None
  Description: Similar to numpy, attempts to import pandas conditionally.

• data_dir
  Type: str
  Description: Path to the data directory relative to this file, typically ../data. This directory is expected to contain fixture files used in tests.

• STR_CACHE
  Type: dict[str, str]
  Description: Cache mapping filenames to their loaded string contents. Helps avoid repeated disk reads and decodings.

• OBJ_CACHE
  Type: dict[str, Any]
  Description: Cache mapping filenames to their loaded and parsed JSON objects.

Functions

read_fixture_bytes(filename: str, subdir: str | None = None) -> bytes

Reads the raw bytes from a fixture file.

• Parameters:

  • filename (str): The name of the fixture file to read.

  • subdir (optional, str): Subdirectory inside data_dir where the file is located. If None, the file is read directly from data_dir.

• Returns:

  • bytes: The raw bytes content of the file.

• Behavior:

  • Constructs the full path to the fixture file.

  • If the file has an .xz extension, it decompresses it using LZMA before returning bytes.

  • Otherwise, reads raw bytes directly from the file.

• Usage Example:

  raw_bytes = read_fixture_bytes("sample.json")
  compressed_bytes = read_fixture_bytes("data.xz", subdir="compressed")
  

read_fixture_str(filename: str, subdir: str | None = None) -> str

Reads the contents of a fixture file as a UTF-8 decoded string, with caching.

• Parameters:

  • filename (str): The name of the fixture file.

  • subdir (optional, str): Subdirectory inside data_dir.

• Returns:

  • str: UTF-8 decoded contents of the fixture file.

• Behavior:

  • Checks if the string content is already cached in STR_CACHE.

  • If not cached, calls read_fixture_bytes and decodes the bytes to a string, storing it in the cache.

  • Returns the cached or newly loaded string.

• Usage Example:

  json_str = read_fixture_str("sample.json")
  

read_fixture_obj(filename: str) -> Any

Loads and returns a parsed JSON object from a fixture file, with caching.

• Parameters:

  • filename (str): The name of the fixture file containing JSON data.

• Returns:

  • Any: The deserialized JSON object (e.g., dict, list).

• Behavior:

  • Checks if the JSON object is already cached in OBJ_CACHE.

  • If not, loads the string content using read_fixture_str, parses it with orjson.loads, caches, and returns the object.

• Usage Example:

  data = read_fixture_obj("data.json")
  # data is now a Python dict or list parsed from JSON
  

Pytest Marker

• needs_data
  A Pytest marker that skips tests if the data directory does not exist. This prevents test failures when fixture data is missing.

  • Defined as:

    needs_data = pytest.mark.skipif(
        not Path(data_dir).exists(),
        reason="Test depends on ./data dir that contains fixtures",
    )
    

  • Usage:

    @needs_data
    def test_using_fixture():
        ...
    

Important Implementation Details

• Conditional Imports of numpy and pandas:
  These imports are wrapped in a try-except and conditional on the Python interpreter not running in free-threading mode (IS_FREETHREADING). This design avoids potential issues with these libraries under certain runtime configurations (e.g., PyPy or specialized Python builds).

• Caching Strategy:
  To optimize test runtime, the module caches string and object representations of fixtures after the first load. This reduces IO overhead and repeated JSON deserialization.

• Support for Compressed Fixtures:
  The read_fixture_bytes function transparently handles .xz compressed files by decompressing them on read, allowing tests to store fixtures efficiently on disk.

• Use of orjson:
  The library orjson is used for fast JSON deserialization, which is beneficial for large fixture files and improves test performance.

Interaction with Other Parts of the System

• This utility module is designed to be used primarily within test suites (e.g., Pytest tests) to provide standardized access to fixture data.

• The data_dir is expected to be populated with test fixture files, organized optionally into subdirectories.

• Test modules import this utility to load input data or expected outputs for tests, leveraging the caching and convenience functions provided.

• The Pytest marker needs_data is used to conditionally skip tests that cannot run without fixture data, improving test robustness in different environments.

Visual Diagram: Flowchart of Utility Functions and Their Relationships

flowchart TD
    A[read_fixture_bytes(filename, subdir)] -->|returns bytes| B[read_fixture_str(filename, subdir)]
    B -->|caches decoded str| C[STR_CACHE]
    B -->|returns str| D[read_fixture_obj(filename)]
    D -->|parses JSON with orjson.loads| E[OBJ_CACHE]
    D -->|returns object| F[Tests]

    subgraph "Cache Layers"
        C
        E
    end

    style C fill:#f9f,stroke:#333,stroke-width:1px
    style E fill:#f9f,stroke:#333,stroke-width:1px

Summary

[util.py](/projects/287/67676) is a focused utility module that simplifies loading test fixture data in raw bytes, string, or JSON object form with efficient caching and support for compressed files. It conditionally manages optional dependencies and provides a Pytest marker to handle test skipping when fixtures are unavailable. This module enhances the testing framework's robustness and performance by centralizing fixture handling logic.