multipython.py

Overview

The `multipython.py` module provides a set of parametrized tests designed to verify cross-version Python serialization compatibility using the `pickle` module. Specifically, it tests whether Python objects serialized (pickled) with one Python interpreter version can be correctly deserialized (unpickled) and validated with another Python interpreter version. This is critical for ensuring backward and forward compatibility of pickled data across multiple Python environments.

The tests leverage pytest fixtures to dynamically run serialization and deserialization steps across multiple Python versions (3.9, 3.10, 3.11) available on the system. The core logic is encapsulated in the `Python` class, which abstracts the invocation of different Python executables to perform pickling and unpickling subprocess operations.

Classes and Functions

Class: Python

Represents a Python interpreter instance for running serialization and deserialization subprocesses with a specified Python version.

__init__(self, version: str, picklefile: pathlib.Path)

• Purpose: Initializes a Python instance for a given Python interpreter version, associating it with a file path for storing serialized data.

• Parameters:

  • version: Name or identifier of the Python executable (e.g., "python3.9").

  • picklefile: A pathlib.Path object pointing to the file where pickled data will be written or read.

• Behavior:

  • Uses shutil.which to find the executable path of the specified Python version.

  • If the executable is not found, it skips the pytest test using pytest.skip.

• Example:

  python39 = Python("python3.9", pathlib.Path("/tmp/data.pickle"))
  

dumps(self, obj: Any) -> None

• Purpose: Serializes (pickles) the given Python object using the Python interpreter represented by this instance.

• Parameters:

  • obj: Any picklable Python object to be serialized.

• Behavior:

  • Generates a temporary Python script (dump.py) that imports pickle, opens the specified pickle file in binary write mode, and dumps the object using pickle.dump with protocol 2.

  • Runs this script as a subprocess using the interpreter path.

• Usage Example:

  python39.dumps({"key": "value"})
  

• Notes:

  • Uses pickle protocol 2 for compatibility across Python versions.

  • The serialized data is saved at self.picklefile.

load_and_is_true(self, expression: str) -> None

• Purpose: Deserializes (unpickles) the object from the pickle file and evaluates a Python expression against it, asserting the expression is True.

• Parameters:

  • expression: A string containing a Python expression to evaluate where the unpickled object is referenced as obj.

• Behavior:

  • Generates a temporary Python script (load.py) that loads the pickled object from self.picklefile, evaluates the expression, and exits with status 1 if the expression evaluates to False.

  • Runs this script using the Python interpreter subprocess; if the expression fails, the subprocess will raise an error and pytest will report test failure.

• Usage Example:

  python310.load_and_is_true("obj == {'key': 'value'}")
  

• Notes:

  • This method acts as a test assertion across Python versions by running actual interpreter subprocesses.

Pytest Fixtures

• pythonlist: A list of Python interpreter versions to test: ["python3.9", "python3.10", "python3.11"].

• python1(request, tmp_path) -> Python

  • Parameterized fixture that creates a Python instance with a temporary pickle file.

  • Runs test with each Python version in pythonlist.

  • The pickle file is created under tmp_path (pytest-managed temporary directory).

• python2(request, python1) -> Python

  • Parameterized fixture similar to python1, but shares the pickle file created by python1.

  • Allows testing cross-version serialization by pickling in python1's interpreter and unpickling in python2's interpreter.

Test Function

test_basic_objects(python1: Python, python2: Python, obj: Any)

• Purpose: Test that basic Python objects serialized by one Python version can be correctly deserialized and validated by another.

• Parameters:

  • python1: The Python interpreter instance used to pickle the object.

  • python2: The Python interpreter instance used to unpickle and assert the object.

  • obj: The Python object to test, parameterized with values [42, {}, {1: 3}].

• Test Steps:

  1. Serialize obj using python1.dumps().

  2. Deserialize and assert obj == <original> using python2.load_and_is_true().

• Behavior:

  • Runs cross-version compatibility tests for integer, empty dictionary, and a simple dictionary.

• Example pytest command:

  pytest multipython.py
  

Important Implementation Details and Algorithms

• Cross-version testing via subprocess:
  Instead of pickling/unpickling in the same Python runtime, this module explicitly creates small Python scripts on disk (dump.py and load.py) which are executed using subprocess calls to the specified Python interpreters. This approach ensures testing of true cross-interpreter compatibility.

• Use of pickle protocol 2:
  The choice of protocol 2 for pickle.dump ensures compatibility across Python 3.9 through 3.11, as protocol 2 was introduced in Python 2.3 and is supported by all these versions.

• Dynamic test parametrization:
  Pytest fixtures dynamically parametrize tests with multiple Python versions, allowing combinatorial testing without manual intervention.

• Temporary files management:
  The pickle file and generated scripts are created in pytest's tmp_path temporary directory, ensuring isolation and cleanup after tests.

Interaction with Other Parts of the System

• pytest framework:
  This file is designed to be run with pytest, which manages fixture lifecycle, parameterization, test discovery, and result reporting.

• System Python installations:
  The module depends on having the specified Python versions (python3.9, python3.10, python3.11) installed and discoverable in the system's PATH.

• Pickle serialization format:
  The tests verify interoperability of serialized data produced by the standard Python pickle module across multiple Python versions.

• No direct dependencies on other project modules:
  This file is a standalone test utility focusing on interpreter compatibility and does not import or interact with other project components.

Visual Diagram

classDiagram
    class Python {
        -pythonpath: str
        -picklefile: pathlib.Path
        +__init__(version: str, picklefile: Path)
        +dumps(obj: Any) void
        +load_and_is_true(expression: str) void
    }

    class pytest_fixture_python1 {
        +params: list[str]
        +python1(request, tmp_path) -> Python
    }

    class pytest_fixture_python2 {
        +params: list[str]
        +python2(request, python1) -> Python
    }

    class test_basic_objects {
        +test_basic_objects(python1: Python, python2: Python, obj: Any)
    }

    pytest_fixture_python1 --> Python : creates instance
    pytest_fixture_python2 --> Python : creates instance (shares picklefile)
    test_basic_objects --> pytest_fixture_python1 : uses
    test_basic_objects --> pytest_fixture_python2 : uses
    test_basic_objects --> Python : calls dumps() and load_and_is_true()

Summary

The `multipython.py` file is a specialized pytest test module that verifies the compatibility of pickled data across multiple Python interpreter versions. It uses subprocess-driven scripts to serialize and deserialize objects in different Python environments and asserts correctness by evaluating expressions in those contexts. This ensures that serialized Python data can be safely shared or migrated between environments running different Python versions.