test_reports.py

Overview

The [test_reports.py](/projects/286/67344) file contains a comprehensive suite of automated tests that verify the serialization, deserialization, and functional correctness of pytest's internal reporting objects, primarily focusing on `TestReport` and `CollectReport` classes. These reports encapsulate information about test execution (e.g., outcomes, tracebacks, timing).

The tests ensure that:

• Reports can be serialized to JSON and deserialized back without loss of critical information.

• Complex traceback and exception chain representations are faithfully preserved.

• Path-like objects in report attributes are properly handled during serialization.

• Hooks related to report serialization/deserialization work correctly.

• Known issues (bugs) related to report serialization are explicitly tested and guarded against regressions.

• Timing and duration data in reports are consistent.

• Report failure cases, including import failures and chained exceptions, are correctly represented.

This file is essential for maintaining the integrity of pytest's distributed and remote testing capabilities (such as `xdist`), where test reports need to be transferred between processes or machines.

Classes and Their Methods

Class: TestReportSerialization

This class contains multiple test methods validating the behavior of `TestReport` serialization and deserialization under various scenarios. It uses the `pytester` fixture extensively to create and run test cases within a temporary pytest environment.

**Usage Example:**

def test_xdist_longrepr_to_str_issue_241(self, pytester: Pytester) -> None:
    pytester.makepyfile(
        '''
        def test_a(): assert False
        def test_b(): pass
        '''
    )
    reprec = pytester.inline_run()
    reports = reprec.getreports("pytest_runtest_logreport")
    test_a_call = reports[1]
    assert test_a_call.when == "call"
    assert test_a_call.outcome == "failed"
    data = test_a_call._to_json()
    assert data["longrepr"]["reprtraceback"]["style"] == "long"

Class: TestHooks

This class tests the integration points (hooks) pytest provides for report serialization and deserialization, ensuring that plugins or other pytest extensions can customize or extend report behavior correctly.

**Usage Example:**

def test_test_report(self, pytester: Pytester, pytestconfig: Config) -> None:
    pytester.makepyfile(
        '''
        def test_a(): assert False
        def test_b(): pass
        '''
    )
    reprec = pytester.inline_run()
    reports = reprec.getreports("pytest_runtest_logreport")
    for rep in reports:
        data = pytestconfig.hook.pytest_report_to_serializable(config=pytestconfig, report=rep)
        assert data["$report_type"] == "TestReport"
        new_rep = pytestconfig.hook.pytest_report_from_serializable(config=pytestconfig, data=data)
        assert new_rep.nodeid == rep.nodeid

Important Implementation Details

• Serialization via _to_json and Deserialization via _from_json:
  The tests revolve around the conversion of report objects to JSON-serializable dictionaries and the reverse process. This is critical for sharing reports between processes or machines, particularly in distributed testing setups (xdist).

• Exception Representation:
  The file extensively tests the handling of complex exception representations:

  • ExceptionRepr: Single exception with traceback.

  • ExceptionChainRepr: Multiple chained exceptions with detailed traceback chains.

  • ReprEntry and ReprEntryNative: Representations of individual traceback entries with local variables and function args.

• Handling of Unknown Types:
  The test_deserialization_failure method validates robust error handling when unknown or unsupported traceback entry types are encountered during deserialization.

• Path-like Object Serialization:
  Attributes that implement the __fspath__ protocol (path-like objects) are serialized as strings to ensure compatibility and correct JSON representation.

• Timestamps and Duration Consistency:
  The test_report_timestamps_match_duration ensures that the difference between stop and start timestamps matches the reported test duration within a tolerance.

• Regression Tests for Known Issues:
  Multiple tests explicitly reference issues (e.g., pytest-xdist#241) to prevent regressions.

• Hooks Integration:
  The TestHooks class confirms that pytest's hook system (pytest_report_to_serializable and pytest_report_from_serializable) correctly serializes and deserializes reports, enabling plugin extensibility.

Interaction with Other Parts of the Application

• Pytest Core and Plugins:

  • Uses pytest's internal APIs (_pytest.reports, _pytest._code.code, _pytest.pytester) for report classes and utilities.

  • Integrates with pytester fixture to create isolated test environments and run test code inline.

  • Relies on pytestconfig and pytest hook system to test plugin hook behavior related to reporting.

• Distributed Testing (xdist):
  Many tests are motivated by issues arising during distributed test execution where test reports must be serialized and sent across process boundaries.

• Exception and Traceback Representations:
  Deep integration with how pytest internally represents exceptions and tracebacks, ensuring these complex structures survive serialization intact.

• Filesystem and Path Handling:
  Tests ensure that path-like objects in reports can be serialized without error, which is important given that test reports often include file locations.

Visual Diagram: Class Diagram for TestReportSerialization and TestHooks

classDiagram
    class TestReportSerialization {
        +test_xdist_longrepr_to_str_issue_241(pytester)
        +test_xdist_report_longrepr_reprcrash_130(pytester)
        +test_reprentries_serialization_170(pytester)
        +test_reprentries_serialization_196(pytester)
        +test_itemreport_outcomes(pytester)
        +test_collectreport_passed(pytester)
        +test_collectreport_fail(pytester)
        +test_extended_report_deserialization(pytester)
        +test_paths_support(pytester)
        +test_deserialization_failure(pytester)
        +test_chained_exceptions(pytester, tw_mock, report_class)
        +test_chained_exceptions_no_reprcrash(pytester, tw_mock)
        +test_report_prevent_ConftestImportFailure_hiding_exception(pytester)
        +test_report_timestamps_match_duration(pytester, mock_timing)
    }

    class TestHooks {
        +test_test_report(pytester, pytestconfig)
        +test_collect_report(pytester, pytestconfig)
        +test_invalid_report_types(pytester, pytestconfig, hook_name)
    }

Summary

[test_reports.py](/projects/286/67344) is a critical test module that ensures pytest's test reporting infrastructure, especially serialization and deserialization of reports, is robust, accurate, and extensible. It covers a wide range of scenarios from basic pass/fail/skipped outcomes to complex chained exceptions and plugin hook integrations. This thorough coverage supports pytest's distributed testing capabilities and plugin ecosystem.

End of Documentation for test_reports.py