pycorrectness

Overview

The **`pycorrectness`** script is a comprehensive JSON parser correctness testing utility. It systematically verifies the ability of multiple JSON parsing libraries—specifically `orjson` and Python's built-in `json`—to correctly accept or reject a diverse collection of JSON fixtures. These fixtures include both valid and invalid JSON documents sourced from standardized test sets.

The script reads test fixtures from compressed and uncompressed files, applies each library's JSON loading function, and records whether the library correctly parses or rejects each document. It distinguishes between expected passes and failures based on filename conventions and whitelists, tabulates results, and reports mistaken acceptance or rejection counts for each library.

This tool is critical for validating the correctness and robustness of JSON parsers, ensuring that they conform to expected JSON standards and behave consistently across edge cases.

Detailed Explanation of Components

Constants and Global Variables

• dirname
  Base directory pointing to the data folder containing JSON fixtures.

• LIBRARIES
  List of JSON libraries under test: ["orjson", "json"].

• LIBRARY_FUNC_MAP
  Maps library names to their respective JSON loading functions for deserialization:

  {
      "orjson": orjson.loads,
      "json": json.loads,
  }
  

• PARSING and JSONCHECKER
  Dictionaries mapping filenames to raw bytes of JSON fixtures loaded from data/parsing and data/jsonchecker directories respectively. Files ending in .xz are decompressed using lzma.

• RESULTS
  A nested dictionary (defaultdict(dict)) storing pass/fail test results keyed by fixture filename and library.

• MISTAKEN_PASSES, MISTAKEN_FAILS
  Dictionaries counting false positives and false negatives per library:

  • Mistaken passes: valid JSON documents rejected by a library.

  • Mistaken fails: invalid JSON documents accepted by a library.

• PASS_WHITELIST
  A tuple of filenames that are exceptions to general pass/fail naming conventions.

Functions

read_fixture_bytes(filename: str, subdir: Optional[str] = None) -> bytes

Reads a fixture file as raw bytes.

• Parameters:

  • filename: Name of the fixture file.

  • subdir: Optional subdirectory under the main data directory.

• Returns:
  Raw bytes of the fixture content; decompresses .xz files automatically.

• Usage:

  fixture_bytes = read_fixture_bytes("pass01.json", "parsing")
  

test_passed(libname: str, fixture: bytes) -> bool

Checks if the given JSON fixture passes parsing tests with the specified library.

• Parameters:

  • libname: Library name ("orjson" or "json").

  • fixture: Raw bytes of JSON data.

• Returns:
  True if both:

  1. Loading the fixture bytes matches orjson.loads result.

  2. Loading the UTF-8 decoded string matches orjson.loads result.

  Otherwise, `False`.

• Behavior:
  Catches exceptions during parsing and treats them as test failures.

• Example:

  result = test_passed("json", fixture_bytes)
  if result:
      print("Test passed for json library")
  

test_failed(libname: str, fixture: bytes) -> bool

Checks if the given JSON fixture fails parsing (i.e., is rejected) by the specified library as expected.

• Parameters:

  • libname: Library name ("orjson" or "json").

  • fixture: Raw bytes of JSON data.

• Returns:
  True if parsing fails for both:

  1. The raw bytes input.

  2. The UTF-8 decoded string input.

  Otherwise, `False`.

• Behavior:
  Catches exceptions and treats them as expected failure signals.

• Example:

  result = test_failed("orjson", invalid_fixture_bytes)
  if result:
      print("Invalid JSON correctly rejected by orjson")
  

should_pass(filename: str) -> bool

Determines if a fixture file is expected to be valid JSON and should parse successfully.

• Parameters:

  • filename: Name of the fixture file.

• Returns:
  True if filename matches pass conditions:

  • Starts with "y_" or "pass".

  • Is in the PASS_WHITELIST.

• Example:

  if should_pass("pass01.json"):
      print("This fixture should pass parsing")
  

should_fail(filename: str) -> bool

Determines if a fixture file is expected to be invalid JSON and should fail parsing.

• Parameters:

  • filename: Name of the fixture file.

• Returns:
  True if filename matches fail conditions:

  • Starts with "n_", "i_string", "i_object", or "fail".

  • Is NOT in the PASS_WHITELIST.

• Example:

  if should_fail("fail02.json"):
      print("This fixture should fail parsing")
  

Main Execution Flow

1. Fixture Loading
   Loads all test fixtures from data/parsing and data/jsonchecker directories into PARSING and JSONCHECKER.

2. Iterate Over Libraries and Fixtures
   For each library and each fixture set (PARSING and JSONCHECKER), iterates through all files:

   • Valid fixtures (should_pass):
     Runs test_passed() and records result.
     Increments MISTAKEN_PASSES if the test unexpectedly fails.

   • Invalid fixtures (should_fail):
     Runs test_failed() and records result.
     Increments MISTAKEN_FAILS if the test unexpectedly passes.

   • Intermediate fixtures (filenames starting with "i_"):
     Skips testing.

   • Unknown fixtures:
     Raises NotImplementedError.

3. Result Tabulation
   Compiles a markdown-formatted table showing each fixture's pass/fail status per library, printed to stdout.

4. Mistaken Acceptance/Rejection Summary
   Prints a summary table showing, per library:

   • Number of invalid documents erroneously accepted.

   • Number of valid documents erroneously rejected.

5. Total Tests Count
   Prints the total number of documents tested.

Important Implementation Details

• Fixture Decoding and Comparison
  The correctness tests compare the output of each library's loads function to that of orjson.loads for the same input. This cross-library comparison ensures a consistent reference point.

• Dual Input Forms Tested
  Each fixture is tested twice per library:

  • As raw bytes.

  • As UTF-8 decoded string.

• Filename-Based Test Classification
  The script relies heavily on naming conventions (prefixes such as y_, n_, pass, fail, i_) and a whitelist to determine expected test outcomes.

• Handling Compressed Fixtures
  Supports .xz compressed fixtures transparently using lzma.

• Results Storage
  Uses nested dictionaries for efficient result lookup and aggregation.

• Tabular Output
  Uses the tabulate library to produce human-readable, GitHub-flavored Markdown tables for easy inspection or integration into reports.

Usage Example

Assuming the `data/parsing` and `data/jsonchecker` directories contain JSON fixtures:

$ python pycorrectness

The script will print two markdown tables:

1. Detailed test results per fixture and library (with "ok" or "fail").

2. Summary of mistaken accepts/fails per library.

Finally, it prints how many documents were tested in total.

Interaction with Other System Components

• Data Fixtures:
  Relies on JSON test documents stored under data/parsing/ and data/jsonchecker/ directories.

• JSON Libraries Under Test:
  Integrates with orjson (Rust-backed JSON library) and Python’s built-in json module.

• Benchmarking Suite:
  Complements the benchmarking framework by focusing on correctness validation rather than performance metrics.

• Test Automation:
  Can be integrated into continuous integration pipelines to detect regressions or inconsistencies in JSON parsing correctness.

Mermaid Class Diagram

classDiagram
    class pycorrectness {
        - dirname: str
        - LIBRARIES: list
        - LIBRARY_FUNC_MAP: dict
        - PARSING: dict
        - JSONCHECKER: dict
        - RESULTS: defaultdict(dict)
        - MISTAKEN_PASSES: dict
        - MISTAKEN_FAILS: dict
        - PASS_WHITELIST: tuple

        + read_fixture_bytes(filename, subdir=None) bytes
        + test_passed(libname, fixture) bool
        + test_failed(libname, fixture) bool
        + should_pass(filename) bool
        + should_fail(filename) bool
        + main() void
    }

Summary

The **`pycorrectness`** script is a specialized correctness validation tool for JSON parsers. It automates the process of verifying that JSON libraries correctly accept valid JSON and reject invalid JSON across extensive standardized test suites. By systematically comparing results, tabulating outcomes, and highlighting mistaken acceptances or rejections, it serves as a critical quality assurance component within the broader JSON benchmarking and testing ecosystem.