vendor-yyjson

Overview

The `vendor-yyjson` file is a shell script designed to automate the **embedding and customization** of the **yyjson** C JSON parsing library into a larger software project. It fetches a specific fixed commit version of the `yyjson` source files (`yyjson.c` and `yyjson.h`) from the official GitHub repository, applies a series of source-code patches and textual modifications to adapt the library to the project’s specific needs, and applies additional patches that address recursion limits and reduce unused code.

This integration approach ensures the project uses a **stable, customized version** of `yyjson` with tailored behavior, memory management, and feature set suitable for the surrounding system (notably a Rust-based JSON parsing backend). The script is idempotent and can be rerun to update or reset the embedded codebase to the exact specified commit and patch state.

File Functionality Summary

Download fixed version of yyjson source files (yyjson.c and yyjson.h) from GitHub based on a commit hash.
Apply targeted sed commands to modify specific parts of the source code and header files:
- Change function inlining and API annotations.
- Disable or simplify certain runtime flags and feature conditionals by replacing them with constants (false or true).
- Remove or comment out error handling related code segments.
Apply custom patches (yyjson-recursion-limit.patch and yyjson-reduce-unused.patch) to:
- Set recursion limits to avoid stack overflow.
- Reduce unused code to optimize footprint.

This process results in a **trimmed, hardened, and controlled version of the embedded `yyjson` library** that fits the project’s performance, feature, and safety requirements.

Detailed Explanation of Script Steps

#!/usr/bin/env bash

set -eou pipefail

The script uses bash and exits immediately on errors, undefined variables, or pipeline errors for strict fail-fast behavior.

yyjson_version="b21c02904188add942d3c7cd4885422e4335f115"

Defines the exact git commit hash of yyjson to fetch, locking the embedded version.

curl -Ls -o include/yyjson/yyjson.c "https://raw.githubusercontent.com/ibireme/yyjson/${yyjson_version}/src/yyjson.c"
curl -Ls -o include/yyjson/yyjson.h "https://raw.githubusercontent.com/ibireme/yyjson/${yyjson_version}/src/yyjson.h"

Downloads the yyjson.c and yyjson.h files into the project’s include/yyjson directory, ensuring the source code is always aligned with the specified version.

Source Code Modifications via `sed`

These inline edits customize the embedded `yyjson` code:

sed -i 's/yyjson_api_inline void yyjson_doc_free/yyjson_api void yyjson_doc_free/g' include/yyjson/yyjson.h

Changes the yyjson_doc_free function from an inline function to a non-inline API function for linkage or binary size reasons.

The following `sed` commands **disable or simplify conditional features and error handling** by replacing complex flag checks or runtime conditionals with `false` or `true` constants to remove code paths:

sed -i 's/(flg & (YYJSON_READ_NUMBER_AS_RAW | YYJSON_READ_BIGNUM_AS_RAW)) != 0/false/g' include/yyjson/yyjson.c
sed -i 's/if (pre)/if (false)/g' include/yyjson/yyjson.c
sed -i 's/!false/true/g' include/yyjson/yyjson.c
sed -i 's/ && true//g' include/yyjson/yyjson.c
sed -i 's/true && //g' include/yyjson/yyjson.c
sed -i 's/unlikely(false)/false/g' include/yyjson/yyjson.c
sed -i 's/YYJSON_TYPE_STR | YYJSON_SUBTYPE_NOESC/YYJSON_TYPE_STR/g' include/yyjson/yyjson.c
sed -i 's/unlikely(pos == src)/false/g' include/yyjson/yyjson.c
sed -i 's/    yyjson_read_err dummy_err;//g' include/yyjson/yyjson.c
sed -i 's/    if (!err) err = &dummy_err;//g' include/yyjson/yyjson.c
sed -i 's/likely(!alc_ptr)/!alc_ptr/g' include/yyjson/yyjson.c
sed -i 's/unlikely(read_flag_eq(flg, YYJSON_READ_##_flag))/false/g' include/yyjson/yyjson.c
sed -i 's/has_read_flag(ALLOW_INF_AND_NAN)/false/g' include/yyjson/yyjson.c
sed -i 's/has_read_flag(ALLOW_COMMENTS)/false/g' include/yyjson/yyjson.c
sed -i 's/has_read_flag(BIGNUM_AS_RAW)/false/g' include/yyjson/yyjson.c
sed -i 's/if (pre && \*pre)/if (false)/g' include/yyjson/yyjson.c
sed -i 's/(pre && !false)/(false)/g' include/yyjson/yyjson.c

These edits effectively disable features such as reading numbers as raw strings, reading big numbers as raw, allowing comments, allowing Inf/NaN, and complex precondition checks.
They also simplify the code paths by removing unlikely branches and dummy error variables, likely reducing binary size and improving performance.

Applying Git Patches

git apply include/yyjson-recursion-limit.patch
git apply include/yyjson-reduce-unused.patch

Applies two local patches that:
- Enforce recursion limits to prevent stack overflows from deeply nested JSON.
- Remove unused code to optimize the embedded library size and complexity.

Usage and Integration

This script is intended to be part of the build or vendor update process for the project.
Running this script ensures the include/yyjson directory contains the exact version of yyjson source files customized for the project.
The patched yyjson files are then compiled along with other project sources, providing a fast, memory-safe JSON parsing and writing backend.
The integration is consumed primarily by the Rust deserialization backend, which calls into the C API exposed by the patched yyjson library.
The Rust layer benefits from the customized yyjson by:
- Ensuring consistent JSON parsing behavior without unwanted features (e.g., no comments or Inf/NaN support).
- Enforcing safety limits (recursion depth).
- Reducing binary size and complexity.

Important Implementation Details

Version Pinning: The script fixes the embedded yyjson version using a specific git commit hash (b21c029...), ensuring deterministic builds.
In-Place Source Editing: Modifications are applied directly on the downloaded files using sed, enabling precise feature disabling and code simplification without maintaining a separate fork.
Patch Application: Additional adjustments via patches complement inline edits to handle recursion safety and code footprint.
No Compilation Step Here: The script only fetches and modifies source code; actual compilation is handled elsewhere in the build system.

Example Usage

To update or embed the customized `yyjson` library, run:

./vendor-yyjson

This will:

Download yyjson.c and yyjson.h at the pinned commit.
Apply all sed modifications.
Apply patches.
Leave the modified source ready for compilation by the build system.

Interaction with Other System Parts

Rust Backend: The Rust JSON deserialization backend imports and calls into this customized yyjson C library for high-performance parsing.
Python Layer: Python bindings indirectly depend on this layer for JSON parsing speed and correctness.
Build System: This script is part of the vendor or pre-build step ensuring dependencies are correctly set up.
Patch Maintenance: Patches applied here are maintained as part of project source control under include/.

Mermaid Diagram: Script Workflow and `yyjson` Integration

flowchart TD
    A[Start vendor-yyjson Script] --> B[Set yyjson_version Commit]
    B --> C[Download yyjson.c and yyjson.h]
    C --> D[Apply sed Modifications to yyjson.h]
    D --> E[Apply sed Modifications to yyjson.c]
    E --> F[Apply yyjson-recursion-limit.patch]
    F --> G[Apply yyjson-reduce-unused.patch]
    G --> H[Source Ready for Compilation]
    H --> I[Build System Compiles Patched yyjson]
    I --> J[Rust Backend Calls yyjson C API]
    J --> K[High-Performance JSON Parsing]

This flowchart illustrates the overall process from running the script to embedding the customized yyjson library into the project and its usage by the Rust backend.

Summary

The `vendor-yyjson` file is a **vendor script** that automates fetching, customizing, and patching the `yyjson` C library for integration into a larger JSON parsing system. Through fixed version fetching, targeted source modifications, and patch application, it ensures the embedded `yyjson` library matches the project’s requirements for speed, safety, and feature set. This forms the foundational JSON parsing engine used primarily by the Rust deserialization backend, enabling blazing-fast, memory-safe JSON processing within a multi-language ecosystem.