lint

Overview

The `lint` file is a shell script designed to automate the static code analysis, formatting, and type checking of the project's Python source code and related scripts. It serves as a quality assurance tool to enforce coding standards, detect potential errors early, and maintain consistent style across various Python files and scripts within the repository.

This script primarily leverages:

• ruff: A fast Python linter and formatter that checks code quality and automatically applies fixes.

• mypy: A static type checker for Python that validates type annotations and detects type inconsistencies.

By running `lint`, developers and CI systems ensure that the Python codebase adheres to defined style and typing conventions before committing or deploying changes.

Detailed Explanation

Script Purpose

• Linting and auto-fixing: Using ruff to identify and correct style violations and common errors.

• Formatting: Ensuring code formatting consistency with ruff format.

• Type checking: Validating Python type annotations with mypy, ignoring missing imports but checking untyped function definitions.

Script Content Breakdown

#!/usr/bin/env bash

set -eou pipefail

• Shebang: Executes the script using the environment's Bash shell.

• set -eou pipefail:

  • -e: Exit immediately if a command exits with a non-zero status.

  • -o pipefail: Return the exit status of the last command in the pipeline that failed.

  • -u: Treat unset variables as an error and exit immediately.

This ensures the script stops on errors or unset variables, making it robust for CI and local runs.

to_lint="./bench/*.py ./pysrc/orjson/__init__.pyi ./test/*.py script/pydataclass
script/pysort script/pynumpy script/pynonstr script/pycorrectness script/graph integration/init
integration/wsgi.py integration/typestubs.py integration/thread script/check-version
script/check-pypi"

• Variable to_lint: Defines the set of files and directories to be linted and type-checked.

• Includes:

  • Python source files in bench/, test/.

  • Type stub file __init__.pyi in pysrc/orjson/.

  • Various scripts under script/ and integration/.

• Supports both .py files and executable scripts without extensions (e.g., script/pydataclass).

ruff check ${to_lint} --fix

• Runs ruff in check mode with the --fix flag, which lints the specified files and automatically fixes issues when possible.

• This step enforces code style and common error detection with auto-correction.

ruff format ${to_lint}

• Runs ruff format on the same files, which specifically formats the codebase (similar to black or autopep8).

• This ensures consistent code formatting post-lint corrections.

mypy --ignore-missing-imports --check-untyped-defs ./bench/*.py ./pysrc/orjson/__init__.pyi ./test/*.py

• Runs mypy for static type checking.

• Flags:

  • --ignore-missing-imports: Ignores unresolved imports, useful when some dependencies or stubs are unavailable.

  • --check-untyped-defs: Checks function bodies even if the function lacks type annotations, increasing type coverage.

• Targets critical Python code areas: benchmarking scripts, core type stubs, and tests.

• Does not run mypy on scripts or integration files, likely because of dynamic typing or external dependencies.

Parameters and Usage

• Parameters: None. The script uses hardcoded file paths for linting.

• Usage: Run the script from the root repository directory to lint and type-check the project’s Python source files.

./lint

Important Implementation Details

• The script uses shell glob patterns (*.py) to specify files, which depends on the shell's glob expansion.

• Files specified include both Python source files and executable scripts without extensions, ensuring comprehensive coverage.

• It uses ruff both for lint checking and formatting, leveraging its speed and auto-fixing capabilities.

• mypy is restricted to specific directories and files to balance thoroughness and practicality.

• The combination of --fix with ruff check ensures immediate correction, reducing manual intervention.

• The script enforces strict error checking (set -eou pipefail) which is essential for CI pipelines to catch issues immediately.

Interaction with Other Parts of the System

• This script is typically invoked manually by developers or automatically within Continuous Integration (CI) pipelines to maintain code quality.

• It complements the build and development scripts such as script/develop and script/debug by ensuring code correctness before building and testing.

• By running on both source files and scripts, it ensures consistency across various project components, preventing style drift.

• It does not modify Rust components, which are handled by separate build and linting tools (e.g., cargo).

• The linting and type checking validate the Python interface and test code that interact with the Rust extension built by the underlying Rust build system.

Example

Assuming a developer modifies some Python files in `test/` and scripts in `script/`, running:

./lint

will:

• Automatically fix any linting issues found by ruff.

• Format the files consistently.

• Check for type errors in critical Python modules and tests.

• Exit with an error status if any problems remain, preventing faulty code from progressing to commit or CI.

Visual Diagram

flowchart TD
    Start[Start lint script]
    DefineFiles[Define files to lint/type-check]
    RuffCheck[Ruff: lint & auto-fix]
    RuffFormat[Ruff: format code]
    MyPyCheck[Mypy: type check]
    Success[All checks passed]
    Fail[Errors found]

    Start --> DefineFiles --> RuffCheck
    RuffCheck --> RuffFormat
    RuffFormat --> MyPyCheck
    MyPyCheck -->|No errors| Success
    MyPyCheck -->|Errors| Fail

This flowchart illustrates the sequential workflow of the `lint` script: defining target files, running linting with auto-fixes, formatting, and static type checking, culminating in success or failure based on detected issues.

Summary

• The lint script is a simple but essential utility that enforces Python code quality across the project.

• It uses ruff for fast linting and formatting, with automatic fixes applied.

• It uses mypy for static type analysis on core Python modules and tests.

• The script ensures that code style and type errors are caught early, improving maintainability and reducing bugs.

• It integrates seamlessly with the project’s development lifecycle, especially CI workflows, contributing to a robust and clean codebase.

**End of `lint` file documentation.**