pybench

Overview

The **pybench** file is a simple Bash script designed to run Python benchmark tests using the `pytest` framework with the `pytest-benchmark` plugin. It provides a standardized way to execute and measure performance of benchmark test scripts located in the `bench/` directory. The script takes a single argument that identifies which benchmark test file to run and configures pytest with specific benchmarking options to control the test duration, garbage collection behavior, data saving, and test execution order.

Detailed Explanation

Script Purpose

Automate running a specific benchmark test file with consistent pytest benchmarking options.
Ensure reliable benchmarking by setting minimum and maximum test times and disabling garbage collection during benchmarks.
Save benchmark results automatically for later analysis.
Run tests in random order to reduce bias from test sequences.

How to Use

./pybench <benchmark_name>

<benchmark_name>: The identifier suffix of the benchmark file located in bench/benchmark_<benchmark_name>.py.

**Example:**

./pybench sorting

This will run:

pytest --verbose \
    --benchmark-min-time=1 \
    --benchmark-max-time=5 \
    --benchmark-disable-gc \
    --benchmark-autosave \
    --benchmark-save-data \
    --random-order \
    bench/benchmark_sorting.py

Script Breakdown

#!/usr/bin/env bash

Uses the system's default Bash shell to execute the script.

set -eou pipefail

-e: Exit immediately if any command exits with a non-zero status.
-o pipefail: Causes a pipeline to return the exit status of the last command in the pipe that failed.
-u: Treat unset variables as an error and exit immediately.

This combination ensures the script halts on errors or usage of undefined variables, improving safety and debugging.

Command Details

The script runs `pytest` with the following options:

Option	Description
`--verbose`	Enables verbose output for clearer test progress visibility.
`--benchmark-min-time=1`	Each benchmark runs for at least 1 second to gather sufficient data for accuracy.
`--benchmark-max-time=5`	Benchmarks run no longer than 5 seconds to limit total test time.
`--benchmark-disable-gc`	Garbage collection is disabled during benchmarks to reduce noise and improve consistency.
`--benchmark-autosave`	Automatically saves benchmark results after the run completes.
`--benchmark-save-data`	Saves detailed benchmark data for future comparisons or analysis.
`--random-order`	Randomizes the order of test execution to avoid order-dependent side effects.
`"bench/benchmark_$1.py"`	Runs the benchmark test file whose name is constructed dynamically from the provided argument.

Implementation Details

The script is minimalistic and relies entirely on pytest and pytest-benchmark features.
No complex algorithms implemented here; the focus is on reliable execution and configuration of benchmarking tests.
The use of set -eou pipefail makes it robust against common shell scripting pitfalls.
The dynamic filename construction bench/benchmark_$1.py allows flexible invocation for different benchmark modules without modifying the script.

Interaction with Other System Components

Benchmark Test Files: The script expects benchmark Python scripts to be located in the bench directory and named as benchmark_<name>.py.
pytest Framework: Relies on pytest to discover and run tests.
pytest-benchmark Plugin: Uses this plugin's options to control benchmarking behavior.
Test Environment: Assumes a Python environment with pytest and pytest-benchmark installed.
Continuous Integration (CI): Can be integrated into CI pipelines to automate performance regression testing.
Benchmark Result Storage: The autosaved benchmark data can be used by other tools or scripts for historical performance analysis.

Summary

Aspect	Description
File Type	Bash script
Purpose	Run Python benchmark tests with predefined options
Input	Benchmark test name (string)
Output	Benchmark results printed and saved automatically
Dependencies	pytest, pytest-benchmark plugin, benchmark test files
Usage Context	Performance testing and benchmarking automation

Mermaid Diagram

flowchart TD
    A[Start: Run pybench <benchmark_name>] --> B[Construct filename: bench/benchmark_<name>.py]
    B --> C[Invoke pytest with benchmark options]
    C --> D{pytest executes benchmark tests}
    D --> E[Run each benchmark for 1-5 seconds]
    E --> F[Disable garbage collection during tests]
    F --> G[Randomize test order]
    G --> H[Autosave benchmark results and data]
    H --> I[Display verbose output]
    I --> J[Exit with status code]

    style A fill:#f9f,stroke:#333,stroke-width:1px
    style J fill:#bbf,stroke:#333,stroke-width:1px

Summary

The `pybench` script is a utility for running Python benchmarking tests in a controlled and consistent manner. By wrapping pytest with specific options, it ensures benchmarks are run with reliable timing, garbage collection disabled for consistency, and results saved automatically. This helps developers and teams easily measure and track performance changes across different benchmark scripts within a project.