acceptance_test.py

Overview

The `acceptance_test.py` file is a comprehensive test suite designed to perform acceptance and integration testing on the core functionalities of the `pytest` testing framework itself. It verifies a wide range of scenarios including error handling, configuration issues, plugin loading, invocation variants, fixture behavior, duration reporting, and various edge cases related to test discovery and execution.

This file serves as a critical component ensuring `pytest`'s robustness, user experience, and backward compatibility by simulating real-world usage patterns and failure modes. It leverages the internal testing utilities of `pytest`, particularly the `Pytester` testing helper, to create isolated test environments, run `pytest` commands, and assert expected outcomes.

Key Classes and Functions

Utility Function

prepend_pythonpath(*dirs) -> str

• Purpose:
  Constructs a new PYTHONPATH environment variable string by prepending given directory paths to the existing PYTHONPATH if any.

• Parameters:

  • *dirs: A variable number of directory paths (strings or Path objects) to prepend.

• Returns:
  A string suitable for setting the PYTHONPATH environment variable, with the specified directories prepended.

• Usage Example:

  new_path = prepend_pythonpath("/path/to/dir1", "/path/to/dir2")
  os.environ["PYTHONPATH"] = new_path
  

Class: TestGeneralUsage

This class encapsulates multiple test methods that verify general usage scenarios of `pytest`. It covers error reporting, configuration loading, plugin behavior, test discovery, parametrization, and more.

• Common Parameter:

  • pytester: Pytester — A fixture that provides a test environment for running pytest subprocesses and managing test files.

• Notable Methods:

  • test_config_error
    Tests that configuration errors in conftest.py files are reported correctly, and that pytest_unconfigure hook is invoked.

  • test_root_conftest_syntax_error
    Ensures that a syntax error in the root conftest.py file results in a non-zero exit code and appropriate error messages.

  • test_early_hook_error_issue38_1 and test_early_hook_configure_error_issue38
    Verify that exceptions raised early in hook implementations (pytest_sessionstart, pytest_configure) are reported as internal errors with tracebacks.

  • test_file_not_found and test_file_not_found_unconfigure_issue143
    Check that running pytest on non-existent files or directories yields usage errors, and that pytest_configure and pytest_unconfigure hooks still run as expected.

  • test_config_preparse_plugin_option
    Tests that plugin options added via pytest_addoption are parsed correctly before configuration.

  • test_early_load_setuptools_name
    Simulates setuptools entry points loading to verify plugin loading order, including an optional load_cov_early flag to test early loading of coverage plugins.

  • test_assertion_rewrite
    Parametrized test to check that assertion rewriting works with different import modes (prepend, append, importlib).

  • test_nested_import_error
    Ensures that import errors nested within test modules are correctly reported with proper exit codes.

  • test_not_collectable_arguments
    Validates that .pyc files or other non-collectable test arguments cause usage errors.

  • test_better_reporting_on_conftest_load_failure
    Confirms that import failures in conftest.py files produce user-friendly tracebacks.

  • test_early_skip
    Verifies that early test skipping inside pytest_collect_file leads to no tests collected and appropriate skip messages.

  • test_docstring_on_hookspec
    Ensures all hook specifications in _pytest.hookspec have documentation strings.

  • And many more tests covering corner cases, warnings, async test support, handling of StopIteration, plugin APIs, and more.

• Usage Example:
  These tests are run automatically by the test framework and are not intended to be called directly by users. However, developers contributing to pytest can run them to verify changes, e.g.:

  pytest acceptance_test.py -k TestGeneralUsage
  

Class: TestInvocationVariants

Tests various ways of invoking `pytest` and ensures consistent behavior.

• Key Tests:

  • test_earlyinit
    Checks that pytest exposes expected attributes early on import.

  • test_pydoc
    Tests that help(pytest) works and includes expected content.

  • test_import_star_pytest
    Verifies that from pytest import * imports key names without errors.

  • test_double_pytestcmdline
    Ensures calling pytest.main() multiple times works in the same process.

  • test_python_minus_m_invocation_ok and test_python_minus_m_invocation_fail
    Test running pytest as a module with Python's -m switch in success and failure scenarios.

  • test_invoke_with_invalid_type
    Ensures pytest.main() raises TypeError if called with invalid argument types.

  • test_invoke_plugin_api
    Tests that plugins passed as objects to pytest.main() are registered and their options are recognized.

  • Various tests for --pyargs option, namespace packages, symlinked packages, and legacy namespace package support.

• Usage Example:
  These tests simulate pytest invocations programmatically and via subprocess, validating that pytest behaves correctly under different CLI and API usages.

Class: TestDurations

Tests the `--durations` feature of `pytest` that reports slowest test durations.

• Key Features:

  • Creates sample tests with controlled sleep durations to simulate varying test run times.

  • Tests the output of duration reports with different verbosity and minimum duration thresholds.

  • Verifies that durations are correctly reported even when tests are deselected or collection fails.

• Important Method:
  check_tests_in_output(lines: Sequence[str], *expected_test_numbers: int, number_of_tests: int = 3) -> None
  Utility to verify that the duration output includes the expected tests.

Class: TestDurationsWithFixture

Similar to `TestDurations`, but tests duration reporting when tests include fixtures with setup times.

• Notable:
  Tests that setup durations are shown along with call durations when --durations is used.

Individual Test Functions

Apart from the classes, the file defines multiple standalone test functions, such as:

• test_zipimport_hook
  Tests that pytest works correctly with Python zip applications.

• test_import_plugin_unicode_name
  Ensures plugins with unicode names are imported without error.

• test_pytest_plugins_as_module
  Verifies that pytest_plugins can be a module and not just a list of strings.

• test_deferred_hook_checking
  Confirms that hooks are checked as late as possible for flexibility.

• test_fixture_values_leak
  Checks that fixture objects are properly garbage collected after tests to avoid memory leaks.

• test_frame_leak_on_failing_test
  Ensures that frame references from failed tests are released to prevent leaks.

• test_fixture_mock_integration
  Validates that decorating fixtures with mocks does not break test execution.

• test_usage_error_code
  Verifies that unknown CLI options return the proper usage error code.

• test_error_on_async_function and related tests
  Check that async test functions are marked as errors since pytest does not natively support them yet.

• test_warning_on_sync_test_async_fixture and related tests
  Confirm warning messages are emitted when synchronous tests depend on async fixtures.

• test_pdb_can_be_rewritten
  Tests that assertion rewriting works even for standard library modules like pdb.

• test_tee_stdio_captures_and_live_prints
  Checks that --capture=tee-sys captures and prints stdio correctly and includes output in junitxml files.

• test_no_brokenpipeerror_message
  Ensures that broken pipe errors on shutdown are properly suppressed.

• test_function_return_non_none_warning
  Detects and warns when test functions return non-None values, suggesting possible mistakes.

• Several tests for compatibility, error handling, and corner cases related to collection, parametrization, and plugin management.

Implementation Details and Algorithms

• Most tests use the Pytester fixture (pytester) extensively to:

  1. Create temporary test files and directories.

  2. Write Python code snippets or entire modules to test files.

  3. Run pytest invocations either as subprocesses or in-process.

  4. Capture and analyze stdout, stderr, and return codes.

  5. Assert expected outcomes, error messages, warnings, and test results.

• The tests simulate a broad spectrum of user scenarios, including plugin loading via setuptools entry points, invocation through different CLI options, and handling of various Python features (like async).

• Many tests use pytest.mark.parametrize to cover multiple input variations efficiently.

• Error and warning message checks use fnmatch style matching to allow flexible substring checks.

• Use of dataclasses and monkeypatching (monkeypatch) facilitates mocking of entry points and environment variables.

• Sleep calls from the _pytest.timing module simulate slow tests for the duration feature tests.

• Use of subprocess calls to isolate tests that may affect global interpreter state or environment.

Interaction with Other System Components

• Pytester (_pytest.pytester.Pytester):
  This class provides the test harness to create test files, run pytest commands, and capture outputs. acceptance_test.py heavily depends on Pytester for orchestrating acceptance tests.

• Pytest Core Hooks and Plugins:
  The tests invoke and verify behavior of core hooks such as pytest_configure, pytest_unconfigure, pytest_collect_file, and plugin loading mechanisms.

• Setuptools Entry Points (importlib.metadata.distributions):
  Used in tests to simulate plugin discovery and loading order.

• System Environment and Python Import System:
  Tests manipulate environment variables (PYTHONPATH), sys.modules, and import behaviors to validate plugin import, name resolution, and error handling.

• Test Duration Timing (_pytest.timing):
  Used to simulate test execution durations for performance reporting.

• Subprocess and File System Utilities:
  Some tests run pytest in subprocesses to isolate effects, test zipapp support, or simulate real CLI runs.

Visual Diagram

classDiagram
    class TestGeneralUsage {
        +test_config_error(pytester)
        +test_root_conftest_syntax_error(pytester)
        +test_early_hook_error_issue38_1(pytester)
        +test_early_hook_configure_error_issue38(pytester)
        +test_file_not_found(pytester)
        +test_config_preparse_plugin_option(pytester)
        +test_early_load_setuptools_name(pytester, monkeypatch, load_cov_early)
        +test_assertion_rewrite(pytester, import_mode)
        +test_nested_import_error(pytester)
        +test_better_reporting_on_conftest_load_failure(pytester)
        +test_early_skip(pytester)
        +... (many more)
    }

    class TestInvocationVariants {
        +test_earlyinit(pytester)
        +test_pydoc(pytester)
        +test_import_star_pytest(pytester)
        +test_double_pytestcmdline(pytester)
        +test_python_minus_m_invocation_ok(pytester)
        +test_invoke_with_invalid_type()
        +test_invoke_plugin_api(capsys)
        +test_pyargs_importerror(pytester, monkeypatch)
        +test_cmdline_python_package(pytester, monkeypatch)
        +... (more)
    }

    class TestDurations {
        +test_calls(pytester, mock_timing)
        +test_calls_show_2(pytester, mock_timing)
        +test_calls_showall(pytester, mock_timing)
        +test_with_deselected(pytester, mock_timing)
        +check_tests_in_output(lines, expected_test_numbers, number_of_tests=3)
    }

    class TestDurationsWithFixture {
        +test_setup_function(pytester, mock_timing)
    }

    TestGeneralUsage --> Pytester : uses
    TestInvocationVariants --> Pytester : uses
    TestDurations --> Pytester : uses
    TestDurationsWithFixture --> Pytester : uses

Summary

The `acceptance_test.py` file is an extensive acceptance test suite for the `pytest` testing framework itself. It:

• Validates error handling and reporting for configuration, import, and runtime issues.

• Tests plugin discovery, loading order, and invocation.

• Verifies duration reporting and fixture timing.

• Covers command-line invocation variants and edge cases.

• Ensures warnings and deprecations are emitted as expected.

• Uses Pytester to create isolated test environments and verify pytest's end-to-end behavior.

This file is essential for maintaining the quality and reliability of `pytest` as a testing tool, by simulating diverse real-world usage scenarios and failure modes.

End of Documentation for acceptance_test.py