pythonpath.rst

Overview

This documentation file explains how pytest manages its import mechanisms related to Python's `sys.path` and [PYTHONPATH](/projects/286/67459) during test discovery and execution. Specifically, it details the different *import modes* pytest supports when importing test modules and `conftest.py` configuration files, how these modes affect `sys.path`, module names, and test isolation, and the scenarios where these behaviors are relevant.

This file serves as a guide for pytest users and developers to understand the underlying import logic, avoid common pitfalls related to Python package layouts, and configure pytest appropriately for their project structure and testing needs.

Import Modes

pytest needs to import test modules and `conftest.py` files to execute tests. Because Python import semantics can be complex in test environments, pytest offers three import modes selectable via the `--import-mode` command-line flag:

1. prepend (default)

• Behavior:
  Inserts the directory containing each module at the beginning of sys.path (if not already present), then imports the module with importlib.import_module.

• Recommendations:
  Tests should be arranged as proper Python packages by including __init__.py files. This allows pytest to resolve full module names (e.g., tests.core.test_core).

• Limitations:
  If the test directory tree is not a package, test filenames must be unique across the tree; otherwise, pytest raises an error due to module name collisions.

• Use case:
  Classic and backward-compatible behavior, dating from Python 2 support.

2. append

• Behavior:
  Appends the directory containing each module to the end of sys.path (if not already present), then imports the module with importlib.import_module.

• Advantages:
  Enables testing code against installed package versions rather than local copies. For example, tests in testing/test_pkg_under_test.py will use the installed pkg_under_test if --import-mode=append is specified.

• Limitations:
  Same as prepend mode regarding unique test module names when not using packages.

• Use case:
  Useful when tests are run against installed packages and to avoid confusion between local and installed modules.

3. importlib

• Behavior:
  Imports test modules using importlib directly without modifying sys.path.

• Advantages:

  • Does not change sys.path at all.

  • Test module names do not need to be unique; pytest auto-generates unique names based on the rootdir.

• Disadvantages:

  • Test modules cannot import each other.

  • Test utility modules inside test directories are not importable. Recommended to place such utilities alongside application code.

• Import Algorithm:

  1. Attempts to import by canonical module name derived from the file path (e.g., tests.core.test_models).

  2. If that fails (common for test modules not on sys.path), imports the module directly using importlib with a unique name added to sys.modules.

• Introduced: pytest 6.0.

• Notes:
  Initially considered for the default import mode, but due to limitations, prepend remains the default.

Important Notes

• pytest does not automatically resolve namespace packages by default. This can be enabled via the consider_namespace_packages configuration option.

• Related configuration variables:

  • pythonpath

  • consider_namespace_packages

• See also the pytest documentation on test layout and discovery for more guidance.

Usage Scenarios: prepend and append Modes

Test Modules and conftest.py Inside Packages

Given this directory structure:

root/
|- foo/
   |- __init__.py
   |- conftest.py
   |- bar/
      |- __init__.py
      |- tests/
         |- __init__.py
         |- test_foo.py

Running:

pytest root/

• pytest detects foo.bar.tests.test_foo as a package module because of __init__.py files.

• Inserts root/ at the front of sys.path (if not present).

• Imports test_foo.py as foo.bar.tests.test_foo.

• Imports conftest.py as foo.conftest.

• This preserves full package names, allowing duplicated test file names in different packages.

• Important for proper test discovery and isolation.

Standalone Test Modules and conftest.py Files

Given this directory structure:

root/
|- foo/
   |- conftest.py
   |- bar/
      |- tests/
         |- test_foo.py

Running:

pytest root/

• pytest finds test_foo.py is not part of a package (no __init__.py).

• Adds root/foo/bar/tests to sys.path to import test_foo.py as a top-level module test_foo.

• Adds root/foo to sys.path to import conftest.py as conftest.

• Because modules are imported globally (not within packages), test filenames must be unique across the entire test tree.

• Potential for import collisions if test files share a name.

Invoking pytest: pytest vs python -m pytest

• Running pytest [...] and python -m pytest [...] are nearly equivalent.

• However, python -m pytest adds the current directory to sys.path by default (standard Python behavior).

• This subtle difference can affect import resolution and test discovery.

• See pytest's invoke-python documentation for more information.

Implementation Details and Algorithms

• pytest manipulates sys.path dynamically depending on the import mode.

• Uses importlib.import_module for prepend and append modes.

• For importlib mode, imports modules directly with unique names in sys.modules, avoiding sys.path modification.

• Determines package roots by searching parent directories for __init__.py files.

• Generates canonical module names based on relative file paths from the root directory.

• Enforces test module name uniqueness when necessary to prevent collisions in the global import namespace.

Interactions with Other System Components

• This file/documentation is tightly coupled with pytest's core test discovery and import system.

• Influences how pytest loads conftest.py files, fixtures, and test modules.

• Affects plugin imports and doctest discovery.

• Works alongside configuration options such as pythonpath and consider_namespace_packages.

• Impacts test isolation, namespace pollution, and compatibility with various test directory layouts (e.g., src-layout).

• Affects user workflows depending on how tests are run (CLI, python -m pytest, CI pipelines).

Example Usage

Run pytest with the default import mode (`prepend`):

pytest --import-mode=prepend tests/

Run pytest without modifying `sys.path` (unique module names, isolated imports):

pytest --import-mode=importlib tests/

Run pytest to test installed packages by appending test directories to the end of `sys.path`:

pytest --import-mode=append tests/

Mermaid Diagram: Import Mode Structure and Workflow

flowchart TD
    A[pytest starts test import] --> B{Select import mode}
    B -->|prepend (default)| C[Insert test module dir at start of sys.path]
    B -->|append| D[Insert test module dir at end of sys.path]
    B -->|importlib| E[Import module directly without sys.path changes]

    C --> F[Import module with importlib.import_module]
    D --> F
    E --> G{Try import by canonical name}
    G -->|Success| H[Return imported module]
    G -->|Fail| I[Import module with unique name using importlib]
    I --> J[Add to sys.modules with unique name]
    J --> H

    F & H --> K[pytest executes tests]

    style A fill:#f9f,stroke:#333,stroke-width:2px
    style B fill:#bbf,stroke:#333,stroke-width:2px
    style E fill:#f96,stroke:#333,stroke-width:2px

References

• importlib module: https://docs.python.org/3/library/importlib.html

• pytest configuration options: pythonpath, consider_namespace_packages

• pytest test layout documentation: https://docs.pytest.org/en/stable/example/pythonpath.html#test-layout

• conftest.py and fixtures usage: https://docs.pytest.org/en/stable/how-to/fixtures.html

• src-layout recommendations for Python projects

Summary

This document clarifies how pytest manages Python imports for test modules and configuration files, explaining the implications of different import modes and directory layouts. Understanding this is crucial for writing robust tests, organizing test code correctly, and avoiding common import-related errors in pytest-based projects.