customdirectory.rst

Overview

This documentation describes the **custom directory collector** pattern for pytest, which enables users to customize how pytest collects test files within specific directories. By default, pytest collects directories as either packages (directories containing `__init__.py`) or simple directories. However, some projects require finer control over which files are collected as tests in a directory, for instance, to restrict test collection to an explicit list of files.

This file presents a practical example of implementing a custom directory collector that reads a `manifest.json` file inside the directory. The manifest explicitly lists test files to be collected, which allows deterministic and reproducible test discovery on a per-directory basis.

The approach involves creating a subclass of `pytest.Directory` (named `ManifestDirectory`) and hooking it into the pytest collection process with the `pytest_collect_directory` hook. This collector reads the manifest and yields only the specified test files, ignoring all others in that directory.

Detailed Explanation

Classes and Functions

ManifestDirectory (class)

• Base class: pytest.Directory

• Purpose: Custom directory collector that reads a manifest.json file inside the directory and collects only the test files listed therein.

• Key method:

  • collect(self): Overrides the base class method. It:

    • Loads the manifest JSON file.

    • Extracts the list of files specified under the "files" key.

    • For each listed file, delegates collection to pytest’s internal file collection hook pytest_collect_file.

    • Yields collected nodes for each file.

• Parameters:

  • Inherited from pytest.Directory: path (the directory path), parent (the parent collector).

• Returns: An iterable of collected test file nodes (pytest.Item or further collectors like pytest.Module).

• Usage example:

class ManifestDirectory(pytest.Directory):
    def collect(self):
        manifest_path = self.path / "manifest.json"
        manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
        ihook = self.ihook
        for file in manifest["files"]:
            yield from ihook.pytest_collect_file(
                file_path=self.path / file, parent=self
            )

This class is instantiated automatically by the hook below when a directory with a manifest is encountered.

pytest_collect_directory(path, parent) (hook implementation)

• Type: pytest hook implementation (@pytest.hookimpl)

• Purpose: Intercepts pytest’s directory collection process. Checks whether a manifest.json file exists in the directory being collected. If so, returns a ManifestDirectory instance to handle collection for that directory instead of the default directory collector.

• Parameters:

  • path (py.path.local or pathlib.Path): The directory path being collected.

  • parent: The parent collector node.

• Returns:

  • ManifestDirectory instance if manifest.json is found in path.

  • None to delegate to default collection otherwise.

• Usage example:

@pytest.hookimpl
def pytest_collect_directory(path, parent):
    if path.joinpath("manifest.json").is_file():
        return ManifestDirectory.from_parent(parent=parent, path=path)
    return None

Implementation Details

• Manifest file format:
  The manifest file is expected to be a JSON file named manifest.json within the directory. Its structure is simple:

{
    "files": [
        "test_first.py",
        "test_second.py"
    ]
}

It lists the relative paths of files within the directory that pytest should collect as test modules.

• Collection flow:

  1. When pytest encounters a directory, it calls the pytest_collect_directory hook.

  2. This hook checks if manifest.json exists.

  3. If yes, it returns ManifestDirectory collector for that directory.

  4. ManifestDirectory.collect() reads the manifest and yields collected files listed in it.

  5. Files not listed are ignored (e.g., test_third.py in the example).

  6. If no manifest is found, pytest uses its default directory collector.

• Integration with pytest internals:
  The custom collector uses self.ihook.pytest_collect_file() to leverage pytest’s built-in file collection logic, ensuring that each file is collected according to standard pytest behavior (e.g., detecting test functions and classes).

How This File Interacts with the System

• pytest Collection Framework:
  This file extends pytest’s core test discovery by providing a custom directory collector plugged in via the pytest_collect_directory hook. It integrates transparently with pytest’s collection tree and item generation.

• User Configuration:
  Users place a manifest.json file in directories where they want to customize file collection. They add the ManifestDirectory collector and hook implementation in their conftest.py or pytest plugin file.

• Test Execution:
  Only test files listed in the manifest are collected and run. Other files are excluded from collection, providing precise control over test suite composition.

• Debugging and Introspection:
  Using pytest --collect-only, users can verify the collection tree and confirm that only manifest-listed tests are collected under the ManifestDirectory node.

Example Workflow and Usage

Suppose you have this directory structure:

customdirectory/
├── conftest.py
├── tests/
│   ├── manifest.json
│   ├── test_first.py
│   ├── test_second.py
│   └── test_third.py

• manifest.json lists only test_first.py and test_second.py.

• Running pytest will collect and run tests only from those two files.

• test_third.py will be ignored.

• Running pytest --collect-only will show a collection tree with a ManifestDirectory for tests containing only the two listed modules.

Visual Diagram

Below is a Mermaid class diagram illustrating the main class and hook in this file:

classDiagram
    class ManifestDirectory {
        +collect()
        -path
        -ihook
    }
    class pytest {
        <<hook>>
        +pytest_collect_directory(path, parent)
    }

    pytest ..> ManifestDirectory : creates instance if manifest.json exists

This diagram shows that the `pytest_collect_directory` hook function decides whether to instantiate `ManifestDirectory` based on the presence of a manifest file, and that the `ManifestDirectory` class implements the `collect()` method to yield test modules.

Summary

This file exemplifies a pytest customization that empowers users to define explicit test discovery rules per directory using a manifest JSON file. By leveraging pytest’s extensible collection hooks and subclassing, it enables:

• Deterministic and declarative control over test file collection.

• Seamless integration with pytest’s test collection and execution machinery.

• Improved test suite maintainability in complex projects by avoiding unintended test file execution.

This technique fits naturally into pytest’s plugin architecture and can be extended with richer manifest schemas (e.g., glob patterns, exclusions) or additional directory-based rules.

For further details, see the example `conftest.py` and test files provided in the referenced `customdirectory` example project.