nonpython.rst

Overview

The [nonpython.rst](/projects/286/67218) file serves as documentation for working with non-Python test specifications within a pytest environment, specifically focusing on YAML-formatted test files. It explains how to write, collect, and execute tests defined in YAML files using pytest, through an example `conftest.py` configuration extracted from the `pytest-yamlwsgi` plugin.

This file guides users on how to extend pytest’s testing capabilities beyond Python code, enabling domain-specific testing languages or custom test formats. It highlights mechanisms for test collection, execution, failure reporting, and verbose output formatting for YAML test files.

Key functionalities documented include:

Collecting .yaml test files as pytest test inputs.
Executing YAML-defined tests and interpreting their outcomes.
Customizing test failure representation and verbosity.
Using pytest command-line options to inspect test collection without running tests.

The file does not contain executable code itself but includes references and embedded code snippets illustrating usage and configuration.

Detailed Explanations

YAML Test Files in pytest

Pytest typically runs tests implemented as Python code. However, this documentation explains how to leverage YAML files as test specifications, allowing tests to be described declaratively.

**Example YAML test file snippet:**

sub1: sub1
hello: !fail
  some: other

This file might define two test cases, `sub1` which passes, and `hello` which fails with some custom failure data.

`conftest.py` for YAML Test Collection and Execution

The documentation references a `conftest.py` file (from the `pytest-yamlwsgi` plugin) which:

Collects files matching the pattern test*.yaml.
Parses the YAML content.
Executes tests based on the parsed YAML structure.

This `conftest.py` is responsible for integrating YAML tests into the pytest workflow.

**Usage Example:**

pytest test_simple.yaml

Expected output shows pytest collecting two tests from the YAML file and executing them, reporting pass/fail status accordingly.

Test Failure Representation

repr_failure(excinfo):
This hook/custom method is used to generate a string representation of test failures. When implementing custom test collection nodes (e.g., for YAML tests), you can override this to provide meaningful failure messages.
reportinfo():
Used to represent the location of tests in pytest reporting and verbose output. This helps pytest display where in the YAML file a test is defined.

Pytest Command-Line Options and Output

Running tests normally:
```
pytest test_simple.yaml
```
Produces a summary of passed and failed tests, including failure details.
Running tests with verbose output:
```
pytest -v test_simple.yaml
```
Shows detailed test names and their pass/fail status.
Collecting tests without running:
```
pytest --collect-only test_simple.yaml
```
Displays the test collection tree, showing the hierarchy of test nodes (package, YAML file, test items).

Implementation Details

The integration relies on pytest’s plugin architecture and its collection mechanism.
The conftest.py file uses pytest hooks to:
- Find YAML files.
- Parse their contents using a YAML parser (e.g., PyYAML).
- Create pytest test items from YAML nodes.
- Represent test failures with custom messages.
Tests defined in YAML can be domain-specific, allowing users to define their own test languages or specifications beyond Python code.
The failure message mechanism provides flexibility to represent complex or structured failure data cleanly in pytest reports.

Interactions with Other System Components

pytest core:
The file documents usage within the pytest testing framework, showing how pytest’s collection, execution, and reporting features can be extended for YAML-based tests.
YAML parser (PyYAML or equivalent):
Required to parse the YAML test files so they can be interpreted as test cases.
pytest plugins (e.g., pytest-yamlwsgi):
The example conftest.py is extracted from pytest-yamlwsgi, illustrating how external plugins can provide custom test collection and execution logic.
User test files:
Users create YAML test files (test_simple.yaml) which are collected and run as tests.

Visual Diagram

The following Mermaid class diagram captures the conceptual structure of the pytest collection and execution model as extended for YAML tests, highlighting key components and their relationships within this context.

classDiagram
    class Pytest {
        +collect_tests()
        +run_tests()
        +report_results()
    }
    class Conftest_py {
        +collect_yaml_files()
        +parse_yaml()
        +create_yaml_test_items()
        +repr_failure(excinfo)
        +reportinfo()
    }
    class YamlFile {
        +filepath: str
        +load_contents()
    }
    class YamlTestItem {
        +name: str
        +run()
        +report_failure()
    }
    class PyYAMLParser {
        +load(file)
    }

    Pytest --> Conftest_py : uses
    Conftest_py --> YamlFile : collects
    YamlFile --> PyYAMLParser : parses
    Conftest_py --> YamlTestItem : creates
    YamlTestItem --> Pytest : reports to

Summary

This documentation file [nonpython.rst](/projects/286/67218) provides a focused explanation and usage guide for running non-Python tests in pytest using YAML files. It outlines the necessary configuration, execution model, and reporting mechanisms to enable YAML as a test definition format. This approach empowers test authors to write tests declaratively and extend pytest’s capabilities to custom testing domains.

By following this guide and using the referenced `conftest.py` approach, users can integrate YAML-based tests seamlessly into pytest workflows, benefiting from pytest’s rich features such as test discovery, detailed reporting, and flexible failure presentation.