test_doctest.py

Overview

`test_doctest.py` is a comprehensive test suite designed to validate the functionality and robustness of pytest's doctest integration. The file contains numerous test cases that exercise different facets of doctest collection, execution, reporting, and interaction with pytest fixtures and configuration options.

Pytest's doctest support allows users to embed tests in docstrings or text files and run them as part of the test suite. This file ensures that pytest can correctly discover, run, and report doctests from various sources, handle edge cases such as Unicode, whitespace, skipping, failure continuation, and support advanced features like fixture injection and doctest-specific option flags.

Detailed Explanation of Classes and Tests

Class: TestDoctests

This class contains tests focused on the collection and execution of doctests from both text files and Python modules. It verifies that doctests are correctly discovered, executed, and reported by pytest.

• Key Tests:

  • test_collect_testtextfile: Verifies that doctests in text files (.txt) are collected as DoctestItem and associated with a DoctestTextfile parent.

  • test_collect_module_empty: Ensures empty Python modules produce no doctest items.

  • test_collect_module_single_modulelevel_doctest: Checks that a module-level (top-level docstring) doctest is collected properly.

  • test_collect_module_two_doctest_one_modulelevel: Confirms collection of multiple doctests in a module including module-level and function-level docstrings.

  • test_simple_doctestfile: Runs a simple failing doctest in a .txt file and asserts failure detection.

  • test_importmode: Tests doctest execution with namespaced packages and importlib import mode.

  • test_new_pattern and test_multiple_patterns: Validate support for custom glob patterns to select doctest files.

  • test_encoding: Confirms support for various file encodings in doctests.

  • test_doctest_unexpected_exception: Checks correct reporting of unexpected exceptions during doctest execution.

  • test_doctest_outcomes: Verifies handling of skip, xfail, and importorskip within doctests.

  • test_docstring_partial_context_around_error and test_docstring_full_context_around_error: Validate that pytest shows appropriate lines of context around failing doctests.

  • test_doctest_linedata_missing and test_doctest_linedata_on_property: Ensure correct line number reporting for doctest failures, including on properties.

  • test_doctest_unex_importerror_only_txt and test_doctest_unex_importerror_with_module: Confirm correct handling of import errors during doctest collection and execution.

  • test_doctestmodule, test_doctestmodule_with_fixtures, and test_doctestmodule_three_tests: Test doctest execution in Python files with multiple doctests and fixture usage.

  • test_ignored_whitespace and test_non_ignored_whitespace: Test effect of doctest_optionflags on whitespace normalization.

  • test_contains_unicode: Ensures doctests containing non-ASCII characters are handled without internal errors.

  • test_ignore_import_errors_on_doctest: Tests the --doctest-ignore-import-errors option.

  • test_junit_report_for_doctest: Verifies that doctest failures are correctly reported in JUnit XML format.

  • test_unicode_doctest and test_unicode_doctest_module: Test for Unicode handling bugs in doctests.

  • test_print_unicode_value: Checks printing Unicode characters in doctests on Python 2.7.

  • test_reportinfo: Ensures DoctestItem.reportinfo() returns proper line numbers.

  • test_valid_setup_py and test_invalid_setup_py: Test that valid setup.py files are ignored and invalid ones collected for doctests.

  • test_setup_module: Regression test ensuring setup_module is called with --doctest-modules.

  • test_main_py_does_not_cause_import_errors: Test that main.py files are handled without import errors.

Class: TestLiterals

Tests related to literal output handling in doctests, focusing on Unicode and bytes literals, and the [NUMBER](/projects/286/67223) option flag that allows approximate number matching.

• Key Tests:

  • test_allow_unicode & test_allow_bytes: Verify that doctest output containing unicode or bytes passes when respective flags (ALLOW_UNICODE or ALLOW_BYTES) are enabled via ini file or inline comment.

  • test_unicode_string & test_bytes_literal: Confirm failure behavior when Unicode or bytes output are present without the appropriate flags.

  • test_number_re: Tests the regular expression used to detect number literals for approximate matching.

  • test_number_precision: Tests the NUMBER option flag with various floating point and scientific notation values, including lists.

  • test_number_non_matches: Tests known cases where approximate number matching should fail.

  • test_number_and_allow_unicode: Combined test for multiple doctest flags in output.

Class: TestDoctestSkips

Focuses on the behavior of skipped tests within doctests, ensuring that:

• If all examples in a doctest are skipped, the entire test is marked as skipped rather than passed.

• Partial skipping behaves correctly.

• Tests continue on failure if --doctest-continue-on-failure is used.

• Skipping works correctly for decorated doctests.

Class: TestDoctestAutoUseFixtures

Tests the integration of pytest's autouse fixtures with doctests, including:

• Ensuring session, module, class, and function scoped autouse fixtures are invoked for doctest modules and text files.

• Confirming that autouse fixtures can be detected and used even when defined in the same module as the doctest.

• Checking that request attributes (module, cls, function) behave as expected inside autouse fixtures when running doctests.

Class: TestDoctestNamespaceFixture

Verifies that fixtures injecting variables into the doctest namespace work correctly for both text file doctests and Python module docstring doctests across different fixture scopes.

Class: TestDoctestReportingOption

Tests different doctest diff reporting styles supported by pytest when showing doctest failures:

• udiff (unified diff)

• cdiff (context diff)

• ndiff (ndiff)

• none and only_first_failure (minimal reporting)

Also tests invalid reporting option handling.

Standalone Tests

• test_doctest_mock_objects_dont_recurse_missbehaved: Ensures mock objects do not cause recursion errors during doctest collection.

• test_warning_on_unwrap_of_broken_object: Verifies that warnings are emitted when unwrapping broken objects with inspect.unwrap patched for mock-awareness.

• test_is_setup_py_not_named_setup_py, test_is_setup_py_is_a_setup_py, test_is_setup_py_different_encoding: Tests to detect if a file is a valid setup.py by detecting the presence of setup() calls from setuptools or distutils.

• test_is_main_py: Checks if a file is a main.py.

Important Implementation Details and Algorithms

• Doctest Collection: Tests verify that pytest correctly collects doctests from various file types (.txt, .py), module-level docstrings, and function/class docstrings.

• Fixture Injection: Autouse fixtures and doctest_namespace are injected into doctest execution namespaces, allowing tests to use pytest fixtures seamlessly.

• Option Flags: Support for doctest option flags such as ALLOW_UNICODE, ALLOW_BYTES, and NUMBER are thoroughly tested, including their application via ini files or inline comments.

• Error Reporting: Different doctest diff formats (udiff, cdiff, ndiff, none) are supported and tested to improve clarity of failure reports.

• Unicode and Encoding Support: Tests verify that doctests handle Unicode literals and encodings properly, including legacy Python 2 compatibility cases.

• Import Error Handling: Tests specifically check that import errors during doctest collection or execution are handled gracefully or cause expected failures/skips.

• Setup.py Handling: Special handling ensures that valid setup.py files do not cause test collection, preventing noise during doctest runs.

• Line Number Accuracy: Tests ensure accurate line number reporting in failure messages, even for doctests defined on properties or overridden properties.

• Continuation on Failure: The --doctest-continue-on-failure flag is tested to confirm pytest continues running subsequent tests after a failure.

Interaction with Other Parts of the System

• Integration with Pytest Core: These tests use the Pytester utility to create test files and invoke pytest programmatically. They exercise the core pytest doctest plugin components such as DoctestItem, DoctestModule, and DoctestTextfile.

• Imports from _pytest.doctest: The file imports internal helpers and classes for doctest handling, such as _get_checker, _is_main_py, and _is_setup_py.

• Fixture System: Tests depend on pytest fixture mechanisms, including autouse fixtures and doctest_namespace fixtures to inject variables into doctests.

• Warning and Error Reporting: Tests verify interactions with pytest's warning capture and failure reporting infrastructure.

• Filesystem and Path Handling: Using pytester and pathlib.Path for creating test files and directories, including handling encoding and file naming conventions.

Usage Examples

Example: Running a simple doctest in a text file and checking failure

def test_simple_doctestfile(self, pytester: Pytester):
    p = pytester.maketxtfile(
        test_doc="""
        >>> x = 1
        >>> x == 1
        False
    """
    )
    reprec = pytester.inline_run(p)
    reprec.assertoutcome(failed=1)

This test creates a `.txt` file with a failing doctest (`x == 1` expected to be `False`) and asserts that pytest detects the failure.

Example: Testing fixture injection into doctest namespace

def test_namespace_doctestfile(self, pytester, scope):
    pytester.makeconftest(
        f"""
        import pytest
        import contextlib

        @pytest.fixture(autouse=True, scope="{scope}")
        def add_contextlib(doctest_namespace):
            doctest_namespace['cl'] = contextlib
    """
    )
    p = pytester.maketxtfile(
        """
        >>> print(cl.__name__)
        contextlib
    """
    )
    reprec = pytester.inline_run(p)
    reprec.assertoutcome(passed=1)

This test ensures that a fixture injecting `contextlib` under the name `cl` into the doctest namespace works correctly in a text file doctest.

Mermaid Class Diagram

classDiagram
    class TestDoctests {
        +test_collect_testtextfile(pytester)
        +test_collect_module_empty(pytester)
        +test_collect_module_single_modulelevel_doctest(pytester)
        +test_collect_module_two_doctest_one_modulelevel(pytester)
        +test_simple_doctestfile(pytester)
        +test_importmode(pytester)
        +test_new_pattern(pytester)
        +test_multiple_patterns(pytester)
        +test_encoding(pytester, test_string, encoding)
        +test_doctest_unexpected_exception(pytester)
        +test_doctest_outcomes(pytester)
        +test_docstring_partial_context_around_error(pytester)
        +test_docstring_full_context_around_error(pytester)
        +test_doctest_linedata_missing(pytester)
        +test_doctest_linedata_on_property(pytester)
        +test_doctest_no_linedata_on_overridden_property(pytester)
        +test_doctest_unex_importerror_only_txt(pytester)
        +test_doctest_unex_importerror_with_module(pytester)
        +test_doctestmodule(pytester)
        +test_doctest_cached_property(pytester)
        +test_doctestmodule_external_and_issue116(pytester)
        +test_txtfile_failing(pytester)
        +test_txtfile_with_fixtures(pytester)
        +test_txtfile_with_usefixtures_in_ini(pytester)
        +test_doctestmodule_with_fixtures(pytester)
        +test_doctestmodule_three_tests(pytester)
        +test_doctestmodule_two_tests_one_fail(pytester)
        +test_ignored_whitespace(pytester)
        +test_non_ignored_whitespace(pytester)
        +test_ignored_whitespace_glob(pytester)
        +test_non_ignored_whitespace_glob(pytester)
        +test_contains_unicode(pytester)
        +test_ignore_import_errors_on_doctest(pytester)
        +test_junit_report_for_doctest(pytester)
        +test_unicode_doctest(pytester)
        +test_unicode_doctest_module(pytester)
        +test_print_unicode_value(pytester)
        +test_reportinfo(pytester)
        +test_valid_setup_py(pytester)
        +test_main_py_does_not_cause_import_errors(pytester)
        +test_invalid_setup_py(pytester)
        +test_setup_module(pytester)
    }

    class TestLiterals {
        +test_allow_unicode(pytester, config_mode)
        +test_allow_bytes(pytester, config_mode)
        +test_unicode_string(pytester)
        +test_bytes_literal(pytester)
        +test_number_re()
        +test_number_precision(pytester, config_mode)
        +test_number_non_matches(pytester, expression, output)
        +test_number_and_allow_unicode(pytester)
    }

    class TestDoctestSkips {
        +test_one_skipped(pytester, makedoctest)
        +test_one_skipped_failed(pytester, makedoctest)
        +test_all_skipped(pytester, makedoctest)
        +test_vacuous_all_skipped(pytester, makedoctest)
        +test_continue_on_failure(pytester)
        +test_skipping_wrapped_test(pytester)
    }

    class TestDoctestAutoUseFixtures {
        +test_doctest_module_session_fixture(pytester)
        +test_fixture_scopes(pytester, scope, enable_doctest)
        +test_fixture_module_doctest_scopes(pytester, scope, autouse, use_fixture_in_doctest)
        +test_auto_use_request_attributes(pytester, scope)
        +test_auto_use_defined_in_same_module(pytester, scope)
    }

    class TestDoctestNamespaceFixture {
        +test_namespace_doctestfile(pytester, scope)
        +test_namespace_pyfile(pytester, scope)
    }

    class TestDoctestReportingOption {
        +_run_doctest_report(pytester, format)
        +test_doctest_report_udiff(pytester, format)
        +test_doctest_report_cdiff(pytester)
        +test_doctest_report_ndiff(pytester)
        +test_doctest_report_none_or_only_first_failure(pytester, format)
        +test_doctest_report_invalid(pytester)
    }

    TestDoctests <|-- TestLiterals
    TestLiterals <|-- TestDoctestSkips
    TestDoctestSkips <|-- TestDoctestAutoUseFixtures
    TestDoctestAutoUseFixtures <|-- TestDoctestNamespaceFixture
    TestDoctestNamespaceFixture <|-- TestDoctestReportingOption

Summary

`test_doctest.py` is a critical component of pytest's test suite that validates the doctest plugin's capabilities across a wide range of scenarios. It ensures that doctests are robustly collected, executed, and reported, including with advanced pytest features like fixtures, option flags, encoding support, and custom reporting formats. The file also guards against regressions related to import errors, Unicode handling, and test skipping logic.

This file interacts closely with pytest's core testing and fixture system and serves as an essential guardrail for maintaining the quality and functionality of pytest's doctest integration.

End of Documentation for test_doctest.py