test_reporting.py

Overview

The [test_reporting.py](/projects/286/66996) file contains a comprehensive suite of automated tests designed to verify and validate logging and reporting functionalities within the pytest framework. These tests primarily cover how pytest handles capturing, displaying, and storing log messages generated during test execution, including standard output/error, Python logging calls, and live logging output.

Key testing scenarios include:

• Capturing stdout and stderr output during test failures.

• Capturing Python logging messages at different log levels.

• Interaction and precedence between CLI logging options (e.g., --log-level, --log-cli-level).

• Logging during different test phases (setup, call, teardown).

• Live logging output with correct section headers and log message formatting.

• Logging to files with different modes, levels, and format configurations.

• Behavior of logger disabling and propagation.

• Handling of logging errors and ANSI color codes.

• Integration of logging within pytest hooks and collection phases.

• Ensuring correctness of log file creation, encoding, and path management.

The file leverages the `pytester` testing utility from pytest for creating temporary test files, running pytest with various configurations, and asserting on the outputs.

Detailed Explanation of Key Functions

Each function in this file is a test case for a specific logging/reporting feature. They follow the pytest convention for test functions and use fixtures like `pytester` and `request` for test environment setup.

test_nothing_logged(pytester: Pytester) -> None

• Purpose: Verifies that when a test writes to stdout and stderr but no logging calls are made, the captured output is correctly shown in the pytest output.

• Parameters:

  • pytester: Fixture providing pytest testing utilities.

• Behavior:

  • Creates a test file that writes to stdout and stderr and fails.

  • Runs pytest and asserts that the captured stdout and stderr are shown.

  • Asserts that no log capture section appears.

• Usage:

  def test_nothing_logged(pytester):
      ...
  

• Returns: None, raises assertion errors if expectations are not met.

test_messages_logged(pytester: Pytester) -> None

• Purpose: Checks that log messages are captured and shown along with stdout and stderr when logging is enabled at INFO level.

• Parameters:

  • pytester: Pytester fixture.

• Behavior:

  • Creates a test that writes to stdout, stderr, and logs an info message.

  • Runs pytest with --log-level=INFO.

  • Asserts captured log, stdout, and stderr outputs appear.

• Returns: None.

test_root_logger_affected(pytester: Pytester) -> None

• Purpose: Ensures that setting the root logger level using --log-level affects which log messages are captured and written to the log file.

• Parameters:

  • pytester: Pytester fixture.

• Behavior:

  • Creates test that logs at info, warning, and error levels.

  • Runs pytest with --log-level=ERROR and --log-file=pytest.log.

  • Asserts only error messages appear in captured output and log file.

• Returns: None.

test_log_cli_level_log_level_interaction(pytester: Pytester) -> None

• Purpose: Tests how --log-cli-level and --log-level interact, ensuring live logs honor the CLI level settings.

• Parameters:

  • pytester: Pytester fixture.

• Behavior:

  • Logs messages at various levels.

  • Runs pytest with --log-cli-level=INFO and --log-level=ERROR.

  • Asserts live log output contains INFO and above, but not DEBUG.

• Returns: None.

test_setup_logging(pytester: Pytester) -> None

• Purpose: Validates that logging messages emitted during test setup are captured and displayed properly.

• Behavior:

  • Creates a test file with logging in setup_function and the test call.

  • Runs pytest with logging enabled.

  • Checks both setup and call logs are shown.

• Returns: None.

test_teardown_logging(pytester: Pytester) -> None

• Purpose: Similar to setup logging test but for teardown phase.

• Behavior:

  • Logs messages in test call and teardown function.

  • Failing teardown to confirm capture.

• Returns: None.

test_log_cli_enabled_disabled(pytester: Pytester, enabled: bool) -> None

• Purpose: Tests enabling and disabling of live logging via log_cli ini option.

• Parameters:

  • enabled: Boolean controlling whether live logging is enabled.

• Behavior:

  • Creates a test that logs a critical message.

  • Configures ini to enable or disable log_cli.

  • Asserts message appears only if enabled.

• Returns: None.

test_log_cli_default_level(pytester: Pytester) -> None

• Purpose: Ensures the default live log level is NOTSET, and only appropriate messages show.

• Behavior:

  • Checks that INFO messages at default level are filtered out unless warning or higher.

• Returns: None.

test_log_cli_default_level_multiple_tests(pytester: Pytester, request: FixtureRequest) -> None

• Purpose: Validates live logging resets newlines properly between multiple tests.

• Behavior:

  • Runs two tests with warnings.

  • Examines the output for proper sectioning.

• Returns: None.

test_live_logs_unknown_sections(pytester: Pytester, request: FixtureRequest) -> None

• Purpose: Tests live logging output when unknown logging sections occur in hooks.

• Behavior:

  • Injects a hook that logs an unknown section.

  • Validates that unknown section message is captured.

• Returns: None.

test_log_cli_level(pytester: Pytester) -> None

• Purpose: Tests the CLI option --log-cli-level sets the live logging level correctly.

• Behavior:

  • Confirms logs below CLI level are not shown.

• Returns: None.

test_log_file_cli(pytester: Pytester) -> None

• Purpose: Tests logging to a file via CLI options with specified log level.

• Behavior:

  • Ensures only messages at or above level are written.

• Returns: None.

test_log_file_mode_cli(pytester: Pytester) -> None

• Purpose: Checks file mode options such as append mode work properly for log files.

• Behavior:

  • Writes a header to file.

  • Ensures logs append to existing content.

• Returns: None.

test_log_file_mode_cli_invalid(pytester: Pytester) -> None

• Purpose: Verifies invalid file mode CLI option causes usage error exit.

• Returns: None.

test_log_file_unicode(pytester: Pytester) -> None

• Purpose: Confirms that Unicode characters are handled correctly in log files.

• Returns: None.

test_live_logging_suspends_capture(has_capture_manager: bool, request: FixtureRequest) -> None

• Purpose: Tests the low-level implementation detail that live logging suspends pytest's capture manager.

• Parameters:

  • has_capture_manager: Whether a capture manager is present.

• Behavior:

  • Mocks capture manager and verifies context manager calls around live logging.

• Returns: None.

test_collection_live_logging(pytester: Pytester) -> None

• Purpose: Checks live logging output during the pytest collection phase.

• Returns: None.

test_log_in_hooks(pytester: Pytester) -> None

• Purpose: Validates logging inside pytest hook implementations (e.g., sessionstart/finish).

• Returns: None.

test_log_set_path(pytester: Pytester) -> None

• Purpose: Tests the ability to dynamically set the log file path per test using the logging plugin API.

• Returns: None.

test_colored_captured_log(pytester: Pytester) -> None

• Purpose: Ensures captured log levels are colored appropriately in terminal output when color is enabled.

• Returns: None.

test_logging_emit_error(pytester: Pytester) -> None

• Purpose: Confirms that logging errors during emit cause test failures by propagating exceptions.

• Returns: None.

test_logging_emit_error_supressed(pytester: Pytester) -> None

• Purpose: Checks that if logging errors are disabled (logging.raiseExceptions = False), pytest does not fail the test.

• Returns: None.

test_disable_loggers(pytester: Pytester) -> None

• Purpose: Tests that loggers can be disabled to suppress output and errors.

• Returns: None.

test_disable_loggers_does_not_propagate(pytester: Pytester) -> None

• Purpose: Validates disabling child loggers does not propagate messages to parent loggers.

• Returns: None.

test_log_disabling_works_with_log_cli(pytester: Pytester) -> None

• Purpose: Checks that disabling loggers works correctly with live logging enabled.

• Returns: None.

test_without_date_format_log(pytester: Pytester) -> None

• Purpose: Ensures that by default, log messages do not include date/time stamps.

• Returns: None.

test_date_format_log(pytester: Pytester) -> None

• Purpose: Validates that custom date format options affect log message timestamps.

• Returns: None.

test_date_format_percentf_log(pytester: Pytester) -> None

• Purpose: Checks that microseconds (%f) are correctly included in log timestamps.

• Returns: None.

test_date_format_percentf_tz_log(pytester: Pytester) -> None

• Purpose: Ensures that timezone and microseconds are formatted correctly in logs.

• Returns: None.

Other Tests

The file also contains multiple tests for:

• Auto-enabling live logs based on CLI arguments.

• Logging during test collection with different verbosity levels.

• Ensuring proper log file creation in nested directories.

• Fallback behaviors for log file formats and levels.

• Proper handling of Unicode in logs.

• Correct interaction of log file modes and levels via ini or CLI.

• Log messages emitted during different pytest phases and hooks.

Implementation Details and Algorithms

• The tests utilize pytester fixture to dynamically generate test scripts and run pytest subprocesses with customized command-line options and configuration files.

• Output assertions rely heavily on fnmatch_lines and re_match_lines for pattern matching of captured log and stdout/stderr content.

• Tests check both the terminal output and log file contents to ensure consistency.

• Parametrization is used to test multiple configurations and boolean options robustly.

• Some tests mock internal pytest logging components to verify the integration and behavior of logging handlers.

• Log disabling tests assert that suppressed loggers produce no stderr or unwanted output.

• Tests cover error propagation by triggering bad logging calls and checking pytest's failure reporting.

• The tests ensure that live logging outputs are correctly sectioned and formatted with appropriate headers for test phases (setup, call, teardown, finish).

Interaction with Other System Components

• pytester: The main driver in these tests, used to create test files, ini files, and run pytest subprocesses.

• pytest Logging Plugin: Many tests interact with the logging-plugin to assert on logger handlers, levels, and emitted messages.

• TerminalReporter: Used internally for capturing and validating test output.

• File System: Tests involving log files interact with the filesystem for reading and writing logs, including tests for creating nested directories.

• pytest Hooks: Several tests define hooks (pytest_runtest_logstart, pytest_sessionstart, etc.) to verify logging integration during pytest's lifecycle.

• Standard Python logging Module: Core to the test scenarios, as these tests verify how pytest captures, displays, and manages Python logging output.

Visual Diagram

flowchart TD
    A[pytester Fixture] -->|Creates test files and ini| B[Dynamic Test Scripts]
    B --> C[Run pytest subprocess]
    C --> D{Logging Features Tested}
    D -->|Capture stdout/stderr| E[Capture Tests]
    D -->|Capture logging calls| F[Logging Capture Tests]
    D -->|CLI options interaction| G[CLI Level Tests]
    D -->|Live logging output| H[Live Logging Tests]
    D -->|Log file handling| I[Log File Tests]
    D -->|Logging in hooks| J[Hook Integration Tests]
    D -->|Logger disabling| K[Disable Logger Tests]
    D -->|Error propagation| L[Logging Error Tests]
    E & F & G & H & I & J & K & L --> M[Assertions on output & file contents]

Summary

[test_reporting.py](/projects/286/66996) is a critical test suite validating pytest's robust logging and reporting capabilities. It ensures that users see correct, well-formatted, and configurable log output during test runs, whether on terminal or in log files. The tests cover a wide range of scenarios from simple stdout capture to complex live logging with dynamic configurations, making this file essential for maintaining the quality and reliability of pytest's logging features.