unraisableexception.py
Overview
The [unraisableexception.py](/projects/286/67428) module in pytest is dedicated to **capturing, managing, and reporting unraisable exceptions** that occur during test execution. Unraisable exceptions are errors that happen in contexts where Python cannot raise them normally, such as during object finalization (`__del__`) or in background cleanup, where exceptions are simply logged or ignored by the interpreter.
Python 3.4+ introduced `sys.unraisablehook` to intercept these exceptions. This module overrides that hook within pytest to **record unraisable exceptions as structured metadata**, stash them temporarily, and then **emit them as warnings** during test phases. This mechanism improves test diagnostics by surfacing otherwise silent failures, integrating them seamlessly into pytest’s warning and reporting system.
The module also performs additional **garbage collection iterations** during cleanup to help flush out any lingering unraisable exceptions before the test session ends.
Detailed Explanation of Components
Constants and Keys
gc_collect_iterations_key: StashKey[int]
A stash key used to store the number of garbage collection iterations performed during cleanup. This is configurable and defaults to 5.[unraisable_exceptions: StashKey[collections.deque[UnraisableMeta | BaseException]]](/projects/286/67382)
A stash key that holds a deque of captured unraisable exceptions or errors encountered during their processing. This queue collects metadata for deferred processing.
Class: UnraisableMeta
class UnraisableMeta(NamedTuple):
msg: str
cause_msg: str
exc_value: BaseException | None
Purpose:
Encapsulates metadata about an unraisable exception captured by the custom unraisable hook.Attributes:
msg: A detailed formatted message including the exception summary, traceback, and memory allocation trace (via tracemalloc). Used for warning messages.cause_msg: A shorter message without the full traceback, used as the cause message when warnings are treated as errors.exc_value: The actual exception instance if available.
Usage:
Instances ofUnraisableMetaare appended to the stash queue during unraisable exception capture and later processed to emit warnings.
Function: gc_collect_harder(iterations: int) -> None
Purpose:
Runs the garbage collector multiple times to ensure that all unreachable objects (including those triggering unraisable exceptions) are finalized.Parameters:
iterations: Number of times to invokegc.collect().
Behavior:
Callsgc.collect()in a loopiterationstimes.Example:
gc_collect_harder(5) # Perform 5 garbage collection cyclesNote:
The number 5 was determined experimentally (by the Trio project) as a reasonable default to flush unraisable exceptions.
Function: unraisable_hook(unraisable: sys.UnraisableHookArgs, /, *, append: Callable[[UnraisableMeta | BaseException], object]) -> None
Purpose:
Custom replacement for Python’ssys.unraisablehook. It intercepts unraisable exceptions, formats detailed diagnostic messages including tracebacks and tracemalloc info, and appends metadata to a stash queue for deferred processing.Parameters:
unraisable: An instance ofsys.UnraisableHookArgscontaining details about the unraisable exception event.append: A callable (usually a deque’sappendmethod) to store the captured metadata or errors.
Behavior:
Constructs a summary message describing the unraisable exception and the object it occurred on.
Formats the exception traceback and adds tracemalloc memory allocation information.
Creates an
UnraisableMetainstance with this data and appends it to the stash.If an error occurs during processing, appends the error itself and re-raises it to avoid silent failures.
Example Usage:
Installed internally by pytest during configuration; not called directly by user code.
Function: collect_unraisable(config: Config) -> None
Purpose:
Processes all accumulated unraisable exceptions from the stash, converting them into warnings (PytestUnraisableExceptionWarning) or errors if warnings are treated as errors.Parameters:
config: The pytestConfigobject containing the stash with unraisable exception metadata.
Behavior:
Pops all unraisable exception metadata from the stash queue.
For each:
If metadata is an exception instance, wraps it into a
RuntimeErrorand collects it as an error.Otherwise, emits a warning with the detailed message.
If warnings are treated as errors (e.g.,
-Werror), raises the corresponding exceptions.
If multiple errors/warnings were encountered, raises an
ExceptionGroupto aggregate them.
Example:
Called internally at various test phases to flush unraisable exceptions.
Function: cleanup(*, config: Config, prev_hook: Callable[[sys.UnraisableHookArgs], object]) -> None
Purpose:
Cleanup function registered during pytest configuration to run after tests complete. It restores the originalsys.unraisablehook, performs multiple garbage collection iterations, collects unraisable exceptions, and clears the stash.Parameters:
config: pytestConfigobject.prev_hook: The originalsys.unraisablehookfunction to restore.
Behavior:
Runs multiple garbage collection cycles (
gc_collect_harder).Calls
collect_unraisableto emit warnings for any unprocessed exceptions.Restores
sys.unraisablehookto its original function.Deletes the stash key for unraisable exceptions to release memory.
Hook Function: pytest_configure(config: Config) -> None
Purpose:
Pytest hook called during configuration. It sets up the unraisable exception capturing system.Behavior:
Saves the current
sys.unraisablehook.Creates a new deque to stash unraisable exception metadata.
Registers the
cleanupfunction to run after tests finish.Overrides
sys.unraisablehookwith the module's customunraisable_hook, passing the deque’sappendmethod to collect exceptions.
Example:
Automatically invoked by pytest during startup.
Hook Functions: pytest_runtest_setup, pytest_runtest_call, pytest_runtest_teardown
@pytest.hookimpl(trylast=True)
def pytest_runtest_setup(item: Item) -> None:
collect_unraisable(item.config)
@pytest.hookimpl(trylast=True)
def pytest_runtest_call(item: Item) -> None:
collect_unraisable(item.config)
@pytest.hookimpl(trylast=True)
def pytest_runtest_teardown(item: Item) -> None:
collect_unraisable(item.config)
Purpose:
These pytest hooks are called before setup, during test call, and after teardown respectively. They ensure that all unraisable exceptions collected so far are emitted as warnings at safe points during the test lifecycle.Behavior:
Calls
collect_unraisablepassing the test item's configuration.
Importance:
This periodic collection prevents accumulation of unraisable exceptions and surfaces them promptly to users.
Implementation Details and Algorithms
Deferred Exception Processing:
Unraisable exceptions are not immediately turned into warnings but are queued in a stash (acollections.deque). This avoids interfering with the interpreter's internal handling and ensures warnings are raised at appropriate pytest lifecycle points.Garbage Collection Iterations:
Python’s garbage collector is invoked multiple times in a row (gc_collect_harder) to finalize objects that may trigger unraisable exceptions. The number of iterations (default 5) balances thoroughness and performance.Warning Escalation:
When warnings are configured as errors (e.g., pytest’s-Werror), the module raises exceptions from the warnings to fail tests properly. This is handled by catchingPytestUnraisableExceptionWarningand converting them into exceptions with tracebacks.Traceback and Memory Info:
The unraisable exception messages include formatted tracebacks and tracemalloc messages to aid debugging by showing where the exception originated and memory allocation details.Restoration of Hooks:
The originalsys.unraisablehookis restored during cleanup to avoid side effects outside pytest runs.
Interaction with Other System Components
Pytest Config and Stash:
Uses the pytestConfigobject’s stash system to hold unraisable exception data per test session.Pytest Hooks:
Integrates with pytest lifecycle hooks (pytest_configure,pytest_runtest_setup,pytest_runtest_call,pytest_runtest_teardown) to install hooks and process unraisable exceptions at key points.Warnings Framework:
Converts unraisable exceptions intoPytestUnraisableExceptionWarningwarnings, which are handled by pytest’s warning capture and reporting mechanisms.Garbage Collector (
gc):
Callsgc.collect()multiple times to flush objects that might cause unraisable exceptions.Traceback and Tracemalloc:
Leverages pytest’s internaltracemalloc_messageto append memory allocation traces.ExceptionGroup Handling:
Supports grouping multiple errors with Python 3.11+ExceptionGroupor backport for earlier versions.
Usage Example
This module is used internally by pytest and not typically invoked directly by users. However, the workflow can be summarized as:
# During pytest startup
pytest_configure(config)
# During test execution phases:
pytest_runtest_setup(item) # Collect unraisable exceptions and emit warnings
pytest_runtest_call(item) # Same as above
pytest_runtest_teardown(item) # Same as above
# During cleanup:
cleanup(config=config, prev_hook=original_unraisablehook)
If an unraisable exception occurs during test execution (for instance, in an object's `__del__` method), it will be captured and later emitted as a warning during one of these phases. If warnings are treated as errors, pytest will fail the test accordingly, making hidden errors visible.
Visual Diagram: Class and Function Structure
classDiagram
class UnraisableMeta {
+msg: str
+cause_msg: str
+exc_value: BaseException | None
}
class gc_collect_harder {
+iterations: int
+__call__()
}
class unraisable_hook {
+unraisable: sys.UnraisableHookArgs
+append: Callable
+__call__()
}
class collect_unraisable {
+config: Config
+__call__()
}
class cleanup {
+config: Config
+prev_hook: Callable
+__call__()
}
class pytest_configure {
+config: Config
+__call__()
}
class pytest_runtest_setup {
+item: Item
+__call__()
}
class pytest_runtest_call {
+item: Item
+__call__()
}
class pytest_runtest_teardown {
+item: Item
+__call__()
}
pytest_configure --> unraisable_hook : sets as sys.unraisablehook
pytest_configure --> cleanup : registers as cleanup callback
cleanup --> gc_collect_harder : calls multiple times
cleanup --> collect_unraisable : calls to emit warnings
pytest_runtest_setup --> collect_unraisable : calls to flush exceptions
pytest_runtest_call --> collect_unraisable
pytest_runtest_teardown --> collect_unraisable
Summary
The [unraisableexception.py](/projects/286/67428) module enhances pytest’s ability to detect and report **unraisable exceptions** by:
Installing a custom
sys.unraisablehookto intercept exceptions ignored by Python's runtime.Collecting detailed metadata about these exceptions, including tracebacks and memory profiles.
Deferring processing and converting them into warnings during test lifecycle phases.
Performing aggressive garbage collection to unearth hidden exceptions.
Integrating tightly with pytest’s stash system, hooks, and warnings framework.
Providing a robust mechanism to prevent silent test failures caused by exceptions in finalizers or background cleanup.
This improves the reliability and observability of tests, making otherwise silent errors visible and actionable.