parametrize.rst

Overview

This file provides comprehensive documentation and practical examples on how to perform test parametrization using the `pytest` testing framework. Parametrization in pytest enables running the same test function with different sets of input values, improving test coverage and reducing code duplication.

The documentation covers a wide range of scenarios and techniques for parametrizing tests, including:

• Basic parametrization using fixtures and command line options.

• Customization of test IDs for better test identification.

• Integration with external scenario-based testing frameworks.

• Deferring expensive resource setup until test runtime.

• Indirect parametrization to combine fixtures with parameters.

• Parametrizing test methods via class-level configurations.

• Combining multiple fixtures with parameters.

• Handling optional imports and implementations gracefully.

• Marking and identifying individual parametrized test cases.

• Testing conditional exception raising within parametrized tests.

This file is primarily instructional and serves as a guide for users to effectively use pytest's parametrization capabilities to write flexible, maintainable, and expressive test suites.

Detailed Explanations

Parametrizing Tests Based on Command Line Options

**Purpose:** Run a test multiple times with different parameter values that depend on a command line flag.

**Example:**

# test_compute.py
def test_compute(param1):
    assert param1 < 4

# conftest.py
def pytest_addoption(parser):
    parser.addoption("--all", action="store_true", help="run all combinations")

def pytest_generate_tests(metafunc):
    if "param1" in metafunc.fixturenames:
        if metafunc.config.getoption("all"):
            end = 5
        else:
            end = 2
        metafunc.parametrize("param1", range(end))

**Explanation:**

• pytest_addoption adds a CLI option --all.

• pytest_generate_tests dynamically parametrizes param1 over different ranges depending on whether --all was specified.

• Running without --all runs tests for param1 in [0, 1].

• Running with --all runs tests for param1 in [0, 1, 2, 3, 4].

**Usage:**

$ pytest -q test_compute.py         # runs 2 test cases
$ pytest -q --all test_compute.py   # runs 5 test cases, last fails due to assertion

Customizing Test IDs in Parametrization

**Purpose:** Control how each parameter set is represented in test IDs, which helps in selective test runs and clearer failure reports.

**Key Points:**

• By default, pytest uses string representations of parameter values.

• IDs can be manually specified as a list of strings or via a callable function.

• pytest.param allows specifying IDs and marks on individual parameter sets.

**Example:**

import pytest
from datetime import datetime, timedelta

testdata = [
    (datetime(2001, 12, 12), datetime(2001, 12, 11), timedelta(1)),
    (datetime(2001, 12, 11), datetime(2001, 12, 12), timedelta(-1)),
]

# Default IDs generated by pytest
@pytest.mark.parametrize("a,b,expected", testdata)
def test_timedistance_v0(a, b, expected):
    assert a - b == expected

# Custom string IDs
@pytest.mark.parametrize("a,b,expected", testdata, ids=["forward", "backward"])
def test_timedistance_v1(a, b, expected):
    assert a - b == expected

# Function-based ID generation for datetime objects
def idfn(val):
    if isinstance(val, datetime):
        return val.strftime("%Y%m%d")

@pytest.mark.parametrize("a,b,expected", testdata, ids=idfn)
def test_timedistance_v2(a, b, expected):
    assert a - b == expected

# Using pytest.param with explicit IDs
@pytest.mark.parametrize(
    "a,b,expected",
    [
        pytest.param(datetime(2001, 12, 12), datetime(2001, 12, 11), timedelta(1), id="forward"),
        pytest.param(datetime(2001, 12, 11), datetime(2001, 12, 12), timedelta(-1), id="backward"),
    ],
)
def test_timedistance_v3(a, b, expected):
    assert a - b == expected

Using pytest_generate_tests to Port TestScenarios

**Purpose:** Support parameterized tests based on scenarios defined as class-level data, commonly used in unittest extensions, now adapted for pytest.

**Example:**

def pytest_generate_tests(metafunc):
    idlist = []
    argvalues = []
    for scenario in metafunc.cls.scenarios:
        idlist.append(scenario[0])
        items = scenario[1].items()
        argnames = [x[0] for x in items]
        argvalues.append([x[1] for x in items])
    metafunc.parametrize(argnames, argvalues, ids=idlist, scope="class")

scenario1 = ("basic", {"attribute": "value"})
scenario2 = ("advanced", {"attribute": "value2"})

class TestSampleWithScenarios:
    scenarios = [scenario1, scenario2]

    def test_demo1(self, attribute):
        assert isinstance(attribute, str)

    def test_demo2(self, attribute):
        assert isinstance(attribute, str)

**Explanation:**

• The pytest_generate_tests hook inspects the class attribute scenarios.

• It extracts names and values to parametrize test methods.

• Tests run once per scenario with IDs "basic" and "advanced".

Deferring Setup of Parametrized Resources

**Purpose:** Avoid expensive resource setup during test collection by combining parametrization with fixtures.

**Example:**

# conftest.py
import pytest

def pytest_generate_tests(metafunc):
    if "db" in metafunc.fixturenames:
        metafunc.parametrize("db", ["d1", "d2"], indirect=True)

class DB1:
    pass

class DB2:
    pass

@pytest.fixture
def db(request):
    if request.param == "d1":
        return DB1()
    elif request.param == "d2":
        return DB2()
    else:
        raise ValueError("invalid internal test config")

# test_backends.py
def test_db_initialized(db):
    if db.__class__.__name__ == "DB2":
        pytest.fail("deliberately failing for demo purposes")

**Explanation:**

• Parametrization is done on db with indirect=True so the fixture db receives the parameter.

• The expensive setup (creating DB objects) happens inside the fixture at test execution time.

• Tests are run once per DB version, allowing selective failure and clear reporting.

Indirect Parametrization

**Purpose:** Use fixtures to process or transform parameter values before running the test.

**Example:**

import pytest

@pytest.fixture
def fixt(request):
    return request.param * 3

@pytest.mark.parametrize("fixt", ["a", "b"], indirect=True)
def test_indirect(fixt):
    assert len(fixt) == 3

**Explanation:**

• The fixt fixture multiplies the input parameter by 3.

• Parametrization passes "a", "b" as inputs indirectly to the fixture.

• The test asserts the length of the processed fixture value.

Applying Indirect Parametrization to Specific Arguments

**Example:**

import pytest

@pytest.fixture
def x(request):
    return request.param * 3

@pytest.fixture
def y(request):
    return request.param * 2

@pytest.mark.parametrize("x, y", [("a", "b")], indirect=["x"])
def test_indirect(x, y):
    assert x == "aaa"
    assert y == "b"

**Explanation:**

• indirect is applied only to argument x.

• x is processed by its fixture; y remains as raw value.

• Demonstrates fine-grained control over which arguments get indirect parametrization.

Parametrizing Test Methods via Class Configuration

**Example:**

import pytest

def pytest_generate_tests(metafunc):
    funcarglist = metafunc.cls.params[metafunc.function.__name__]
    argnames = sorted(funcarglist[0])
    metafunc.parametrize(
        argnames, [[funcargs[name] for name in argnames] for funcargs in funcarglist]
    )

class TestClass:
    params = {
        "test_equals": [dict(a=1, b=2), dict(a=3, b=3)],
        "test_zerodivision": [dict(a=1, b=0)],
    }

    def test_equals(self, a, b):
        assert a == b

    def test_zerodivision(self, a, b):
        with pytest.raises(ZeroDivisionError):
            a / b

**Explanation:**

• Class-level params dictionary maps test method names to parameter sets.

• The pytest_generate_tests hook reads these and parametrizes accordingly.

• Provides a way to organize test parameters alongside test methods in classes.

Parametrization with Multiple Fixtures

**Purpose:** Test serialization/deserialization with various Python interpreters and objects.

**Description:** The file includes a real-life multipython example (not shown here) that runs tests combining multiple interpreters and objects, skipping tests if interpreters are missing.

Parametrizing Optional Implementations or Imports

**Example:**

# conftest.py
import pytest

@pytest.fixture(scope="session")
def basemod(request):
    return pytest.importorskip("base")

@pytest.fixture(scope="session", params=["opt1", "opt2"])
def optmod(request):
    return pytest.importorskip(request.param)

# base.py
def func1():
    return 1

# opt1.py
def func1():
    return 1.0001

# test_module.py
def test_func1(basemod, optmod):
    assert round(basemod.func1(), 3) == round(optmod.func1(), 3)

**Explanation:**

• Fixtures import different implementations, skipping tests if modules are missing.

• Tests compare functionality across multiple implementations.

• Skipped tests are reported accordingly.

Setting Marks or Test IDs for Individual Parameter Sets

**Example:**

import pytest

@pytest.mark.parametrize(
    "test_input,expected",
    [
        ("3+5", 8),
        pytest.param("1+7", 8, marks=pytest.mark.basic),
        pytest.param("2+4", 6, marks=pytest.mark.basic, id="basic_2+4"),
        pytest.param("6*9", 42, marks=[pytest.mark.basic, pytest.mark.xfail], id="basic_6*9"),
    ],
)
def test_eval(test_input, expected):
    assert eval(test_input) == expected

**Explanation:**

• Individual parameter sets can be marked with custom or built-in pytest markers.

• IDs can be assigned to parameter sets for clearer test naming.

• Markers enable selective test execution using -m option.

• Tests expected to fail (xfail) are reported but do not cause overall failure.

Parametrizing Conditional Exception Raising

**Example:**

from contextlib import nullcontext
import pytest

@pytest.mark.parametrize(
    "example_input,expectation",
    [
        (3, nullcontext(2)),
        (2, nullcontext(3)),
        (1, nullcontext(6)),
        (0, pytest.raises(ZeroDivisionError)),
    ],
)
def test_division(example_input, expectation):
    with expectation as e:
        assert (6 / example_input) == e

**Explanation:**

• Some tests expect exceptions (pytest.raises) while others expect a value (nullcontext).

• This pattern allows writing tests that handle both normal and exceptional cases cleanly.

• The with statement manages the expected exception context or passes through the expected value.

Implementation Details and Algorithms

• The file relies heavily on pytest's hook system, particularly pytest_generate_tests, to dynamically generate test parameter combinations.

• Indirect parametrization leverages pytest fixtures with request.param to defer or customize resource setup based on parameters.

• Custom test IDs are generated using string lists, callable functions, or pytest.param constructs to improve test reporting and selection.

• The example of integrating with testscenarios demonstrates adapting external scenario-based test patterns into pytest's parametrization mechanism.

• Marking individual parameter sets provides fine-grained control over test execution and reporting, supporting expected failures and selective runs.

• Use of contextlib.nullcontext in conditional exception tests elegantly handles cases where exceptions are not expected but the test logic remains uniform.

Interaction with Other Parts of the System

• Fixtures and conftest.py: Many examples show how parametrization interacts with fixtures defined in conftest.py for resource setup.

• Command Line Interface: Parametrization can respond to pytest command line options, such as --all, to control test ranges.

• Markers and Test Selection: Markers applied via parametrization influence test selection through pytest CLI options (-m).

• Integration with External Modules: The optional imports example shows interaction with external modules (base, opt1, opt2) and how missing modules are handled gracefully.

• Testing Framework Extensions: The scenario-based parametrization example adapts unittest extension patterns to pytest.

• Test Reporting: Custom test IDs and marks improve the clarity of test reports and failure diagnostics.

Usage Summary

• Add parametrization using @pytest.mark.parametrize or metafunc.parametrize inside pytest_generate_tests.

• Customize test IDs for easier identification and filtering.

• Use indirect parametrization to combine fixtures with parameters.

• Apply marks on individual parameter sets to control test behavior and selection.

• Handle exceptions conditionally within parametrized tests.

• Leverage class-level dictionaries or scenario lists to organize parameter sets for test methods.

Visual Diagram of File Structure and Relationships

flowchart TD
    A[pytest_generate_tests hook] -->|generates| B[Parametrized test cases]
    B --> C{Parametrization Types}
    C --> D[Direct parameters]
    C --> E[Indirect parameters (fixtures)]
    C --> F[Class-level params dict]
    C --> G[Scenario-based params]

    E --> H[Fixtures with request.param]
    G --> I[Scenario tuples: (id, dict)]

    B --> J[Custom test IDs]
    B --> K[pytest.param with marks and IDs]

    L[Command Line Options] --> A
    M[Markers] --> K
    N[Optional Imports] --> H

    style A fill:#f9f,stroke:#333,stroke-width:2px
    style B fill:#bbf,stroke:#333,stroke-width:1px
    style C fill:#afa,stroke:#333,stroke-width:1px

*Diagram Explanation:*

• The central node is pytest_generate_tests, which orchestrates test parametrization.

• Parametrization splits into direct parameters, indirect parameters via fixtures, class-level parameter dictionaries, and scenario-based parameters.

• Fixtures use request.param to handle indirect parameters, including optional imports.

• Custom test IDs and pytest.param support enhance test case identification and marking.

• Command line options and markers influence the parametrization and selection process.

This documentation equips users with both conceptual understanding and practical code snippets to master pytest parametrization for robust and maintainable testing workflows.