Automating Python Multi-Version Testing With Tox, Nox and CI/CD

In modern Python development, maintaining compatibility across multiple Python versions is super critical, especially for libraries and tools that target a diverse user base. Here, we explore how to use Tox and Nox, two powerful tools for Python test automation, to validate projects across multiple Python versions. Using a concrete project as an example, we’ll walk through setting up multi-version testing, managing dependencies with Poetry, and using Pytest for unit testing. CI/CD is used for DevOps automation.

Why Automate Multi-Version Testing?

Automating tests across Python versions ensures your project remains robust and reliable in diverse environments. Multi-version testing can:

• Catch compatibility issues early.
• Provide confidence to users that your package works on their chosen Python version.
• Streamline the development process by reducing manual testing.

What Are Tox, Nox, and CI/CD?

Tox

Tox is a well-established tool designed to automate testing in isolated environments for multiple Python versions. It is declarative, with configurations written in an ini file, making it easy to set up for standard workflows.

Nox

Nox is a more flexible alternative that uses Python scripts for configuration, allowing dynamic and programmable workflows. It is based on a python code config file, so it is more robust than Tox to complex project configurations.

Continuous Integration and Continuous Deployment (CI/CD)

CI/CD is a set of practices and tools designed to streamline and automate the software development lifecycle. Continuous Integration (CI) focuses on automatically building and testing code changes as they are integrated into a shared repository, ensuring that new commits do not introduce defects and that the codebase remains stable. Continuous Deployment (CD) extends this automation by automatically deploying validated code to production or staging environments, facilitating rapid and reliable delivery of features and updates. By implementing CI/CD pipelines, teams can enhance collaboration, reduce integration issues, accelerate release cycles, and maintain high code quality, ultimately enabling more efficient and resilient software development processes.

Those tools aim to simplify testing across multiple versions, but their approaches make them suited for different needs.

Project Setup: A Practical Example

Let’s set up a project named tox-nox-python-tests to demonstrate the integration of Tox, Nox, Poetry, and Pytest. The project includes simple addition and subtraction functions, tested across Python versions 3.8 to 3.13.

1. Directory Structure

Here’s how the project is structured:

tox-nox-python-tests/
├── .github
│   └── workflows
│      └── python-tests-matrix.yml
├── tox_nox_python_test_automation/
│   ├── __init__.py
│   ├── main.py
│   └── calculator.py
├── tests/
│   ├── __init__.py
│   └── test_calculator.py
├── .gitlab-ci.yml
├── pyproject.toml
├── tox.ini
├── noxfile.py
└── README.md

2. Core Functionality

The calculator module contains two basic functions:

calculator.py

def add(a, b):
    """Returns the sum of two numbers."""
    return a + b

def subtract(a, b):
    """Returns the difference of two numbers."""
    return a - b

We will use pytest for unit testing (with parametrized tests):

test_calculator.py

import pytest
from tox_nox_python_tests.calculator import add, subtract

@pytest.mark.parametrize("a, b, expected", [
    (1, 2, 3),
    (-1, 1, 0),
    (0, 0, 0),
])
def test_add(a, b, expected):
    assert add(a, b) == expected

@pytest.mark.parametrize("a, b, expected", [
    (5, 3, 2),
    (10, 5, 5),
    (-1, -1, 0),
])
def test_subtract(a, b, expected):
    assert subtract(a, b) == expected

3. Dependency Management With Poetry

Poetry manages dependencies and environments. The pyproject.toml file is the modern way to manage Python projects and replaces traditional setup.py and setup.cfg files:

pyproject.toml

[tool.poetry]
name = "tox_nox_python_tests"
version = "0.1.0"
description = "Automate Python testing across multiple Python versions using Tox and Nox."
authors = ["Wallace Espindola <wallace.espindola@gmail.com>"]
readme = "README.md"
license = "MIT"

[tool.poetry.dependencies]
python = "^3.8"
pytest = "^8.3"

[build-system]
requires = ["poetry-core>=1.8.0"]
build-backend = "poetry.core.masonry.api"

Run the following commands to install dependencies and create a virtual environment:

poetry install

4. Running Unit Tests With Pytest

 You can run the basic unit tests this way:

poetry run pytest --verbose

5. Multi-Version Testing With Tox

Tox automates testing across Python versions using isolated virtual environments. The configuration is declarative and resides in a tox.ini file, testing the app with Python versions from 3.8 to 3.13:

tox.ini

[tox]
envlist = py38, py39, py310, py311, py312, py313

[testenv]
allowlist_externals = poetry
commands_pre =
    poetry install --no-interaction --no-root
commands =
    poetry run pytest --verbose

Run Tox using:

poetry run tox

Tox will automatically create environments for each Python version and execute the pytest suite in isolation.

Example output:

6. Flexible Testing With Nox

For more dynamic configurations, use Nox. Its Python-based scripts allow complex logic and custom workflows and test the app with Python versions from 3.8 to 3.13:

noxfile.py

import nox

@nox.session(python=["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"])
def tests(session):
    session.install("poetry")
    session.run("poetry", "install", "--no-interaction", "--no-root")
    session.run("pytest")

Run Nox using:

poetry run nox

Nox offers flexibility for customizing workflows, such as conditional dependency installation or environment-specific configurations.

Example output:

7. Integration With CI/CD

Automate the testing process further by integrating Tox and/or Nox into a CI/CD pipeline. 

For example, a GitHub Actions workflow file .github/workflows/python-tests.yml can look like this for Tox or Nox tests:

name: Python Tests Tox (or Nox)

on:
  push:
    branches: [ main, develop ]

jobs:
  test:
    runs-on: ubuntu-latest
   
    steps:
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: ${PYTHON_VERSION}
          
      - name: Install Poetry
        run: pip install poetry
        
      - name: Install Dependencies
        run: poetry install --no-interaction --no-root
        
      - name: Run Tests with Tox (or Nox)
        run: poetry run tox (or nox)

And it can look like this for parallel matrix tests (pure CI/CD):

name: Python Tests with Matrix

on:
  push:
    branches: [ main, develop ]

jobs:
  test:
    runs-on: ubuntu-latest
    
    strategy:
      matrix:
        python-version: [3.8, 3.9, 3.10, 3.11, 3.12, 3.13]
    
    steps:
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          
      - name: Install Poetry
        run: pip install poetry
        
      - name: Install Dependencies
        run: poetry install --no-interaction --no-root
        
      - name: Run Tests with Pytest
        run: poetry run pytest --verbose

As an option, the following GitLab CI/CD pipeline .gitlab-ci.yml runs tests with Tox or Nox (keep in mind that you run either Tox or Nox in your project, not both, since they have similar goals):

stages:
  - test

tox_tests:
  stage: test
  image: python:${PYTHON_VERSION}
  variables:
    PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  before_script:
    - pip install poetry
    - poetry install --no-interaction --no-root
  script:
    - poetry run tox (or nox)
  cache:
    paths:
      - .cache/pip/

Also, this one runs the tests across multiple versions with a parallel matrix (pure CI/CD):

stages:
  - test

matrix_tests:
  stage: test
  image: python:${PYTHON_VERSION}
  variables:
    PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  before_script:
    - pip install poetry
    - poetry install --no-interaction --no-root
  script:
    - poetry run pytest --verbose
  cache:
    paths:
      - .cache/pip/
  parallel:
    matrix:
      - PYTHON_VERSION: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]

This configuration uses the matrix keyword to run tests across specified Python versions in parallel. 

Conclusion

By combining Tox or Nox with Poetry and Pytest, you can achieve seamless multi-version test automation for Python projects. Whether you prefer the declarative approach of Tox or the programmable flexibility of Nox, these tools ensure your code is validated across a wide range of Python environments. That approach is especially useful for shared libs projects and multi-user environments projects. Also, for a full DevOps approach, you can use CI/CD matrix tests for multiple versions, a powerful and clean solution.

For the complete example, check the project repository GitHub: tox-nox-python-tests.

For other engaging tech discussion, follow me on my LinkedIn.

References

This project uses Tox, Nox, CI/CD, Poetry, and Pytest for test automation. For detailed documentation, refer to:

• Tox Documentation
• Nox Documentation
• Poetry Documentation
• Pytest Documentation
• GitLab CI/CD Documentation
• GitHub Actions CI/CD Documentation