Virtual environments

Lesson 15 — Python

Lesson 15 of 20 Intermediate ~60 min

Learning objectives

Explain why environment isolation is not optional in Python
Create and manage environments with venv, uv and conda
Pin dependencies with a lockfile
Structure a project with pyproject.toml
Reproduce an environment exactly on another machine

Why this matters more in Python than in R

R installs packages into a shared library and, for the most part, packages coexist. Python’s dependency resolution is stricter: two projects needing different versions of the same package genuinely cannot share an installation.

Without isolation:

pip install pandas==1.5    # for the old study
pip install pandas==2.2    # for the new one — overwrites the first

The old study now silently runs on pandas 2.2, where applymap is deprecated and copy-on-write semantics differ. This is renv’s argument, but the failure mode is more immediate.

`venv` — the standard library

python -m venv .venv

source .venv/bin/activate          # macOS / Linux
.venv\Scripts\activate             # Windows

which python                       # /path/to/project/.venv/bin/python

pip install pandas numpy
pip list
pip freeze > requirements.txt

deactivate

Always create the environment inside the project as .venv, and add it to .gitignore. Editors and tools discover .venv automatically.

# .gitignore
.venv/
__pycache__/
*.pyc
.pytest_cache/
.coverage

`uv` — the modern tool

uv is a Rust reimplementation of pip and venv that is 10–100× faster and handles locking properly. It has become the default recommendation.

curl -LsSf https://astral.sh/uv/install.sh | sh

uv venv                            # create .venv
uv pip install pandas numpy        # install, very fast
uv pip compile requirements.in -o requirements.txt   # lock
uv pip sync requirements.txt       # install EXACTLY the lockfile

# Project management
uv init myproject
uv add pandas
uv add --dev pytest ruff
uv run pytest                      # runs in the project environment
uv lock                            # write uv.lock
uv sync                            # reproduce from uv.lock

uv sync is the equivalent of renv::restore(). The difference from pip install -r requirements.txt is that sync also removes packages not in the lockfile, so the environment matches exactly rather than being a superset.

`conda` and `mamba`

Manages non-Python dependencies too — compilers, CUDA, system libraries. The right choice for scientific computing with binary dependencies.

conda create -n study python=3.11
conda activate study
conda install pandas numpy scipy -c conda-forge
conda env export > environment.yml
conda env create -f environment.yml

mamba create -n study python=3.11    # much faster resolver

# environment.yml
name: study
channels:
  - conda-forge
dependencies:
  - python=3.11
  - pandas=2.2
  - numpy=1.26
  - pyreadstat=1.2
  - pip
  - pip:
      - shiny==0.10.2

Which tool

uv — the default for new projects. Fast, correct locking, actively developed.
venv + pip — no installation needed, universally available.
conda/mamba — when you need non-Python dependencies, or on a platform where wheels are unavailable.

Do not mix them in one project. Mixing conda install and pip install in the same environment is a documented source of broken installations.

Dependency specification

# requirements.txt — direct dependencies, loosely pinned
pandas>=2.0,<3.0
numpy>=1.24
pyreadstat>=1.2
plotly>=5.18

# requirements-dev.txt
-r requirements.txt
pytest>=7.4
pytest-cov
ruff
mypy

pip install -r requirements-dev.txt

Version specifiers

pandas==2.2.0          exactly
pandas>=2.0            at least
pandas>=2.0,<3.0       a range
pandas~=2.2.0          compatible: >=2.2.0, <2.3.0
pandas!=2.1.0          exclude a known-bad version

Use ranges in requirements.in (what you need) and exact pins in requirements.txt (what you tested). That distinction is the whole point of locking.

`pyproject.toml`

The modern single configuration file for the whole project.

[project]
name = "study-abc101"
version = "1.2.0"
description = "Analysis programs for study ABC-101"
requires-python = ">=3.11"
authors = [{name = "Ram Gaduputi", email = "ram@example.com"}]
dependencies = [
    "pandas>=2.0,<3.0",
    "numpy>=1.24",
    "pyreadstat>=1.2",
    "plotly>=5.18",
]

[project.optional-dependencies]
dev = ["pytest>=7.4", "pytest-cov", "ruff", "mypy"]
app = ["shiny>=0.10", "shinywidgets"]

[project.scripts]
run-adam = "study.cli:main"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-q --strict-markers --cov=study"

[tool.ruff]
line-length = 88
target-version = "py311"

[tool.ruff.lint]
select = ["E", "F", "I", "N", "UP", "B", "SIM", "PD"]
# E/F pycodestyle+pyflakes, I isort, N naming, UP pyupgrade,
# B bugbear, SIM simplify, PD pandas-vet

[tool.mypy]
python_version = "3.11"
warn_return_any = true
disallow_untyped_defs = true

pip install -e .              # editable install of the project
pip install -e ".[dev]"       # with the dev extras

One file replaces setup.py, setup.cfg, requirements.txt, pytest.ini, .flake8 and mypy.ini.

Locking

The distinction that matters:

File	Contains	Purpose
`pyproject.toml` / `requirements.in`	Direct dependencies, ranges	What you need
`requirements.txt` / `uv.lock`	Every package, exact versions, hashes	What you tested

# pip-tools
pip install pip-tools
pip-compile requirements.in -o requirements.txt
pip-sync requirements.txt

# uv (faster, same idea)
uv pip compile requirements.in -o requirements.txt
uv pip sync requirements.txt

# uv project workflow
uv lock
uv sync

A generated lockfile:

# This file was autogenerated by uv via the following command:
#    uv pip compile requirements.in -o requirements.txt
numpy==1.26.4
    # via
    #   pandas
    #   pyreadstat
pandas==2.2.2
    # via -r requirements.in
python-dateutil==2.9.0.post0
    # via pandas
pytz==2024.1
    # via pandas

Every transitive dependency, exactly pinned, with the reason it is there. Commit it.

pip freeze is not a lockfile

pip freeze > requirements.txt

This records what happens to be installed — including packages you installed and forgot, editable installs with local paths, and nothing about why anything is there. It also cannot distinguish direct from transitive dependencies, so you can never safely remove anything.

Use pip-compile or uv lock, which take a declared set of direct dependencies and resolve them.

Hashes

For a regulated or security-sensitive environment:

uv pip compile requirements.in --generate-hashes -o requirements.txt

pandas==2.2.2 \
    --hash=sha256:9e79019aba43cb4fda9e4d983f8e88ca0373adbb697ae9c6c43093218de28b54 \
    --hash=sha256:1cb51fe389360f3b5a4d57dbd2848a5f033350336ca3b340d1c53a1fad33bcad

pip install then verifies the download against the hash, so a compromised or substituted package fails to install. This is what an archived, auditable environment requires.

Reproducing an environment

git clone https://github.com/org/study-abc101
cd study-abc101

uv venv
uv pip sync requirements.txt
uv run pytest

Or with plain tooling:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pytest

A lockfile is not full reproducibility

It pins Python packages. It does not pin:

The Python version — python_requires constrains but does not fix it
The operating system and system libraries
Compiler versions for anything built from source
Locale, timezone, and environment variables

For a deliverable that must be reproducible in three years, you need the lockfile and a container image, exactly as the R lesson argues for renv plus Docker.

Containers

FROM python:3.11.9-slim

WORKDIR /app

# Install dependencies first — this layer caches
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Then the code — changes here do not reinstall packages
COPY src/ ./src/
COPY pyproject.toml .
RUN pip install --no-cache-dir -e . --no-deps

RUN useradd -m -u 1000 appuser && chown -R appuser /app
USER appuser

CMD ["python", "-m", "study.pipeline"]

Faster, with uv:

FROM python:3.11.9-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv

WORKDIR /app
COPY requirements.txt .
RUN uv pip install --system --no-cache -r requirements.txt

COPY src/ ./src/
CMD ["python", "-m", "study.pipeline"]

Pin the base image tag (3.11.9-slim, not 3.11 or latest) or a rebuild in six months produces a different Python.

Project layout

study-abc101/
├── pyproject.toml
├── requirements.in
├── requirements.txt          # locked, committed
├── .python-version           # for pyenv / uv
├── .gitignore
├── README.md
├── Dockerfile
├── .venv/                    # NOT committed
├── src/
│   └── study/
│       ├── __init__.py
│       ├── derivations.py
│       ├── validation.py
│       └── pipeline.py
├── tests/
│   ├── conftest.py
│   └── test_derivations.py
├── data/                     # NOT committed
│   ├── raw/
│   └── derived/
├── notebooks/
└── output/                   # NOT committed

The src/ layout matters: it forces you to install the package to import it, which means the tests exercise the installed package rather than the working directory. Bugs where a module works locally and fails on install are caught immediately.

R and Python side by side

Task	R	Python
Isolated environment	`renv::init()`	`python -m venv .venv` or `uv venv`
Install a package	`install.packages("x")`	`pip install x` or `uv add x`
Record the state	`renv::snapshot()`	`uv lock` / `pip-compile`
Restore the state	`renv::restore()`	`uv sync` / `pip-sync`
Lockfile	`renv.lock`	`uv.lock` / `requirements.txt`
Project config	`DESCRIPTION`	`pyproject.toml`
Check state	`renv::status()`	`uv pip check`
Update	`renv::update()`	`uv lock --upgrade`
Activate	Automatic in the project	`source .venv/bin/activate`

The main practical difference: renv activates automatically when you open the project, and Python environments must be activated explicitly. uv run and editor integration mostly hide this, but it is the source of the most common Python confusion — “I installed it and it says it is not installed” almost always means the wrong environment is active.

Common mistakes

Mistake	Consequence	Fix
No virtual environment	Version conflicts across projects	One `.venv` per project
Committing `.venv/`	A huge, machine-specific repository	`.gitignore`
`pip freeze` as a lockfile	Unremovable cruft, no provenance	`pip-compile` / `uv lock`
Only loose version ranges	Different versions on different machines	Commit the lockfile
Mixing conda and pip	Broken installations	Pick one per environment
`latest` base image	Non-reproducible rebuilds	Pin the exact tag
Installing into the system Python	Breaks OS tooling	Always a virtual environment
Wrong environment active	“Module not found” for an installed package	`which python`

Exercise 10.1 — Set up a reproducible project

Create a project structure for a clinical analysis with locked dependencies, dev tooling, a Dockerfile, and instructions a colleague can follow to reproduce it exactly.

Show solution

uv init study-abc101
cd study-abc101

pyproject.toml

[project]
name = "study-abc101"
version = "1.0.0"
description = "Analysis programs for study ABC-101"
readme = "README.md"
requires-python = ">=3.11,<3.13"
authors = [{name = "Ram Gaduputi", email = "ram@example.com"}]

dependencies = [
    "pandas>=2.0,<3.0",
    "numpy>=1.24,<2.0",
    "pyreadstat>=1.2",
    "pyarrow>=15.0",
    "plotly>=5.18",
    "openpyxl>=3.1",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.4",
    "pytest-cov>=4.1",
    "ruff>=0.4",
    "mypy>=1.9",
    "pandas-stubs",
]
app = ["shiny>=0.10", "shinywidgets>=0.3"]

[project.scripts]
run-adam = "study.pipeline:main"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/study"]

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-q --strict-markers --cov=study --cov-report=term-missing"
markers = ["slow: long-running tests"]

[tool.ruff]
line-length = 88
target-version = "py311"
src = ["src", "tests"]

[tool.ruff.lint]
select = ["E", "F", "I", "N", "UP", "B", "SIM", "PD", "RUF"]
ignore = ["E501"]          # line length handled by the formatter

[tool.mypy]
python_version = "3.11"
disallow_untyped_defs = true
warn_return_any = true
warn_unused_ignores = true

[[tool.mypy.overrides]]
module = ["pyreadstat.*", "plotly.*"]
ignore_missing_imports = true

[tool.coverage.report]
fail_under = 85
exclude_lines = ["pragma: no cover", "if TYPE_CHECKING:"]

Lock it

uv lock
uv sync --extra dev

.python-version

3.11.9

.gitignore

.venv/
__pycache__/
*.py[cod]
.pytest_cache/
.mypy_cache/
.ruff_cache/
.coverage
htmlcov/
dist/
build/
*.egg-info/

# Patient data — never commit
data/raw/
data/derived/
data/submission/
*.sas7bdat
*.xpt
*.parquet

# Output — regenerable
output/
.ipynb_checkpoints/
.env

Dockerfile

FROM python:3.11.9-slim AS base

COPY --from=ghcr.io/astral-sh/uv:0.4.20 /uv /usr/local/bin/uv

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    UV_COMPILE_BYTECODE=1 \
    UV_LINK_MODE=copy

WORKDIR /app

# Dependency layer — caches unless the lockfile changes
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-install-project --no-dev

# Application layer
COPY src/ ./src/
RUN uv sync --frozen --no-dev

RUN useradd --create-home --uid 1000 appuser && chown -R appuser:appuser /app
USER appuser

ENV PATH="/app/.venv/bin:$PATH"

CMD ["python", "-m", "study.pipeline"]

--frozen makes the build fail if uv.lock is out of date with pyproject.toml, rather than silently resolving something different. That is what makes the container reproducible.

Makefile

.PHONY: install test lint format check clean docker

install:
    uv sync --extra dev

test:
    uv run pytest

lint:
    uv run ruff check .
    uv run mypy src/

format:
    uv run ruff format .
    uv run ruff check --fix .

check: lint test

clean:
    rm -rf .pytest_cache .mypy_cache .ruff_cache htmlcov .coverage
    find . -type d -name __pycache__ -exec rm -rf {} +

docker:
    docker build -t study-abc101:$$(grep '^version' pyproject.toml | cut -d'"' -f2) .

README.md

# Study ABC-101 Analysis

## Reproducing this environment

Requires Python 3.11 and [uv](https://docs.astral.sh/uv/).

```bash
git clone https://github.com/org/study-abc101
cd study-abc101
uv sync --extra dev
uv run pytest
```

`uv sync` installs exactly the versions in `uv.lock` — including removing
anything not in it. The environment will be byte-identical to the one used to
produce the delivered output.

### Without uv

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
```

This resolves versions afresh and may differ from the lockfile. Use uv for
anything that must match a delivery.

## Running the pipeline

```bash
uv run run-adam --config config/study.yaml
```

## Reproducibility

| Component | Pinned by |
|---|---|
| Python packages | `uv.lock` |
| Python version | `.python-version`, `requires-python` |
| OS and system libraries | `Dockerfile` base image tag |
| Source code | Git tag |

For a delivery, record: the Git tag, the `uv.lock` hash, and the Docker image
digest.

```bash
git rev-parse HEAD
sha256sum uv.lock
docker inspect --format='{{index .RepoDigests 0}}' study-abc101:1.0.0
```

Verification

# Does a clean clone actually reproduce?
git clone . /tmp/verify && cd /tmp/verify
uv sync --frozen
uv run pytest
uv pip list > /tmp/verify-packages.txt
diff /tmp/verify-packages.txt <(cd - && uv pip list)

That last check is the one worth actually running. “It works on my machine after a fresh clone” is the claim the whole setup exists to support, and it is cheap to verify.

Recap

One virtual environment per project, named .venv, gitignored
uv is the modern default; venv+pip always works; conda for binary dependencies
pip freeze is not a lockfile — use pip-compile or uv lock
Ranges in the declaration, exact pins in the lockfile; commit the lockfile
pyproject.toml replaces six other configuration files
--generate-hashes for auditable environments
A lockfile plus a pinned container image is what full reproducibility requires
src/ layout so tests exercise the installed package

Next: Git and deployment.