Git and deployment

Lesson 16 — Python

Lesson 16 of 20 Intermediate ~75 min

Learning objectives

• Apply Git effectively to a Python project
• Enforce quality automatically with pre-commit hooks
• Package a project for distribution
• Containerise and deploy an application
• Build a CI/CD pipeline
• Structure logging and configuration for production

Git for Python projects

The Git fundamentals are language-agnostic — see R Programming lesson 12 for commits, branches, pull requests and recovery. This lesson covers what is specific to Python.

.gitignore

# Environments
.venv/
venv/
env/
.conda/

# Byte-compiled
__pycache__/
*.py[cod]
*$py.class
*.so

# Packaging
build/
dist/
*.egg-info/
.eggs/

# Tooling caches
.pytest_cache/
.mypy_cache/
.ruff_cache/
.tox/
.coverage
coverage.xml
htmlcov/

# Notebooks
.ipynb_checkpoints/

# Editors
.idea/
.vscode/
*.swp
.DS_Store

# Secrets
.env
.env.local
*.pem
credentials.json

# Data — never commit patient data
data/raw/
data/derived/
*.sas7bdat
*.xpt
*.parquet
output/

Start from github/gitignore’s Python template and add the data and secrets sections.

WarningNotebooks in Git

.ipynb files store output, execution counts and metadata as JSON. Every run produces a diff even when the code did not change, and cell output can contain patient data.

pip install nbstripout
nbstripout --install          # a git filter that strips output on commit

Or use jupytext to pair each notebook with a .py file and commit only that:

jupytext --set-formats ipynb,py:percent notebook.ipynb

Committing raw notebooks with output is how patient data ends up in a repository.

pre-commit

Runs checks before each commit, so problems never reach the repository.

pip install pre-commit
pre-commit install

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-toml
      - id: check-added-large-files
        args: ["--maxkb=1000"]
      - id: check-merge-conflict
      - id: detect-private-key
      - id: no-commit-to-branch
        args: ["--branch", "main"]

  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.4.10
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format

  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.10.0
    hooks:
      - id: mypy
        additional_dependencies: [pandas-stubs, types-requests]

  - repo: https://github.com/kynan/nbstripout
    rev: 0.7.1
    hooks:
      - id: nbstripout

pre-commit run --all-files      # run against everything, first time
pre-commit autoupdate           # bump the hook versions
git commit --no-verify          # bypass — use sparingly

detect-private-key and check-added-large-files are the two that earn their place immediately: they prevent the two mistakes that are painful to undo.

Linting and formatting

ruff replaces flake8, isort, pyupgrade, pydocstyle and several others, and is fast enough to run on save.

pip install ruff

ruff check .                    # lint
ruff check --fix .              # autofix
ruff format .                   # format (replaces black)
ruff format --check .           # check without changing

[tool.ruff]
line-length = 88
target-version = "py311"

[tool.ruff.lint]
select = [
    "E", "W",   # pycodestyle
    "F",        # pyflakes
    "I",        # isort
    "N",        # pep8-naming
    "UP",       # pyupgrade
    "B",        # flake8-bugbear — catches the mutable-default trap
    "SIM",      # flake8-simplify
    "PD",       # pandas-vet
    "RUF",      # ruff-specific
]
ignore = ["E501"]

[tool.ruff.lint.per-file-ignores]
"tests/*" = ["S101"]            # assert is fine in tests

B (bugbear) is the ruleset worth enabling above all others — it catches mutable default arguments, loop variable binding in closures, and several other real bugs rather than style issues.

Packaging

# pyproject.toml
[project]
name = "study-abc101"
version = "1.2.0"
description = "Analysis programs for study ABC-101"
readme = "README.md"
requires-python = ">=3.11"
license = {text = "MIT"}
dependencies = ["pandas>=2.0", "pyreadstat>=1.2"]

[project.scripts]
run-adam = "study.pipeline:main"
validate = "study.validation:cli"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/study"]

pip install build twine

python -m build                 # builds sdist and wheel into dist/
twine check dist/*
twine upload --repository testpypi dist/*
twine upload dist/*             # PyPI

# Internal index
pip install --index-url https://pypi.company.com/simple/ study-abc101

[project.scripts] creates command-line entry points:

# src/study/pipeline.py
import argparse
import sys


def main() -> int:
    parser = argparse.ArgumentParser(description="Run the ADaM pipeline")
    parser.add_argument("--config", required=True, help="Path to the config file")
    parser.add_argument("--stage", choices=["adam", "tlf", "all"], default="all")
    parser.add_argument("--dry-run", action="store_true")
    args = parser.parse_args()

    try:
        run_pipeline(args.config, args.stage, dry_run=args.dry_run)
    except Exception:
        logging.exception("Pipeline failed")
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

run-adam --config config/study.yaml --stage adam

Returning an exit code — 0 for success, non-zero for failure — is what lets a scheduler or CI system detect that the run failed.

Configuration

# src/study/config.py
from dataclasses import dataclass
from pathlib import Path
import os
import yaml


@dataclass(frozen=True)
class Config:
    study_id: str
    sdtm_path: Path
    adam_path: Path
    output_path: Path
    log_level: str = "INFO"
    n_workers: int = 4

    @classmethod
    def from_yaml(cls, path: str | Path) -> "Config":
        raw = yaml.safe_load(Path(path).read_text())

        # Environment variables override the file
        env = os.getenv("STUDY_ENV", "development")
        merged = {**raw.get("default", {}), **raw.get(env, {})}

        return cls(
            study_id=merged["study_id"],
            sdtm_path=Path(merged["sdtm_path"]),
            adam_path=Path(merged["adam_path"]),
            output_path=Path(merged["output_path"]),
            log_level=os.getenv("LOG_LEVEL", merged.get("log_level", "INFO")),
            n_workers=int(os.getenv("N_WORKERS", merged.get("n_workers", 4))),
        )

# config/study.yaml
default:
  study_id: ABC-101
  sdtm_path: data/raw/sdtm
  adam_path: data/adam
  output_path: output
  log_level: INFO

production:
  sdtm_path: /mnt/production/abc101/sdtm
  adam_path: /mnt/production/abc101/adam
  output_path: /mnt/production/abc101/output
  log_level: WARNING
  n_workers: 16

Secrets never go in the YAML:

db_password = os.environ["DB_PASSWORD"]        # KeyError if unset — fail loudly
api_key = os.getenv("API_KEY")                 # None if unset

yaml.safe_load, never yaml.load — the latter can execute arbitrary Python from the file.

Logging

# src/study/logging_config.py
import logging
import logging.config
from pathlib import Path


def configure_logging(level: str = "INFO", log_file: Path | None = None) -> None:
    handlers: dict = {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "simple",
            "level": level,
        }
    }
    if log_file:
        log_file.parent.mkdir(parents=True, exist_ok=True)
        handlers["file"] = {
            "class": "logging.handlers.RotatingFileHandler",
            "filename": str(log_file),
            "maxBytes": 10_000_000,
            "backupCount": 5,
            "formatter": "detailed",
            "level": "DEBUG",
        }

    logging.config.dictConfig({
        "version": 1,
        "disable_existing_loggers": False,
        "formatters": {
            "simple":   {"format": "%(levelname)-8s %(message)s"},
            "detailed": {"format":
                "%(asctime)s %(levelname)-8s [%(name)s:%(lineno)d] %(message)s"},
        },
        "handlers": handlers,
        "root": {"level": "DEBUG", "handlers": list(handlers)},
    })

import logging

logger = logging.getLogger(__name__)          # module-level, named after the module


def derive_adsl(dm, ex):
    logger.info("Deriving ADSL from %d DM and %d EX records", len(dm), len(ex))
    try:
        adsl = _derive(dm, ex)
    except Exception:
        logger.exception("ADSL derivation failed")     # includes the traceback
        raise
    logger.info("ADSL: %d subjects, %d variables", len(adsl), len(adsl.columns))
    return adsl

Use %s placeholders rather than f-strings in log calls: the formatting is deferred and skipped entirely if the level is disabled. logger.exception() inside an except block includes the traceback automatically.

Docker

# syntax=docker/dockerfile:1

FROM python:3.11.9-slim AS builder

COPY --from=ghcr.io/astral-sh/uv:0.4.20 /uv /usr/local/bin/uv

ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy

WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-install-project --no-dev

COPY src/ ./src/
RUN uv sync --frozen --no-dev


FROM python:3.11.9-slim AS runtime

RUN apt-get update && apt-get install -y --no-install-recommends curl \
    && rm -rf /var/lib/apt/lists/*

RUN useradd --create-home --uid 1000 appuser

WORKDIR /app
COPY --from=builder --chown=appuser:appuser /app/.venv /app/.venv
COPY --from=builder --chown=appuser:appuser /app/src   /app/src

ENV PATH="/app/.venv/bin:$PATH" \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1

USER appuser

HEALTHCHECK --interval=30s --timeout=5s --start-period=20s \
  CMD curl -fsS http://localhost:8000/health || exit 1

EXPOSE 8000
CMD ["python", "-m", "study.pipeline"]

The multi-stage build keeps build tools out of the runtime image, which is smaller and has a smaller attack surface.

# docker-compose.yml
services:
  app:
    build: .
    image: study-abc101:1.2.0
    ports: ["127.0.0.1:8000:8000"]
    environment:
      DB_PASSWORD: ${DB_PASSWORD:?must be set}
      STUDY_ENV: production
      LOG_LEVEL: INFO
    volumes:
      - type: bind
        source: /mnt/studies/abc101
        target: /data
        read_only: true
      - ./logs:/app/logs
    restart: unless-stopped
    deploy:
      resources:
        limits: {memory: 4G, cpus: "2.0"}

CI/CD

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
    tags: ["v*"]
  pull_request:

env:
  PYTHON_VERSION: "3.11"

jobs:
  quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
        with:
          enable-cache: true
      - run: uv sync --frozen --extra dev
      - run: uv run ruff check --output-format=github .
      - run: uv run ruff format --check .
      - run: uv run mypy src/

  test:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, windows-latest]
        python-version: ["3.11", "3.12"]
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv python install ${{ matrix.python-version }}
      - run: uv sync --frozen --extra dev
      - run: uv run pytest --cov=study --cov-report=xml
      - uses: codecov/codecov-action@v4
        if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11'

  build:
    needs: [quality, test]
    if: startsWith(github.ref, 'refs/tags/v')
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4

      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - uses: docker/metadata-action@v5
        id: meta
        with:
          images: ghcr.io/${{ github.repository }}
          tags: |
            type=semver,pattern={{version}}
            type=sha

      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

Building and pushing only on a tag, and only after quality and tests pass, means every deployed image corresponds to a named, reviewable version.

Deployment targets

Target	Suits	Notes
Docker on a VM	Full control	You manage everything
Kubernetes	Scale, multiple services	Substantial operational overhead
AWS ECS / Fargate	Containers without cluster management
Azure Container Apps	Same, Azure
Google Cloud Run	Scale to zero, per-request billing	Good for intermittent workloads
Streamlit Community Cloud	Streamlit apps	Free, public
Posit Connect	Shiny for Python, Quarto, APIs	Enterprise, authentication included
Databricks / Snowflake	Data-platform-native	Where the data already is
Airflow / Prefect / Dagster	Scheduled pipelines	Orchestration, retries, monitoring

For a scheduled analysis pipeline, an orchestrator is usually the right answer rather than a cron job:

# Prefect
from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def read_sdtm(domain: str):
    ...

@task
def derive_adsl(dm, ex):
    ...

@flow(name="ABC-101 ADaM pipeline", log_prints=True)
def adam_pipeline():
    dm = read_sdtm("dm")
    ex = read_sdtm("ex")
    adsl = derive_adsl(dm, ex)
    return adsl

R and Python side by side

Task	R	Python
Project config	DESCRIPTION	pyproject.toml
Lockfile	renv.lock	uv.lock
Build	R CMD build	python -m build
Install	install.packages()	pip install
Lint	lintr	ruff check
Format	styler	ruff format
Type check	none standard	mypy
Test	testthat	pytest
Pre-commit	precommit package	pre-commit
CI	r-lib/actions	setup-python / setup-uv
Container	rocker/*	python:*-slim
Publish	CRAN	PyPI

Common mistakes

Mistake	Consequence	Fix
Committing notebooks with output	Diff noise; possible patient data	nbstripout
Secrets in the repository	Leaked permanently	.env, detect-private-key
No pre-commit hooks	Problems reach the repository	Install them
latest base image	Non-reproducible builds	Pin the exact tag
Root user in the container	Larger blast radius	USER appuser
f-strings in log calls	Formatting cost even when disabled	%s placeholders
yaml.load	Arbitrary code execution	yaml.safe_load
Deploying without a tag	Cannot say what is running	Tag, and deploy on tags
No exit code from the CLI	Failures look like successes	sys.exit(main())

Exercise 11.1 — Production-ready project setup

Set up a complete Python project with pre-commit hooks, CI, a container build, and a CLI entry point. Explain each choice.

Show solution

Structure

study-abc101/
├── .github/workflows/ci.yml
├── .pre-commit-config.yaml
├── .gitignore
├── .python-version
├── Dockerfile
├── docker-compose.yml
├── Makefile
├── README.md
├── pyproject.toml
├── uv.lock
├── config/
│   └── study.yaml
├── src/study/
│   ├── __init__.py
│   ├── cli.py
│   ├── config.py
│   ├── logging_config.py
│   ├── derivations.py
│   └── pipeline.py
└── tests/
    ├── conftest.py
    └── test_derivations.py

src/study/cli.py

"""Command-line interface for the ABC-101 analysis pipeline."""

from __future__ import annotations

import argparse
import logging
import sys
from pathlib import Path

from study import __version__
from study.config import Config
from study.logging_config import configure_logging
from study.pipeline import run_pipeline

logger = logging.getLogger(__name__)


def build_parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(
        prog="run-adam",
        description="Run the ABC-101 ADaM and TLF pipeline.",
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )
    p.add_argument("--config", type=Path, default=Path("config/study.yaml"),
                   help="Path to the configuration file")
    p.add_argument("--stage", choices=["adam", "tlf", "all"], default="all",
                   help="Which stage to run")
    p.add_argument("--log-file", type=Path, default=None,
                   help="Write a detailed log to this file")
    p.add_argument("--log-level", default="INFO",
                   choices=["DEBUG", "INFO", "WARNING", "ERROR"])
    p.add_argument("--dry-run", action="store_true",
                   help="Validate inputs and report the plan without writing output")
    p.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
    return p


def main(argv: list[str] | None = None) -> int:
    """Entry point. Returns a process exit code."""
    args = build_parser().parse_args(argv)

    configure_logging(level=args.log_level, log_file=args.log_file)

    logger.info("study-abc101 %s starting (stage=%s, dry_run=%s)",
                __version__, args.stage, args.dry_run)

    try:
        config = Config.from_yaml(args.config)
    except FileNotFoundError:
        logger.error("Configuration file not found: %s", args.config)
        return 2
    except (KeyError, ValueError) as e:
        logger.error("Invalid configuration: %s", e)
        return 2

    try:
        summary = run_pipeline(config, stage=args.stage, dry_run=args.dry_run)
    except KeyboardInterrupt:
        logger.warning("Interrupted by user")
        return 130
    except Exception:
        logger.exception("Pipeline failed")
        return 1

    n_failed = sum(1 for s in summary if s["status"] == "FAILED")
    if n_failed:
        logger.error("%d of %d step(s) failed", n_failed, len(summary))
        return 1

    logger.info("Completed %d step(s) successfully", len(summary))
    return 0


if __name__ == "__main__":
    sys.exit(main())

Why each choice

Distinct exit codes. 0 success, 1 pipeline failure, 2 configuration error, 130 interrupted. A scheduler can distinguish “the data was bad” from “the config was wrong” and alert differently. A single 1 for everything loses that.

main(argv=None) taking an argument list. This makes the CLI testable without subprocesses:

def test_missing_config_returns_2(tmp_path):
    assert main(["--config", str(tmp_path / "nope.yaml")]) == 2

configure_logging before anything else. A configuration error must be logged, so logging has to be up first.

logger.exception rather than logger.error in the catch-all — it includes the traceback, which is the whole reason you are looking at the log.

--dry-run. Validating inputs and printing the plan without writing anything is the difference between discovering a path problem in ten seconds and discovering it after two hours of derivation.

Explicit KeyboardInterrupt handling. Ctrl-C should log a clean message and return 130 (the Unix convention), not print a traceback.

Makefile

.PHONY: help install test lint format check run docker clean

help:
    @grep -E '^[a-z-]+:.*?## .*$$' $(MAKEFILE_LIST) | \
      awk 'BEGIN {FS = ":.*?## "}; {printf "  %-12s %s\n", $$1, $$2}'

install:   ## Install the project and dev dependencies
    uv sync --frozen --extra dev
    uv run pre-commit install

test:      ## Run the test suite
    uv run pytest

lint:      ## Lint and type-check
    uv run ruff check .
    uv run ruff format --check .
    uv run mypy src/

format:    ## Autoformat and autofix
    uv run ruff format .
    uv run ruff check --fix .

check: lint test   ## Everything CI runs

run:       ## Run the pipeline locally
    uv run run-adam --config config/study.yaml --log-file logs/run.log

docker:    ## Build the container
    docker build -t study-abc101:$$(grep '^version' pyproject.toml | cut -d'"' -f2) .

clean:
    rm -rf .pytest_cache .mypy_cache .ruff_cache htmlcov .coverage dist build
    find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true

The help target with ## comments means make alone documents the project. A new team member runs make install and make check and is productive without reading anything.

What this setup actually prevents

Control	Prevents
pre-commit + detect-private-key	Credentials in the repository
pre-commit + nbstripout	Notebook output (possibly patient data) in Git
pre-commit + check-added-large-files	A 2 GB dataset committed by accident
ruff rule B	Mutable default arguments and closure bugs
mypy	Type errors that would surface at runtime
uv.lock + --frozen	Different package versions between machines
Pinned base image	A rebuild producing a different Python
Non-root container user	Container escape having root
Tag-gated Docker push	Deploying an unreviewed commit
Distinct exit codes	A scheduler reporting success on failure

Each is a few lines of configuration and each prevents a class of problem that is expensive to diagnose after the fact. The whole setup is perhaps an hour of work once, and it is reusable across every subsequent project.

Recap

• Strip notebook output before committing — it is diff noise and a data risk
• pre-commit with detect-private-key and check-added-large-files prevents the expensive mistakes
• ruff replaces five tools; enable the B ruleset for real bugs
• pyproject.toml + [project.scripts] gives a proper CLI entry point
• Multi-stage Docker builds, pinned base tags, non-root user
• %s placeholders in log calls; logger.exception() inside except
• Build and deploy on a tag, after tests pass
• Distinct exit codes so a scheduler can tell what went wrong

That completes the core Python track. You can read and write idiomatic Python, wrangle data with pandas, run statistical analyses, read and build clinical datasets, ship an application, and deploy it reproducibly.

The remaining four lessons cover generative AI — a distinct subject that builds on everything above, particularly virtual environments for dependency management and testing for the non-determinism that LLMs introduce.

Next: Large language models and prompt engineering.